Why do UDP latencies spike?
As described in TCP/IP Illustrated, Volume 1, by W. Richard Stevens, Section 11.9, ARP entries that expire may introduce high latencies on the next send. Entry expiration details are system-dependent; for instance, a packet sent to a particular address may refresh the timeout.
On a VxWorks target, the RTI latency test exhibited this behavior as an unusually high maximum latency for one trial (see the third row from the bottom):
bytes, stdev us,ave us, min us, 50% us, 90% us, 99% us, 99.99%, max us ------,-------, -------,------, ------, ------, ------, ------, ------- 16, 2.6, 192.9, 175.6, 193.0, 194.0, 195.0, 242.0, 289.8 32, 2.2, 197.8, 180.1, 197.0, 198.0, 199.0, 246.0, 308.9 64, 2.4, 208.3, 192.8, 208.0, 208.0, 212.0, 258.0, 307.5 128, 2.9, 228.8, 217.5, 228.0, 228.0, 236.0, 278.0, 342.0 256, 3.1, 274.3, 264.0, 272.0, 274.0, 290.0, 322.0, 385.2 512, 3.0, 363.7, 351.9, 362.0, 364.0, 376.0, 412.0, 468.5 1024, 94.5, 542.2, 535.8, 540.0, 540.0, 556.0, 592.0, 67350.9 2048, 3.3, 795.8, 790.8, 792.0, 792.0, 804.0, 844.0, 923.4 4096, 3.6, 1169.8, 1162.8, 1160.0, 1170.0, 1180.0, 1240.0, 1395.1
One workaround is to increase the ARP timeout. The workaround is again system-dependent; for example, on VxWorks 5.5, the timeout is defined by the global variable, arpt_keep, which defaults to 1200 (20 minutes).