Dell Networking Intermittent High ICMP Latency seen in S-series Switches running OS9 Firmware
Summary: Dell Networking Intermittent High ICMP Latency seen in S-series Switches running OS9 Firmware
Symptoms
An intermittent issue regarding high latency is seen when attempting to perform an ICMP ping from a client device to the gateway, generally the IP of the VLAN interface. This issue can still be seen even when directly connected to a gateway switch that has little to no production traffic. There has been some customer concern that this reflects an impact on normal production traffic.
Issue seen across all S-series platforms running OS9 firmware.
Test scenario for ICMP test (No production traffic)
Topology Configuration
PC1: 192.168.10.100/24, Gateway: 192.168.10.1/24, VLAN10
PC2: 192.168.10.200/24, Gateway: 192.168.10.1/24, VLAN10
OrPC2: 192.168.20.200/24, Gateway: 192.168.20.1/24, VLAN20
PC1 ping switch gateway (directly connected) Test 1-6; same result seen, regardless of topology
Pinging 192.168.10.1 with 32 bytes of data: Reply from 192.168.10.1: bytes=32 time<1ms TTL=255 Reply from 192.168.10.1: bytes=32 time=5ms TTL=255 Reply from 192.168.10.1: bytes=32 time=47ms TTL=255 <------ high latency ! Reply from 192.168.10.1: bytes=32 time=5ms TTL=255 Reply from 192.168.10.1: bytes=32 time=5ms TTL=255 Reply from 192.168.10.1: bytes=32 time=1ms TTL=255 Reply from 192.168.10.1: bytes=32 time<1ms TTL=255 Reply from 192.168.10.1: bytes=32 time<1ms TTL=255 Reply from 192.168.10.1: bytes=32 time<1ms TTL=255 Reply from 192.168.10.1: bytes=32 time<1ms TTL=255 Reply from 192.168.10.1: bytes=32 time=32ms TTL=255 <------ high latency ! Reply from 192.168.10.1: bytes=32 time<1ms TTL=255 Reply from 192.168.10.1: bytes=32 time<1ms TTL=255 Reply from 192.168.10.1: bytes=32 time<1ms TTL=255 Reply from 192.168.10.1: bytes=32 time<1ms TTL=255 Reply from 192.168.10.1: bytes=32 time<1ms TTL=255 Reply from 192.168.10.1: bytes=32 time=6ms TTL=255 Reply from 192.168.10.1: bytes=32 time<1ms TTL=255 Reply from 192.168.10.1: bytes=32 time=40ms TTL=255 <------ high latency ! Reply from 192.168.10.1: bytes=32 time<1ms TTL=255 Reply from 192.168.10.1: bytes=32 time<1ms TTL=255 …… >>> (truncated) Ping statistics for 192.168.10.1: Packets: Sent = 8280,Received = 8280,Lost = 0 (0% Loss) Approximate round trip times in milli-seconds: Minimum = 0ms,Maximum = 55ms,Average = 1ms
Ping between two PCs, transit traffic (Layer2, in same and different vlan's) (Test 1-3)
Pinging 192.168.10.200 with 32 bytes of data: Reply from 192.168.10.200: bytes=32 time=1ms TTL=128 Reply from 192.168.10.200: bytes=32 time=1ms TTL=128 Reply from 192.168.10.200: bytes=32 time<1ms TTL=128 Reply from 192.168.10.200: bytes=32 time=1ms TTL=128 Reply from 192.168.10.200: bytes=32 time=1ms TTL=128 Reply from 192.168.10.200: bytes=32 time<1ms TTL=128 …… >>> (truncated), …… Reply from 192.168.10.200: bytes=32 time<1ms TTL=128 Reply from 192.168.10.200: bytes=32 time<1ms TTL=128 Reply from 192.168.10.200: bytes=32 time<1ms TTL=128 Reply from 192.168.10.200: bytes=32 time<1ms TTL=128 Reply from 192.168.10.200: bytes=32 time<1ms TTL=128 Reply from 192.168.10.200: bytes=32 time=1ms TTL=128 Ping statistics for 192.168.10.200: Packets: Sent = 1233,Received = 1233,Lost = 0 (0% Loss) Approximate round trip times in milli-seconds: Minimum = 0ms,Maximum = 2ms,Average = 0ms
Ping between two PCs, transit traffic (Layer3, in two vlans) (Tests 4-6)
Pinging 192.168.20.200 with 32 bytes of data: Reply from 192.168.20.200: bytes=32 time<1ms TTL=127 Reply from 192.168.20.200: bytes=32 time<1ms TTL=127 Reply from 192.168.20.200: bytes=32 time<1ms TTL=127 Reply from 192.168.20.200: bytes=32 time=1ms TTL=127 Reply from 192.168.20.200: bytes=32 time<1ms TTL=127 Reply from 192.168.20.200: bytes=32 time=1ms TTL=127 Reply from 192.168.20.200: bytes=32 time=1ms TTL=127 Reply from 192.168.20.200: bytes=32 time=1ms TTL=127 Reply from 192.168.20.200: bytes=32 time=1ms TTL=127 …… >>> (truncated), …… Reply from 192.168.20.200: bytes=32 time<1ms TTL=127 Reply from 192.168.20.200: bytes=32 time<1ms TTL=127 Reply from 192.168.20.200: bytes=32 time=1ms TTL=127 Reply from 192.168.20.200: bytes=32 time=1ms TTL=127 Reply from 192.168.20.200: bytes=32 time<1ms TTL=127 Reply from 192.168.20.200: bytes=32 time=1ms TTL=127 Ping statistics for 192.168.20.200: Packets: Sent = 808,Received = 808,Lost = 0 (0% Loss) Approximate round trip times in milli-seconds: Minimum = 0ms,Maximum = 2ms,Average = 0ms
Cause
Explanation of Results
It has been confirmed with Dell Engineering that this is normal and expected behavior due to CPU scheduling and queuing.
Resolution
ICMP testing should only be used as a means to test reachability, rather than latency or loss due to this behavior. ICMP traffic latency is not a reflection of the impact to normal production traffic. This behavior can also be exacerbated by a busy CPU - that is for a switch that is in full production, with a higher amount of traffic on it.