UNSOLVED
packets dropped > MTU?
I am seeing a problem with new N4032F and N2048 switches and jumbo frames. Things appear to be working, but the statistics show packets being dropped. For example:
show interfaces counters tengigabitethernet 1/0/5
Port InTotalPkts InUcastPkts InMcastPkts InBcastPkts
--------- ---------------- ---------------- ---------------- ----------------
Te1/0/5 2914419918 2914262020 58 157840
Port OutTotalPkts OutUcastPkts OutMcastPkts OutBcastPkts
--------- ---------------- ---------------- ---------------- ----------------
Te1/0/5 3585251489 3553257957 16903382 15090150
FCS Errors: ................................... 0
Single Collision Frames: ...................... 0
Late Collisions: .............................. 0
Excessive Collisions: ......................... 0
Multiple Collisions: .......................... 0
Received packets dropped > MTU: ............... 425952374
Transmitted packets dropped > MTU: ............ 211808792
Internal MAC Rx Errors: ....................... 0
Received Pause Frames: ........................ 37536
Transmitted Pause Frames: ..................... 0
Receive Packets Discarded: .................... 21
Transmit Packets Discarded: ................... 38475257
Received PFC Frames: .......................... 0
Transmitted PFC Frames: ....................... 0
console#show system mtu
System jumbo MTU size is 9216 bytes
The configuration does include:
system jumbo mtu 9216
We have several Vmware ESXi hosts configured for 9000 MTU, and MD3220i with the MTU set for 9000 (and vlan tagged all ports in question).
Here is our version info:
console#show version
Machine Description............... Dell Networking Switch
System Model ID................... N4032F
Machine Type...................... Dell Networking N4032F
Serial Number..................... CN05KGDH282984470025A00
Manufacturer...................... 0xbc00
Burned In MAC Address............. F8B1.5636.E046
System Object ID.................. 1.3.6.1.4.1.674.10895.3044
CPU Version....................... XLP308H-B2
SOC Version....................... BCM56842_A1
HW Version........................ 3
CPLD Version...................... 17
unit active backup current-active next-active
---- ----------- ----------- -------------- --------------
1 6.1.0.1 6.0.0.8 6.1.0.1 6.1.0.1
(The 2048 is also running 6.1.0.1)
Any ideas why the switch is reporting dropped packets? Is it just a bug on the counters, or are they actually being dropped?
Thanks,
John
PS: This shows a breakdown on different size packets:
console#show statistics tengigabitethernet 1/0/5
Total Packets Received (Octets)................ 1098152047898
Packets Received 64 Octets..................... 37536
Packets Received 65-127 Octets................. 1387863561
Packets Received 128-255 Octets................ 417995386
Packets Received 256-511 Octets................ 593761779
Packets Received 512-1023 Octets............... 66923691
Packets Received 1024-1518 Octets.............. 23713526
Packets Received > 1518 Octets................. 426111463
Packets RX and TX 64 Octets.................... 701251
Packets RX and TX 65-127 Octets................ 3225823472
Packets RX and TX 128-255 Octets............... 900048265
Packets RX and TX 256-511 Octets............... 931452081
Packets RX and TX 512-1023 Octets.............. 236045021
Packets RX and TX 1024-1518 Octets............. 571934636
Packets RX and TX 1519-2047 Octets............. 3089
Packets RX and TX 2048-4095 Octets............. 10130
Packets RX and TX 4096-9216 Octets............. 45523
Total Packets Received Without Errors.......... 2916369418
Unicast Packets Received....................... 2916211515
Multicast Packets Received..................... 58
Broadcast Packets Received..................... 157845
Receive Packets Discarded...................... 21
Total Packets Received with MAC Errors......... 0
Jabbers Received............................... 0
Fragments/Undersize Received................... 0
Alignment Errors............................... 0
FCS Errors..................................... 0
Overruns....................................... 0
Total Received Packets Not Forwarded........... 37557
802.3x Pause Frames Received................... 37536
Unacceptable Frame Type........................ 21
Total Packets Transmitted (Octets)............. 1573471772584
Packets Transmitted 64 Octets.................. 663715
Packets Transmitted 65-127 Octets.............. 1837959911
Packets Transmitted 128-255 Octets............. 482052879
Packets Transmitted 256-511 Octets............. 337690302
Packets Transmitted 512-1023 Octets............ 169121330
Packets Transmitted 1024-1518 Octets........... 548221110
Packets Transmitted > 1518 Octets.............. 211957517
Max Frame Size................................. 9216
Total Packets Transmitted Successfully......... 3587725506
Unicast Packets Transmitted.................... 3555702876
Multicast Packets Transmitted.................. 16918888
Broadcast Packets Transmitted.................. 15103742
Transmit Packets Discarded..................... 38475257
Total Transmit Errors.......................... 0
Total Transmit Packets Discarded............... 38475257
Single Collision Frames........................ 0
Multiple Collision Frames...................... 0
Excessive Collision Frames..................... 0
802.3x Pause Frames Transmitted................ 0
GVRP PDUs received............................. 0
GVRP PDUs Transmitted.......................... 0
GVRP Failed Registrations...................... 0
GMRP PDUs Received............................. 0
GMRP PDUs Transmitted.......................... 0
GMRP Failed Registrations...................... 0
BPDU: sent 555130, received 0
EAPOL Frames Transmitted....................... 0
EAPOL Start Frames Received.................... 0
Time since counters last cleared............... 16 day 19 hr 58 min 40 sec
It's pretty much all ports that have iscsi traffic, including the lags. There are multiple vlans on the ports, and only some of the vlans carry jumbo packets. It's mostly Dell twinax cables with integrated SFP+ transceivers, but also forty gigabit QSFP+ for 4032F to 4032F is showing problems. Most cables are 5M or less. Seems unlikely it's a cable problem when it's pretty much every port. I am not certain the switches are actually dropping the packets, it seems like it is only reporting it as doing so... just that these stats seem to indicate a problem as they continue to increase...
Received packets dropped > MTU: ............... 425952374
Transmitted packets dropped > MTU: ............ 211808792
DELL-Josh Cr
Community Manager
•
9777 Posts
•
44928 Points
7204
0
Posted June 30th, 2014 13:00
If it is the cables to the equallogic and they are all trying to access it, it could show dropped packets on multiple ports. How long are the twinax cables? They shouldn’t be longer than 7m. If you ping server to server does it drop pings? Try doing a show memory cpu and show process cpu and make sure all of the resources are not being used up and causing packet loss.
No cables longer than 7M. Pinging server to server shows no loss and verified it is doing jumbo packets on pings server to server (can't as easily on iscsi, also can't verify if dropped counts are going up faster as result of ping traffic or not.) Maybe I can evacuate the iscsi vms off one of the hosts to tell better if server to server shows as dropped packets or not.
iSCSI volumes still on one of the servers, but no traffic so would expect number of jumbo packets to be minimal. The number of reported packets dropped > MTU keeps increase fairly rapidly, and sending out lots of jumbo packets with ping doesn't appear to impact the rate..
I am starting to think the stats are just broken on 6.1.0.1, but it switches them correctly??? Cleared the stats and in a few minutes, get the following:
Note the 0 Octets on the RX and TX from 1519-9216 is all listed as 0, but packets Recived >1518 octets is 227125, and the packets Trasmitted > 1518 octets is 83048.
console#show statistics tengigabitethernet 1/0/5
Total Packets Received (Octets)................ 483483476
Packets Received 64 Octets..................... 0
Packets Received 65-127 Octets................. 518507
Packets Received 128-255 Octets................ 116271
Packets Received 256-511 Octets................ 156695
Packets Received 512-1023 Octets............... 21828
Packets Received 1024-1518 Octets.............. 7398
Packets Received > 1518 Octets................. 227125
Packets RX and TX 64 Octets.................... 170
Packets RX and TX 65-127 Octets................ 1156380
Packets RX and TX 128-255 Octets............... 186112
Packets RX and TX 256-511 Octets............... 264397
Packets RX and TX 512-1023 Octets.............. 90339
Packets RX and TX 1024-1518 Octets............. 264531
Packets RX and TX 1519-2047 Octets............. 0
Packets RX and TX 2048-4095 Octets............. 0
Packets RX and TX 4096-9216 Octets............. 0
Total Packets Received Without Errors.......... 1047824
Unicast Packets Received....................... 1047821
Multicast Packets Received..................... 0
Broadcast Packets Received..................... 3
Receive Packets Discarded...................... 0
Total Packets Received with MAC Errors......... 0
Jabbers Received............................... 0
Fragments/Undersize Received................... 0
Alignment Errors............................... 0
FCS Errors..................................... 0
Overruns....................................... 0
Total Received Packets Not Forwarded........... 0
802.3x Pause Frames Received................... 0
Unacceptable Frame Type........................ 0
Total Packets Transmitted (Octets)............. 645713375
Packets Transmitted 64 Octets.................. 170
Packets Transmitted 65-127 Octets.............. 637873
Packets Transmitted 128-255 Octets............. 69841
Packets Transmitted 256-511 Octets............. 107702
Packets Transmitted 512-1023 Octets............ 68511
Packets Transmitted 1024-1518 Octets........... 257133
Packets Transmitted > 1518 Octets.............. 83048
Max Frame Size................................. 9216
Total Packets Transmitted Successfully......... 1224279
Unicast Packets Transmitted.................... 1213540
Multicast Packets Transmitted.................. 5239
Broadcast Packets Transmitted.................. 5500
Transmit Packets Discarded..................... 0
Total Transmit Errors.......................... 0
Total Transmit Packets Discarded............... 0
Single Collision Frames........................ 0
Multiple Collision Frames...................... 0
Excessive Collision Frames..................... 0
802.3x Pause Frames Transmitted................ 0
GVRP PDUs received............................. 0
GVRP PDUs Transmitted.......................... 0
GVRP Failed Registrations...................... 0
GMRP PDUs Received............................. 0
GMRP PDUs Transmitted.......................... 0
GMRP Failed Registrations...................... 0
BPDU: sent 157, received 0
EAPOL Frames Transmitted....................... 0
EAPOL Start Frames Received.................... 0
Time since counters last cleared............... 0 day 0 hr 5 min 16 sec
DELL-Josh Cr
Community Manager
•
9777 Posts
•
44928 Points
7230
0
Posted July 1st, 2014 08:00
Are you seeing any performance related issues because of this?
I am not seeing any performance related issues. For testing, I was able to exceed over 5gb/sec between a VM on one server and a VM on another server.
It appears to be the stats are just wrong. It will be more of an issue when there is a problem and I need to diagnose what the cause is.
DELL-Josh Cr
Community Manager
•
9777 Posts
•
44928 Points
7230
0
Posted July 1st, 2014 13:00
You can try clearing the counters and seeing if there is less packet drop during times when there is less iSCSI traffic. It does look like it is sending pause frames, which would indicate that it is dropping packets and not a false positive. It might be reaching the limit that the MD3220i can send over the network.
I cleared all the counters, and receive relatively few pause packets (many ports no pauses). Would be good if someone who has jumbo packets on vlans can confirm if they are seeing the same thing, or not seeing the same counted dropped packets (running latest 6.1.0.1)
DELL-Josh Cr
Community Manager
•
9777 Posts
•
44928 Points
7204
0
Posted June 30th, 2014 11:00
Hi jlauro,
Is it just that one port that is dropping packets or all of them? Is the switch only doing iscsi traffic or are there other VLANs carrying network traffic? What type of SFP+ transceivers and cables are you using? Is it a long cable run? It sounds like it could be something with the hosts communicating with the MD3220i that is the problem. I don’t think it is because of jumbo frames, unless if you turn off jumbo frames the packet dropping stops.