Dell N4032F excessive amount of transmit discards

Here is output from one pair of those ports

fr-n4032f-01#show running-config interface tengigabitethernet 1/0/1

channel-group 10 mode active
description "PO10-FR-CH01"
switchport mode trunk
switchport trunk allowed vlan 10,19,22,176

fr-n4032f-01#show running-config interface tengigabitethernet 2/0/1

channel-group 10 mode active
description "PO10-FR-CH01"
switchport mode trunk
switchport trunk allowed vlan 10,19,22,176

No specific log messages were generated, network infrastructure didn't change since installation.

Yes on 6.3.0.15 we had several unexpected stack crash and mac address wasn't populated correctly for certain vlans. Btw, I asked this question here, but nobody from Dell responded there.

I wasn't able to check other firmware images, since this is production and I don't have enough time for experiments.

Can you show us the configuration for the ports showing these discards? Are any messages seen in the logs that may be pertinent to the discarded packets.

On 6.3.2.7 the issue is seen? On 6.3.0.15, the issue is not seen, but other issues are present? Have you tried any other firmware revisions?

Are these transmit discards occurring consistently? or intermittent? Can you show us the counters of these port-channels? the counters will help look other packet issues. When comparing total transmit packets with transmit discard, what is the percentage of drops?

After reboot switch could work without any issue about 8 hours, after discards are starting to grow, what I did last time (approximately one hour ago). 

I shut te1/0/1, te1/0/2 and te1/0/8 those ports are members of port-channels, rebooted switch (it was master in stack), added them back to port-channel and turned off ports te2/0/1, te2/0/2 and te2/0/8.

fr-n4032f-01#show interfaces tengigabitethernet 1/0/1

Interface Name................................. Te1/0/1
SOC Hardware Info.............................. BCM56842_A1
Link Status.................................... Up /None
Keepalive Enabled.............................. FALSE
Err-disable Cause.............................. None
MTU Size....................................... 9000
Port Mode [Duplex]............................. Full
Port Speed..................................... 10000
Link Debounce Flaps............................ 0
Auto-Negotation Status......................... Off
Burned MAC Address............................. F8B1.5695.C854
L3 MAC Address................................. F8B1.5695.C855
Sample Load Interval........................... 300
Received Input Rate Bits/Sec................... 400956184
Received Input Rate Packets/Sec................ 219285
Transmitted Input Rate Bits/Sec................ 403578048
Transmitted Input Rate Packets/Sec : .......... 221527
Total Packets Received Without Errors.......... 531554051
Unicast Packets Received....................... 531533783
Multicast Packets Received..................... 2306
Broadcast Packets Received..................... 17962
Total Packets Received with MAC Errors......... 24
Jabbers Received............................... 0
Fragments/Undersize Received................... 0
Alignment Errors............................... 0
FCS Errors..................................... 0
Overruns....................................... 0
Total Received Packets Not Forwarded........... 1602
Total Packets Transmitted Successfully......... 533630363
Unicast Packets Transmitted.................... 533607174
Multicast Packets Transmitted.................. 16248
Broadcast Packets Transmitted.................. 6941
Transmit Packets Discarded..................... 2
Total Transmit Errors.......................... 0
Total Transmit Packets Discarded............... 2
Single Collision Frames........................ 0
Multiple Collision Frames...................... 0
Excessive Collision Frames..................... 0

fr-n4032f-01#show interfaces tengigabitethernet 2/0/1

Interface Name................................. Te2/0/1
SOC Hardware Info.............................. BCM56842_A1
Link Status.................................... shut /None
Keepalive Enabled.............................. FALSE
Err-disable Cause.............................. None
MTU Size....................................... 9000
Port Mode [Duplex]............................. Full
Port Speed..................................... 10000
Link Debounce Flaps............................ 0
Auto-Negotation Status......................... Off
Burned MAC Address............................. F8B1.5695.C854
L3 MAC Address................................. F8B1.5695.C855
Sample Load Interval........................... 300
Received Input Rate Bits/Sec................... 14520
Received Input Rate Packets/Sec................ 1
Transmitted Input Rate Bits/Sec................ 16152
Transmitted Input Rate Packets/Sec : .......... 3
Total Packets Received Without Errors.......... 5720611746
Unicast Packets Received....................... 5720372692
Multicast Packets Received..................... 28134
Broadcast Packets Received..................... 210920
Total Packets Received with MAC Errors......... 210
Jabbers Received............................... 0
Fragments/Undersize Received................... 0
Alignment Errors............................... 0
FCS Errors..................................... 0
Overruns....................................... 0
Total Received Packets Not Forwarded........... 18908
Total Packets Transmitted Successfully......... 5772487538
Unicast Packets Transmitted.................... 5772213070
Multicast Packets Transmitted.................. 190920
Broadcast Packets Transmitted.................. 83548
Transmit Packets Discarded..................... 826475
Total Transmit Errors.......................... 0
Total Transmit Packets Discarded............... 826475
Single Collision Frames........................ 0
Multiple Collision Frames...................... 0
Excessive Collision Frames..................... 0

fr-n4032f-01#show interfaces counters port-channel 10

Ch InTotalPkts InUcastPkts InMcastPkts InBcastPkts
--------- ---------------- ---------------- ---------------- ----------------
Po10 621033222 621010146 2621 20455

Ch OutTotalPkts OutUcastPkts OutMcastPkts OutBcastPkts
--------- ---------------- ---------------- ---------------- ----------------
Po10 624129886 624102611 18526 8749

Alignment Errors: ............................. 0
FCS Errors: ................................... 0
Single Collision Frames: ...................... 0
Multiple Collision Frames: .................... 0
Late Collisions: .............................. 0
Excessive Collisions: ......................... 0
Received packets dropped > MTU: ............... 26
Transmitted oversized packets: ................ 2754780
Internal MAC Rx Errors: ....................... 26
Received Pause Frames: ........................ 0
Transmitted Pause Frames: ..................... 0

Also my management pushing me with roll back, so I'm going to rollback on previous firmware within 2 hours.

I do not see anything in this output to indicate why the discards.

Will you have enough downtime in your rollback windows to try a different 6.3 firmware?

http://dell.to/2sR6CH7

Daniel, I'll try to talk with my manager about alternative firmware, but I don't think that it'll happen during planned maintenance. Probably next planned maintenance with additional verification.

We did roll back successfully, but we tried to clarify that we are good and applied traffic load (tcp around 10 Gig) between two chassis connected to this stack and it CRASHED!!! Just crashed and stop to respond on any requests

Those chassis were connected using 20Gig  port-channels.

Likely we're going to replace this switch and never use this series.

Just for information. Roll back also didn't help and discards occurred again.

We found that root cause of this problem was monitor session configured 2 days after software upgrade. For now without span port it looks stable on the regular load.

Looks like there is an issue with span setup for those switches. Any answers from Dell team would be much appreciated