Unsolved
This post is more than 5 years old
7 Posts
0
169803
Just installed Solarwinds NPM - showing high receive discards PC6224 stack
Showing high receive discards with some interfaces at over 1 million discards per hour. This has probably been going on for some time but just now got visibility into the issue. The ports are for iSCSI traffic back to PS4100 SAN. Discards in solarwinds do correlate to what is in switch. Any help would b greatly appreciated.
bdorr1
7 Posts
0
May 21st, 2014 10:00
Update to post....when I show statistics on one of the interfaces the discards correlate directly to 802.3x pause frames rcvd:
ps4100SwitchStack#show statistics ethernet 1/g08
Total Packets Received (Octets)................ 361189052
Packets Received 64 Octets..................... 4825105
Packets Received 65-127 Octets................. 91732
Packets Received 128-255 Octets................ 4080
Packets Received 256-511 Octets................ 724
Packets Received 512-1023 Octets............... 309
Packets Received 1024-1518 Octets.............. 20
Packets Received > 1518 Octets................. 5551
Packets RX and TX 64 Octets.................... 4834408
Packets RX and TX 65-127 Octets................ 98356
Packets RX and TX 128-255 Octets............... 4596
Packets RX and TX 256-511 Octets............... 3042
Packets RX and TX 512-1023 Octets.............. 346
Packets RX and TX 1024-1518 Octets............. 1165
Packets RX and TX 1519-2047 Octets............. 1143
Packets RX and TX 2048-4095 Octets............. 399430
Packets RX and TX 4096-9216 Octets............. 2832893
Total Packets Received Without Errors.......... 4927521
Unicast Packets Received....................... 1708665
Multicast Packets Received..................... 3218658
Broadcast Packets Received..................... 198
--More-- or (q)uit
Receive Packets Discarded...................... 3218642
Total Packets Received with MAC Errors......... 0
Jabbers Received............................... 0
Fragments/Undersize Received................... 0
Alignment Errors............................... 0
FCS Errors..................................... 0
Overruns....................................... 0
802.3x Pause Frames Received................... 3218642
Unacceptable Frame Type........................ 0
Total Packets Transmitted (Octets)............. 26647472775
Packets Transmitted 64 Octets.................. 9303
Packets Transmitted 65-127 Octets.............. 6624
Packets Transmitted 128-255 Octets............. 516
Packets Transmitted 256-511 Octets............. 2318
Packets Transmitted 512-1023 Octets............ 37
Packets Transmitted 1024-1518 Octets........... 1145
Packets Transmitted > 1518 Octets ............. 3227915
Max Frame Size................................. 9216
Total Packets Transmitted Successfully......... 3247858
Unicast Packets Transmitted.................... 3243989
--More-- or (q)uit
Multicast Packets Transmitted.................. 1782
Broadcast Packets Transmitted.................. 2087
Transmit Packets Discarded..................... 0
Total Transmit Packets Discarded............... 0
Single Collision Frames........................ 0
Multiple Collision Frames...................... 0
Excessive Collision Frames..................... 0
802.3x Pause Frames Transmitted................ 0
GVRP PDUs received............................. 0
GVRP PDUs Transmitted.......................... 0
GVRP Failed Registrations...................... 0
BPDU: sent 1188, received 0
EAPOL Frames Transmitted....................... 0
EAPOL Start Frames Received.................... 0
Time Since Counters Last Cleared............... 0 day 0 hr 39 min 36 sec
**Flow control is supposed to be enabled according to Dell's whitepaper 'iSCSI optimization for dell powerconnect switches'.
bdorr1
7 Posts
0
May 21st, 2014 11:00
as an added note I notice that the number of discards on the interfaces correlate directly to the number of 802.3x pause frames rcvd. Anyone else encountered this? According to Dell's guide on iSCSI optimization between switch and storage flow control is required.
Anonymous
5 Practitioner
5 Practitioner
•
274.2K Posts
0
May 21st, 2014 12:00
one of the first things i would look at are the firmware on the devices. The latest for the 6200 switches is 3.3.10.3
www.dell.com/.../drivers
what are the port settings for the ports you are seeing the discards?
And then also the output from [tag:show] interface counters, for the specific interfaces.
Thanks
bdorr1
7 Posts
0
May 21st, 2014 14:00
Currently running FW 3.3.8.2 and plan to update to 3.3.10.3 next week..
Port settings are ( no storm control unicast, spanning-tree portfast, mtu 9216, flow control enable, GB Full)
Here are the counters from an interface - notice how rcv discards are equal to 802.3x pause frames rcvd:
ps4100SwitchStack#show statistics ethernet 1/g08
Total Packets Received (Octets)................ 361189052
Packets Received 64 Octets..................... 4825105
Packets Received 65-127 Octets................. 91732
Packets Received 128-255 Octets................ 4080
Packets Received 256-511 Octets................ 724
Packets Received 512-1023 Octets............... 309
Packets Received 1024-1518 Octets.............. 20
Packets Received > 1518 Octets................. 5551
Packets RX and TX 64 Octets.................... 4834408
Packets RX and TX 65-127 Octets................ 98356
Packets RX and TX 128-255 Octets............... 4596
Packets RX and TX 256-511 Octets............... 3042
Packets RX and TX 512-1023 Octets.............. 346
Packets RX and TX 1024-1518 Octets............. 1165
Packets RX and TX 1519-2047 Octets............. 1143
Packets RX and TX 2048-4095 Octets............. 399430
Packets RX and TX 4096-9216 Octets............. 2832893
Total Packets Received Without Errors.......... 4927521
Unicast Packets Received....................... 1708665
Multicast Packets Received..................... 3218658
Broadcast Packets Received..................... 198
--More-- or (q)uit
Receive Packets Discarded...................... 3218642
Total Packets Received with MAC Errors......... 0
Jabbers Received............................... 0
Fragments/Undersize Received................... 0
Alignment Errors............................... 0
FCS Errors..................................... 0
Overruns....................................... 0
802.3x Pause Frames Received................... 3218642
Unacceptable Frame Type........................ 0
Total Packets Transmitted (Octets)............. 26647472775
Packets Transmitted 64 Octets.................. 9303
Packets Transmitted 65-127 Octets.............. 6624
Packets Transmitted 128-255 Octets............. 516
Packets Transmitted 256-511 Octets............. 2318
Packets Transmitted 512-1023 Octets............ 37
Packets Transmitted 1024-1518 Octets........... 1145
Packets Transmitted > 1518 Octets ............. 3227915
Max Frame Size................................. 9216
Total Packets Transmitted Successfully......... 3247858
Unicast Packets Transmitted.................... 3243989
--More-- or (q)uit
Multicast Packets Transmitted.................. 1782
Broadcast Packets Transmitted.................. 2087
Transmit Packets Discarded..................... 0
Total Transmit Packets Discarded............... 0
Single Collision Frames........................ 0
Multiple Collision Frames...................... 0
Excessive Collision Frames..................... 0
802.3x Pause Frames Transmitted................ 0
GVRP PDUs received............................. 0
GVRP PDUs Transmitted.......................... 0
GVRP Failed Registrations...................... 0
BPDU: sent 1188, received 0
EAPOL Frames Transmitted....................... 0
EAPOL Start Frames Received.................... 0
Time Since Counters Last Cleared............... 0 day 0 hr 39 min 36 sec
bdorr1
7 Posts
0
May 21st, 2014 15:00
Running FW 3.3.8.2 currently but plan to update to latest next week.
What is interesting that I didn't catch when I initially posted is that the receive discards matched exactly to the 802.3 pause frames rcvd...This is flow control from what I understand but looks to be required when following the Dell whitepaper on iSCSI optimization between switch and storage.
Anonymous
5 Practitioner
5 Practitioner
•
274.2K Posts
0
May 22nd, 2014 07:00
Thanks for this information, sorry i didn't see your previous posts with that info. the switch does show to be receiving pause frames. Which means the PS4100 is the device sending them. I would check and see if there are updates for the PS4100 as well. From my understanding Equalogic recommends flow control to be on. Let us know if the updates help at all.
bdorr1
7 Posts
0
May 22nd, 2014 11:00
You know what else is interesting - the issue appears to only be on switch ports with Broadcom (5709C) NICs attached. I updated the NIC firmware and drivers on one of my hosts but that doesn't resolve the issue. Have you heard of anything like this regarding Broadcom NICs?
Anonymous
5 Practitioner
5 Practitioner
•
274.2K Posts
0
May 22nd, 2014 12:00
I have not heard of it before. When you get the switch firmware let me know if makes any difference. I will see if any of the storage techs have seen this before.
shawn_b1
15 Posts
0
May 23rd, 2014 10:00
I have several PCT6200 switches on my network.
All are running firmware 3.3.4.1 and several have high discard counters
Port 1/0/48 · Uplink_Core_1 0 errors 516,207 discards 0 errors 0 discards
this is prob the highest that I have seen today
I think this is due to the limited memory that the switches have.
I have replaced some that had very high discards with PCT7048 and they have been off my Solarwinds charts
Anonymous
5 Practitioner
5 Practitioner
•
274.2K Posts
0
May 23rd, 2014 11:00
Interesting. Did your discards correlate with high received pause frames from other devices too?
bdorr1
7 Posts
0
May 30th, 2014 09:00
Updated switch firmware to 3.3.10.3 and discards still exist and equal to 802.3x pause frames:
Receive Packets Discarded...................... 16394610
Total Packets Received with MAC Errors......... 0
Jabbers Received............................... 0
Fragments/Undersize Received................... 0
Alignment Errors............................... 0
FCS Errors..................................... 0
Overruns....................................... 0
802.3x Pause Frames Received................... 16394610
Unacceptable Frame Type........................ 0
shawn_b1
15 Posts
0
June 2nd, 2014 08:00
"Updated switch firmware to 3.3.10.3 and discards still exist"
I do not think that that firmware will resolve the discards situation you are experiencing.
PCT6248 is an old switch with a small amount of memory available for it to perform its "switching" tasks.
It is a great switch and I wish DELL would still sell them
I went through several versions of firmware form 2.0 to latest 3.3 and then only thing that stopped my discards was traffic shaping, the reduce the amount of traffic that the switches received, OR upgrade to PCT7048 (another switch that I wish DELL still sold)
Anonymous
5 Practitioner
5 Practitioner
•
274.2K Posts
0
June 2nd, 2014 10:00
Shawn, thank you for sharing your findings, This is very helpful information to have. By chance to you recall if ports with the pause frames were connected to Broadcom NICs as well? What bandwidth did you set on your traffic shaping?
bdorr, i would try implementing traffic shaping on the port as Shawn has suggested. And see if it reduces the pause frames.
Example
The following example displays the setting of traffic-shape to a maximum
bandwidth of 1024 Kpbs.
console(config-if-1/g1)#traffic-shape 1024 kbps