Unsolved
This post is more than 5 years old
11 Posts
0
1356
May 6th, 2020 08:00
Dell MXL-10/40Gbe Dropping Packets
We have a Dell Chassis 1000e with 16 M60 blades. We also have two Dell Force MXL-10/40GbE installed with 4-port SFP+ module installed in each switch. The two MXL-10/40GbE switches are stacked. We are also using Port channels consisting of both Te0/52 and Te1/51 ports which are connected to Cisco Nexus 5k switch. We also have ports Te0/50 and Te1/50 connecting to Cisco 5k switch for iscsi connections.
The issue with this Dell Chassis is that when i run the following command:
sh hardware drops
Im seeing that all blade ports from Te 0/1 -16, Te 1/1-16 and Te0/52, Te1/52 and Te0/50, Te1/50 are dropping packets. Here are some samples:
Blade:
Drops in Interface Te 0/1:
--- Ingress Drops ---
Ingress Drops : 332670
IBP CBP Full Drops : 0
PortSTPnotFwd Drops : 0
IPv4 L3 Discards : 0
Policy Discards : 0
Packets dropped by FP : 332670
(L2+L3) Drops : 0
Port bitmap zero Drops : 0
Rx VLAN Drops : 96908
iscsi:
Drops in Interface Te 0/50:
--- Ingress Drops ---
Ingress Drops : 190975
IBP CBP Full Drops : 0
PortSTPnotFwd Drops : 2564
Port-Channel Uplink Member:
Drops in Interface Te 0/52:
--- Ingress Drops ---
Ingress Drops : 15272129
IBP CBP Full Drops : 0
PortSTPnotFwd Drops : 20
IPv4 L3 Discards : 0
Policy Discards : 145
Packets dropped by FP : 15216021
(L2+L3) Drops : 0
Port bitmap zero Drops : 8473414
Rx VLAN Drops : 8417141
IPv4 L3 Discards : 0
Policy Discards : 1
Packets dropped by FP : 166516
(L2+L3) Drops : 0
Port bitmap zero Drops : 87517
Rx VLAN Drops : 163925
I did some reading about this issue and noticed that buffers could be at play over here. But i never dealt with that before and would like to find out if anyone has any suggestions as to where to look for the problem and to fix it?
I have also attached my Dell Force MXL-10/40GbE configuration.
Any help is greatly appreciated, Thank You
MXL Running Configuration:
! Version 9.14(1.9P1)
! Last configuration change at Wed May 6 09:20:48 2020 by root
! Startup-config last updated at Wed May 6 09:21:25 2020 by root
!
boot system stack-unit 0 primary system://A
boot system stack-unit 0 secondary system://B
boot system stack-unit 1 primary system://A
boot system stack-unit 1 secondary system://B
!
logging buffered 40960 informational
!
hostname ts-tor-pn-chassis5-sw
!
protocol lldp
!
redundancy auto-synchronize full
redundancy protocol lacp
redundancy protocol xstp
!
username root password 7 d7acc8a1dcd4f6983hdfs privilege 15 role sysadmin
!
default-vlan disable
!
protocol spanning-tree rstp
no disable
!
stack-unit 0 provision MXL-10/40GbE
!
stack-unit 1 provision MXL-10/40GbE
!
stack-unit 0 port 41 portmode quad
!
stack-unit 0 port 45 portmode quad
!
stack-unit 0 stack-group 1
!
interface TenGigabitEthernet 0/1
description Dell Blades
no ip address
mtu 9216
portmode hybrid
switchport
spanning-tree rstp edge-port
no shutdown
!
interface TenGigabitEthernet 0/2
description Dell Blades
no ip address
mtu 9216
portmode hybrid
switchport
spanning-tree rstp edge-port
no shutdown
!
interface TenGigabitEthernet 0/3
description Dell Blades
no ip address
mtu 9216
portmode hybrid
switchport
spanning-tree rstp edge-port
no shutdown
!
interface TenGigabitEthernet 0/4
description Dell Blades
no ip address
mtu 9216
portmode hybrid
switchport
spanning-tree rstp edge-port
no shutdown
!
interface TenGigabitEthernet 0/5
description Dell Blades
no ip address
mtu 9216
portmode hybrid
switchport
spanning-tree rstp edge-port
no shutdown
!
interface TenGigabitEthernet 0/6
description Dell Blades
no ip address
mtu 9216
portmode hybrid
switchport
spanning-tree rstp edge-port
no shutdown
!
interface TenGigabitEthernet 0/7
description Dell Blades
no ip address
mtu 9216
portmode hybrid
switchport
spanning-tree rstp edge-port
no shutdown
!
interface TenGigabitEthernet 0/8
description Dell Blades
no ip address
mtu 9216
portmode hybrid
switchport
spanning-tree rstp edge-port
no shutdown
!
interface TenGigabitEthernet 0/9
description Dell Blades
no ip address
mtu 9216
portmode hybrid
switchport
spanning-tree rstp edge-port
no shutdown
!
interface TenGigabitEthernet 0/10
description Dell Blades
no ip address
mtu 9216
portmode hybrid
switchport
spanning-tree rstp edge-port
no shutdown
!
interface TenGigabitEthernet 0/11
description Dell Blades
no ip address
mtu 9216
portmode hybrid
switchport
spanning-tree rstp edge-port
no shutdown
!
interface TenGigabitEthernet 0/12
description Dell Blades
no ip address
mtu 9216
portmode hybrid
switchport
spanning-tree rstp edge-port
no shutdown
!
interface TenGigabitEthernet 0/13
description Dell Blades
no ip address
mtu 9216
portmode hybrid
switchport
spanning-tree rstp edge-port
no shutdown
!
interface TenGigabitEthernet 0/14
description Dell Blades
no ip address
mtu 9216
portmode hybrid
switchport
spanning-tree rstp edge-port
no shutdown
!
interface TenGigabitEthernet 0/15
description Dell Blades
no ip address
mtu 9216
portmode hybrid
switchport
spanning-tree rstp edge-port
no shutdown
!
interface TenGigabitEthernet 0/16
description Dell Blades
no ip address
mtu 9216
portmode hybrid
switchport
spanning-tree rstp edge-port
no shutdown
!
interface TenGigabitEthernet 0/17
no ip address
portmode hybrid
switchport
no shutdown
!
interface TenGigabitEthernet 0/18
no ip address
portmode hybrid
switchport
no shutdown
!
interface TenGigabitEthernet 0/19
no ip address
portmode hybrid
switchport
no shutdown
!
interface TenGigabitEthernet 0/20
no ip address
portmode hybrid
switchport
no shutdown
!
interface TenGigabitEthernet 0/21
no ip address
portmode hybrid
switchport
no shutdown
!
interface TenGigabitEthernet 0/22
no ip address
portmode hybrid
switchport
no shutdown
!
interface TenGigabitEthernet 0/23
no ip address
portmode hybrid
switchport
no shutdown
!
interface TenGigabitEthernet 0/24
no ip address
portmode hybrid
switchport
no shutdown
!
interface TenGigabitEthernet 0/25
no ip address
portmode hybrid
switchport
no shutdown
!
interface TenGigabitEthernet 0/26
no ip address
portmode hybrid
switchport
no shutdown
!
interface TenGigabitEthernet 0/27
no ip address
portmode hybrid
switchport
no shutdown
!
interface TenGigabitEthernet 0/28
no ip address
portmode hybrid
switchport
no shutdown
!
interface TenGigabitEthernet 0/29
no ip address
portmode hybrid
switchport
no shutdown
!
interface TenGigabitEthernet 0/30
no ip address
portmode hybrid
switchport
no shutdown
!
interface TenGigabitEthernet 0/31
no ip address
portmode hybrid
switchport
no shutdown
!
interface TenGigabitEthernet 0/32
no ip address
portmode hybrid
switchport
no shutdown
!
interface fortyGigE 0/33
no ip address
shutdown
!
interface TenGigabitEthernet 0/41
no ip address
shutdown
!
interface TenGigabitEthernet 0/42
no ip address
shutdown
!
interface TenGigabitEthernet 0/43
no ip address
shutdown
!
interface TenGigabitEthernet 0/44
no ip address
shutdown
!
interface TenGigabitEthernet 0/45
no ip address
shutdown
!
interface TenGigabitEthernet 0/46
no ip address
shutdown
!
interface TenGigabitEthernet 0/47
no ip address
shutdown
!
interface TenGigabitEthernet 0/48
no ip address
shutdown
!
interface TenGigabitEthernet 0/49
no ip address
mtu 9216
no shutdown
!
interface TenGigabitEthernet 0/50
description nexus01 e1/15
no ip address
mtu 9216
portmode hybrid
switchport
no shutdown
!
interface TenGigabitEthernet 0/51
no ip address
mtu 9216
no shutdown
!
interface TenGigabitEthernet 0/52
description nexus01 e1/4
no ip address
mtu 9216
!
port-channel-protocol LACP
port-channel 55 mode active
no shutdown
!
stack-unit 1 port 41 portmode quad
!
stack-unit 1 port 45 portmode quad
!
stack-unit 1 stack-group 1
!
interface TenGigabitEthernet 1/1
description Dell Blades
no ip address
mtu 9216
portmode hybrid
switchport
spanning-tree rstp edge-port
no shutdown
!
interface TenGigabitEthernet 1/2
description Dell Blades
no ip address
mtu 9216
portmode hybrid
switchport
spanning-tree rstp edge-port
no shutdown
!
interface TenGigabitEthernet 1/3
description Dell Blades
no ip address
mtu 9216
portmode hybrid
switchport
spanning-tree rstp edge-port
no shutdown
!
interface TenGigabitEthernet 1/4
description Dell Blades
no ip address
mtu 9216
portmode hybrid
switchport
spanning-tree rstp edge-port
no shutdown
!
interface TenGigabitEthernet 1/5
description Dell Blades
no ip address
mtu 9216
portmode hybrid
switchport
spanning-tree rstp edge-port
no shutdown
!
interface TenGigabitEthernet 1/6
description Dell Blades
no ip address
mtu 9216
portmode hybrid
switchport
spanning-tree rstp edge-port
no shutdown
!
interface TenGigabitEthernet 1/7
description Dell Blades
no ip address
mtu 9216
portmode hybrid
switchport
spanning-tree rstp edge-port
no shutdown
!
interface TenGigabitEthernet 1/8
description Dell Blades
no ip address
mtu 9216
portmode hybrid
switchport
spanning-tree rstp edge-port
no shutdown
!
interface TenGigabitEthernet 1/9
description Dell Blades
no ip address
mtu 9216
portmode hybrid
switchport
spanning-tree rstp edge-port
no shutdown
!
interface TenGigabitEthernet 1/10
description Dell Blades
no ip address
mtu 9216
portmode hybrid
switchport
spanning-tree rstp edge-port
no shutdown
!
interface TenGigabitEthernet 1/11
description Dell Blades
no ip address
mtu 9216
portmode hybrid
switchport
spanning-tree rstp edge-port
no shutdown
!
interface TenGigabitEthernet 1/12
description Dell Blades
no ip address
mtu 9216
portmode hybrid
switchport
spanning-tree rstp edge-port
no shutdown
!
interface TenGigabitEthernet 1/13
description Dell Blades
no ip address
mtu 9216
portmode hybrid
switchport
spanning-tree rstp edge-port
no shutdown
!
interface TenGigabitEthernet 1/14
description Dell Blades
no ip address
mtu 9216
portmode hybrid
switchport
spanning-tree rstp edge-port
no shutdown
!
interface TenGigabitEthernet 1/15
description Dell Blades
no ip address
mtu 9216
portmode hybrid
switchport
spanning-tree rstp edge-port
no shutdown
!
interface TenGigabitEthernet 1/16
description Dell Blades
no ip address
mtu 9216
portmode hybrid
switchport
spanning-tree rstp edge-port
no shutdown
!
interface TenGigabitEthernet 1/17
no ip address
portmode hybrid
switchport
shutdown
!
interface TenGigabitEthernet 1/18
no ip address
portmode hybrid
switchport
shutdown
!
interface TenGigabitEthernet 1/19
no ip address
portmode hybrid
switchport
shutdown
!
interface TenGigabitEthernet 1/20
no ip address
portmode hybrid
switchport
shutdown
!
interface TenGigabitEthernet 1/21
no ip address
portmode hybrid
switchport
shutdown
!
interface TenGigabitEthernet 1/22
no ip address
portmode hybrid
switchport
shutdown
!
interface TenGigabitEthernet 1/23
no ip address
portmode hybrid
switchport
shutdown
!
interface TenGigabitEthernet 1/24
no ip address
portmode hybrid
switchport
shutdown
!
interface TenGigabitEthernet 1/25
no ip address
portmode hybrid
switchport
shutdown
!
interface TenGigabitEthernet 1/26
no ip address
portmode hybrid
switchport
shutdown
!
interface TenGigabitEthernet 1/27
no ip address
portmode hybrid
switchport
shutdown
!
interface TenGigabitEthernet 1/28
no ip address
portmode hybrid
switchport
shutdown
!
interface TenGigabitEthernet 1/29
no ip address
portmode hybrid
switchport
shutdown
!
interface TenGigabitEthernet 1/30
no ip address
portmode hybrid
switchport
shutdown
!
interface TenGigabitEthernet 1/31
no ip address
portmode hybrid
switchport
shutdown
!
interface TenGigabitEthernet 1/32
no ip address
portmode hybrid
switchport
shutdown
!
interface fortyGigE 1/33
no ip address
no shutdown
!
interface TenGigabitEthernet 1/41
no ip address
no shutdown
!
interface TenGigabitEthernet 1/42
no ip address
no shutdown
!
interface TenGigabitEthernet 1/43
no ip address
no shutdown
!
interface TenGigabitEthernet 1/44
no ip address
no shutdown
!
interface TenGigabitEthernet 1/45
no ip address
no shutdown
!
interface TenGigabitEthernet 1/46
no ip address
no shutdown
!
interface TenGigabitEthernet 1/47
no ip address
no shutdown
!
interface TenGigabitEthernet 1/48
no ip address
no shutdown
!
interface TenGigabitEthernet 1/49
no ip address
mtu 9216
!
interface TenGigabitEthernet 1/50
description ts-tor-pn-nx-02 e1/15
no ip address
mtu 9216
switchport
no shutdown
!
interface TenGigabitEthernet 1/51
no ip address
mtu 9216
no shutdown
!
interface TenGigabitEthernet 1/52
description ts-tor-pn-nx-02 e1/4
no ip address
mtu 9216
!
port-channel-protocol LACP
port-channel 55 mode active
no shutdown
!
interface ManagementEthernet 0/0
ip address 172.26.104.59/22
no shutdown
!
interface Port-channel 55
description Uplink to the Nexus
no ip address
mtu 9216
switchport
no shutdown
!
interface Vlan 1
ip address dhcp
shutdown
!
interface Vlan 101
description OPS Prod
no ip address
tagged Port-channel 55
no shutdown
!
interface Vlan 111
description Hypervisors
no ip address
tagged Port-channel 55
no shutdown
!
interface Vlan 2008
description Production
no ip address
tagged Port-channel 55
untagged TenGigabitEthernet 0/1-16
untagged TenGigabitEthernet 1/1-16
no shutdown
!
interface Vlan 2088
description Testing
no ip address
tagged Port-channel 55
no shutdown
!
interface Vlan 3254
description iSCSI-A
no ip address
mtu 9216
tagged TenGigabitEthernet 0/50
no shutdown
!
interface Vlan 3255
description iSCSI-B
no ip address
mtu 9216
tagged TenGigabitEthernet 1/50
tagged Port-channel 55
no shutdown
!
management route 0.0.0.0/0 172.26.104.1
!
service-class dynamic dot1p
!
stack-unit 0 priority 12
stack-unit 1 priority 11
!
ntp server 172.26.104.2
ntp server 172.26.104.3
ntp source managementethernet 0/0
!
clock timezone EST -5
clock summer-time EDT recurring
!
no dcb enable
!
dcb-map FLEXIO_DCB_MAP_PFC_OFF
no pfc mode on
!
line console 0
line vty 0
line vty 1
line vty 2
line vty 3
line vty 4
line vty 5
line vty 6
line vty 7
line vty 8
line vty 9
!
reload-type
boot-type normal-reload
config-scr-download enable
!
end
0 events found
No Events found!
DELL-Josh Cr
Moderator
•
9.6K Posts
•
113 Points
0
May 6th, 2020 13:00
Hi,
Is the firmware up to date? Do the drops happen at certain times or under certain loads?
pawbe
11 Posts
0
May 7th, 2020 06:00
Hi Josh, the firmware is pretty recent:
Dell Application Software Version: 9.14(1.9P1)
The drops are happening all the time. We are also using Jumbo frames, could that be the source of the problem?
DELL-Josh Cr
Moderator
•
9.6K Posts
•
113 Points
0
May 8th, 2020 09:00
it is possible, you probably would want to use wireshark to get more information.
pawbe
11 Posts
0
May 11th, 2020 12:00
Hi Josh, to get the wireshark, that would require me to take a trip to the DC and set up some ports spannings on the Cisco switch where that chassis is connected etc... If it will need to come to that then i will do it...
Also, i made some progress, the following packets are no longer being dropped:
The issue was that the default-vlan was disabled.
The other packets drops are still persisting... im trying to get to the bottom of them, but not having too much luck so far.
pawbe
11 Posts
0
May 13th, 2020 11:00
So i stripped down the whole switch to default factory settings, no stack is configured, no uplinks to the nexus switch.... all factory defaults. I have only one blade running, the rest of the blades are disabled. Im still seeing packet drops, so this is somehow hardware related. Any ideas?
DELL-Josh Cr
Moderator
•
9.6K Posts
•
113 Points
0
May 13th, 2020 12:00
Is all of the firmware on the switch, the blade and the chassis up to date?
pawbe
11 Posts
0
May 13th, 2020 19:00
Hi Josh, yes i just finished up updating the switch, blades and chassis, all the latest firmware and drivers.... so like i mentioned before, i stripped down the switch to the default factory settings. There is no uplink to the main nexus switch at the moment. As soon as one of the blade comes up, i start to see these:
I am out of clues at the moment as to why this is happening, any ideas?
All help is appreciated.
Thanks
DELL-Josh Cr
Moderator
•
9.6K Posts
•
113 Points
0
May 14th, 2020 09:00
Are you able to mirror the port and use wireshark to monitor?