14 Posts
0
2247
m1000e no traffic between A2 and blades with esxi
Enivornment: m1000e with m6220 switches in the A1/A2. Blades are M630
our ESXI hosts are teamed between A1 and A2. A1 and A2 are not stacked. The interfaces for vmnic0(a1) and vmnic1(a2) on the m6220 are configured the same.
Right now, all the blades function when only assigned to vmnic0(a1), however traffic stops when we add in vmnic1(a2) as a redundant link. if we just use vmnic1 and remove vmnic0, they also do not send traffic.
Running show interface counters gi1/0/9, i'm seeing minimal traffic, with all traffic being discarded on transmit.
Port InTotalPkts InUcastPkts InMcastPkts InBcastPkts
--------- ---------------- ---------------- ---------------- ----------------
Gi1/0/9 2685 434 58 2193
Port OutTotalPkts OutUcastPkts OutMcastPkts OutBcastPkts
--------- ---------------- ---------------- ---------------- ----------------
Gi1/0/9 158902 2380 48510 108012
FCS Errors: ................................... 0
Single Collision Frames: ...................... 0
Late Collisions: .............................. 0
Excessive Collisions: ......................... 0
Multiple Collisions: .......................... 0
Received Oversize Packets: .................... 0
Transmitted Oversize Packets: ................. 0
Internal MAC Rx Errors: ....................... 0
Received Pause Frames: ........................ 0
Transmitted Pause Frames: ..................... 0
Receive Packets Discarded: .................... 0
Transmit Packets Discarded: ................... 559886
I'm having trouble identifying if there is a configuration issue on the ESXI host, or the m1000e, or if there is a hardware failure.
Everything was working fine until some weeks ago. The server admin noted that no changes were made on his end in ESXI or in the dell CMC, and no configuration changes were made on the switches.
The A2 switch shows as connected, and no errors in the Dell CMC for it. CDP is successful over the link between the A2 and the blades.
I'm trying to find out why the transmits are all being discarded. Any help is appreciated
d_ready
14 Posts
0
April 5th, 2021 10:00
I don't mean to Triple Post, but I've found what appears to be the cause. Root still to be determined by the server Admin.
With noticing traffic failing on the link between my c9500 and the m6220, I looked into the counters and saw the TX pause frames:
BldChs1A2#show interfaces counters te1/1/1
Port InTotalPkts InUcastPkts InMcastPkts InBcastPkts
--------- ---------------- ---------------- ---------------- ----------------
Te1/1/1 1128521 18698 375403 734420
Port OutTotalPkts OutUcastPkts OutMcastPkts OutBcastPkts
--------- ---------------- ---------------- ---------------- ----------------
Te1/1/1 9363 1208 7828 327
FCS Errors: ................................... 0
Single Collision Frames: ...................... 0
Late Collisions: .............................. 0
Excessive Collisions: ......................... 0
Multiple Collisions: .......................... 0
Received Oversize Packets: .................... 0
Transmitted Oversize Packets: ................. 0
Internal MAC Rx Errors: ....................... 0
Received Pause Frames: ........................ 0
Transmitted Pause Frames: ..................... 13641269
Receive Packets Discarded: .................... 32
Transmit Packets Discarded: ................... 0
I then checked all my interface counters on the m6220, to see if there was excessive traffic coming from any of it's active interfaces (Only blade 9 was supposed to have any vnic's attached to a2).
BldChs1A2#show interfaces counters
Port InTotalPkts InUcastPkts InMcastPkts InBcastPkts
--------- ---------------- ---------------- ---------------- ----------------
Gi1/0/1 0 0 0 0
Gi1/0/2 0 0 0 0
Gi1/0/3 108 0 108 0
Gi1/0/4 3775806 0 3775806 0
Gi1/0/5 0 0 0 0
Gi1/0/6 0 0 0 0
Gi1/0/7 0 0 0 0
Gi1/0/8 0 0 0 0
Gi1/0/9 637 224 113 300
Gi1/0/10 0 0 0 0
Gi1/0/11 86 0 20 66
Gi1/0/12 0 0 0 0
Gi1/0/13 0 0 0 0
Gi1/0/14 0 0 0 0
Gi1/0/15 0 0 0 0
Gi1/0/16 0 0 0 0
Gi1/0/17 0 0 0 0
Gi1/0/18 0 0 0 0
Gi1/0/19 0 0 0 0
Gi1/0/20 0 0 0 0
Te1/1/1 1134693 18753 377354 738586
Te1/1/2 0 0 0 0
Gi1/0/9 was supposed to have traffic, as well as TE 1/1/1. Gi1/0/11 as low, and negligible, but Gi1/0/4 stuck out.
I shut down interface gi1/0/4, and immediately the pause frames stopped. I was then able to reach the m6220 from my c9500, as well as reach all my test VMs.
We're currently checking out what on blade 4 would be causing this kind of traffic that caused the pause frames. It's al older 610 half height blade, but vcenter doesn't show any NIC's that would appear on the A2 switch (vmnic1). We might just decommission this blade, but I'd like to find the actual cause first.
For now, it appears the issue is resolved.
DELL-Josh Cr
Moderator
Moderator
•
8.5K Posts
0
March 30th, 2021 14:00
Hi,
What does "show spanning-tree” show for that port? Page 756 https://dell.to/3fwPLl9
d_ready
14 Posts
0
March 31st, 2021 06:00
Hi Josh,
Here is the Spanning Tree output:
BldChs1A2#show spanning-tree gi1/0/9
Port Gi1/0/9 Enabled
State: Forwarding Role: Designated
Port id: 128.9 Port Cost: 20000
Port Fast: Yes Root Protection: No
Designated bridge Priority: 32768 Address: 0025.6413.AB6E
Designated port id: 128.9 Designated path cost: 0
CST Regional Root: 80:00:00:25:64:13:AB:6E
Root Guard..................................... FALSE
Loop Guard..................................... FALSE
TCN Guard...................................... FALSE
Auto Portfast.................................. TRUE
Port Up Time Since Counters Last Cleared....... 0 day 1 hr 42 min 38 sec
BPDU: sent 43268, received 0
Dell-Stephane T
4 Operator
4 Operator
•
3.7K Posts
0
March 31st, 2021 07:00
Hi,
Spanning tree seems to be activated, could you send the command show spanning-tree detail for deeper analysis? Could you also check/deactivate all firewall rules on the VM?
Regards,
d_ready
14 Posts
0
March 31st, 2021 09:00
Here's the output of the detail for the port:
Spanning tree Enabled (BPDU flooding : Disabled) Portfast BPDU filtering Disabled mode rstp
CST Regional Root: 80:00:00:25:64:13:AB:6E
Regional Root Path Cost: 0
ROOT ID
Address 80:00:00:25:64:13:AB:6E
This Switch is the Root.
Hello Time 2 Sec Max Age 20 sec Forward Delay 15 sec TxHoldCount 6 sec
Bridge Max Hops 20
Number of topology changes 0 last change occurred 13d21h49m42s ago
Times: hold 6, hello 2, max age 20, forward delay 15
Port Gi1/0/9 Enabled
State: Forwarding Role: Designated
Port id: 128.9 Port Cost: 20000
Root Protection: No
Designated bridge Priority: 32768 Address: 0025.6413.AB6E
Designated port id: 128.9 Designated path cost: 0
CST Regional Root: 80:00:00:25:64:13:AB:6E CST Port Cost: 0
BPDU: sent 48271, received 0
I'll need to get a request in to disable the firewall on esxi instance, however here is the firewall rule list
My test VM in this blade has it's windows firewall disabled.
esxcli network firewall ruleset list
Name Enabled
---------------------- -------
sshServer true
sshClient false
nfsClient false
nfs41Client false
dhcp true
dns true
snmp true
ntpClient false
CIMHttpServer true
CIMHttpsServer true
CIMSLP true
iSCSI true
vpxHeartbeats true
updateManager true
faultTolerance true
webAccess true
vMotion true
vSphereClient true
activeDirectoryAll true
NFC true
HBR true
ftpClient false
httpClient false
gdbserver false
DVFilter false
DHCPv6 false
DVSSync true
syslog false
WOL true
vSPC false
remoteSerialPort false
rdt false
cmmds false
vsanvp false
rabbitmqproxy true
ipfam false
vvold false
iofiltervp true
esxupdate false
vit false
vsanEncryption false
pvrdma false
vsanhealth-unicasttest false
DELL-Josh Cr
Moderator
Moderator
•
8.5K Posts
0
March 31st, 2021 10:00
Spanning-tree looks fine. Can you reboot that switch?
d_ready
14 Posts
0
March 31st, 2021 10:00
We've rebooted the switch multiple times, and also loaded the config onto a spare unit and swapped them out. No success with either option.
d_ready
14 Posts
0
March 31st, 2021 11:00
It was updated in one of our troubleshooting steps. We still have the old version saved under the 2nd image.
I don't have evidence for it, as I'm not very familiar with the m1000e, however I'm suspecting an issue on the Chassis A2 slot. The VM we have attached to it currently is able to get a DHCP address, but no ICMP or other traffic is passing. First I need to rule out any possible network issues.
DELL-Josh Cr
Moderator
Moderator
•
8.5K Posts
0
March 31st, 2021 11:00
Was it updated as a troubleshooting step or was it always a different version from the other switch?
DELL-Josh Cr
Moderator
Moderator
•
8.5K Posts
0
March 31st, 2021 11:00
What firmware version are you on?
d_ready
14 Posts
0
March 31st, 2021 11:00
Our A2 m6220 is on 5.1.16.1
Our functioning A1 m6220 is on 5.1.8.2
DELL-Josh Cr
Moderator
Moderator
•
8.5K Posts
0
March 31st, 2021 12:00
Do you have any blades that are not running ESX or can have downtime? so we can rule out the vswitch as the issue.
d_ready
14 Posts
0
March 31st, 2021 19:00
I'm running my tests on blade 9, it is running a separate instance of esxi from the rest of my blades so I have a workable test environment, It can be taken offline as needed
DELL-Joey C
Moderator
Moderator
•
3K Posts
0
April 1st, 2021 01:00
Hi,
Do you have any additional slot in the chassis that you can swap the M630 into? Alternative, could you run show spanning-tree detail on A1 to compare the configuration with A2 switch port.
d_ready
14 Posts
0
April 1st, 2021 06:00
Hi Joey,
The issue can be replicated on every slot in the chassis, they are all using m630s.
We're just using blade 9 for testing, so we can continue running production services on the remaining blades, without the redundant a2 connection.
Here is the detail for the A1
Spanning tree Enabled (BPDU flooding : Disabled) Portfast BPDU filtering Disabled mode rstp
CST Regional Root: 80:00:5C:26:0A:C6:B3:5F
Regional Root Path Cost: 0
ROOT ID
Address 80:00:00:1F:9A:3E:A0:00
Path Cost 2004
Root Port Po1
Hello Time 2 Sec Max Age 20 sec Forward Delay 15 sec TxHoldCount 6 sec
Bridge Max Hops 20
Bridge ID
Priority 32768
Address 80:00:5C:26:0A:C6:B3:5F
Hello Time 2 Sec Max Age 20 sec Forward Delay 15 sec
Number of topology changes 1 last change occurred 188d0h28m40s ago
Times: hold 6, hello 2, max age 20, forward delay 15
Port Gi1/0/9 Enabled
State: Forwarding Role: Designated
Port id: 128.9 Port Cost: 20000
Root Protection: No
Designated bridge Priority: 32768 Address: 5C26.0AC6.B35F
Designated port id: 128.9 Designated path cost: 2004
CST Regional Root: 80:00:5C:26:0A:C6:B3:5F CST Port Cost: 0
BPDU: sent 8122435, received 0