Unsolved
This post is more than 5 years old
20 Posts
0
52557
July 23rd, 2014 08:00
Spanning Tree issues - really out of no where.
Hello all, Sometime during the day yesterday our network went down and I attest this to an STP issue. At the time nothing major was happening except for I was rolling out a couple servers from a template (usual stuff) I have since got it back in a state where we can function but one of the 6248's is out of commission so we are not as redundant. If I use port 48 for the lag port connecting to the other 6248, everything goes down (they are both configured for LAG 1 on that port). The same goes for trying to connect the functioning 6248 to the 6224 switch, one connection via LAG 2 working, but If I use a 2nd connection traffic is lost. So as of now it is isolated until I understand the situation. I upgraded the firmware on the 6248 this morning but was hoping for some help since STP is now to me, especially from a troubleshooting perspective. We have 2 6224 switches running the core redundant iscsi network that are LAG'd to 2 more 6248 switches (LAGS 2 and 3 respectively), and those 6248's where LAG'd into themselves (LAG 1). I had STP enabled for all switches, with the recommended portfast setting on the 6224 ports that were connected to the hosts and shared storage. Everything was working great up until yesterday. I also tried reconfiguring the LAG connections but one switch would work while connectivity to another wouldn't. or would the 2 6248's would see each other but wouldn't traverse LAG2 and 3 to see the iscsi network and servers. no spanning-tree spanning-tree mode stp
No Events found!
psytek
20 Posts
0
July 23rd, 2014 08:00
I realize some ports show as DIS as opposed to FWD state, would it be safe to say that those connected ports need to be in a FWD and if so, should I just manually no shutdown the Ethernet ports?
psytek
20 Posts
0
July 23rd, 2014 10:00
Also, this is what reads in the startup-config now of the 6224 switches (single block for the Ethernet port), clearly I was messing with STP settings...should I change anything? Should I run spanning-tree to remove the no spanning tree line? Also, when looking at show spanning-tree results, I see a lot of ports (that are legitimate ports) are set to DIS and not FWD. Should I manually sent them all to forwarding?
no spanning-tree spanning-tree mode stp
interface ethernet 2/g11 spanning-tree cost 20000 spanning-tree mst 0 external-cost 20000 mtu 9216 switchport access vlan 2 exit
1/g7 Enabled 128.7 20000 DIS Disb Yes No 1/g8 Enabled 128.8 20000 DIS Disb No No 1/g9 Enabled 128.9 20000 DIS Disb Yes No 1/g10 Enabled 128.10 0 DIS Disb No No 1/g11 Enabled 128.11 20000 DIS Disb No No 1/g12 Enabled 128.12 20000 DIS Disb No No 1/g13 Enabled 128.13 20000 DIS Disb No No 1/g14 Enabled 128.14 0 DIS Disb No No 1/g15 Enabled 128.15 0 FWD Disb No No 1/g16 Enabled 128.16 0 FWD Disb No No
psytek
20 Posts
0
July 23rd, 2014 10:00
Daniel,
The network worked perfectly up until that moment yesterday.
I can post a network topology later but I can quickly describe it too.
6248-1 and 6248-2 both have LAG 1 assigned to ports 47 and 48
6224 (2 switches stacked) are configured with LAG 2 and LAG 3
LAG 2 is assigned to ports 23 and 24 on 6224-1 and linked to ports 45 and 46 on 6248-1
LAG 3 is assigned to ports 23 and 24 on 6224-2 and linked to ports 45 and 46 on 6248-2
psytek
20 Posts
0
July 23rd, 2014 11:00
I was thinking of doing the following:
Removing the 'no spanning-tree' and enabling it
Removing the cost
Removing the mst 0 option
Manually 'no shutdown' of all DIS ports under the spanning-tree list and this includes the DIS port-channel groups too [2 and 3]
Confirm that STP is in the same mode across all switches...spanning-tree, then spanning-tree mode stp
psytek
20 Posts
0
July 23rd, 2014 11:00
I was thinking of doing the following:<><>
Removing the 'no spanning-tree' and enabling it
Removing the cost
Removing the mst 0 option
Manually 'no shutdown' of all DIS ports under the spanning-tree list and this includes the DIS port-channel groups too [2 and 3]
Confirm that STP is in the same mode across all switches...spanning-tree, then spanning-tree mode stp
psytek
20 Posts
0
July 23rd, 2014 12:00
Some ports go to hosts and storage and they are seen as disabled now. I will manually turn them up after I begin the session.
When you say most servers what do you mean? And should I not use portfast on a EQL connection?
I will set RSTP on all switches as to be consistent.
Thanks again, I will keep you posted.
psytek
20 Posts
0
July 23rd, 2014 12:00
Firmware is the latest.
I will make those priority settings tonight.
I was going to slightly modify the LAG a little bit as (better optimization).
Questions though:
Should I remove the COSTand MST settings for those ports? I didn't see those before and not sure how they came about.
Should I also manually set the ports to FWD iD they are disabled while a device is plugged into those ports?
Should I only enable PORTFAST on ports connected to storage or hosts?
Do all 4 switches need matching STP modes?
Do I need to configure STP for my LAG connections?
Thanks a lot.
psytek
20 Posts
0
July 23rd, 2014 13:00
Yes, those hosts house our virtual environment. Management, VMs, vswitches, etc...
Thanks.
psytek
20 Posts
0
July 30th, 2014 06:00
Daniel,
Just an update. That night, since we last spoke, I updated the firmware all around for the remaining switches, changed all configs to be consistent (enabled STP, made the mode RSTP and configured the root switch)), then I reconfigured the LAGS. On the 2 core 6224 switches, 6224-1 and 6224-2 I created LAG groups 2 and 3 respectively. On the 6224-1, ports 23 and 24 are LAG 2, and these LAGS got to ports 45 on the two 6248's. While on the 6224-2, ports 23 and 24 are LAG 3, these uplink to ports 46 on each of the 6248's. Then the 6248's use ports 47 and 48 as LAG1 and uplink to each other. Does this seem like proper config?
Also, we had another 'network downtime' episode this morning, although I was able to resolve it much quicker this time. I reseated all the LAG uplinks and the loop or error was removed, at least I think it was.
Anything I should look for? Show spanning-tree seems to show a proper status. I set the root like you had mentioned earlier, this switch is closest to the router/gateway. I did notice one thing though, that portfast was enabled on the lag ports and lags themselves - I believe this should be disabled correct?
psytek
20 Posts
0
July 30th, 2014 08:00
I'd like to add that I just noticed a portfast setting discrepancy. I noticed it's on for the client backbone including all the LAG ports, while it's off for the 6224 core switches including the LAG ports (except the ports connected to the hosts)...
it is recommened that portfast be turned off on lags, trunks or connections between switches and routers correct?