Unsolved
8 Posts
0
1524
April 20th, 2021 05:00
portflapping (like) behaviour powerconnect 5500. everyones experience welcome
Hello everyone
first: yes, I know a lot of the existing settings of our ancestors are... suboptimal, and we are working hard to build a new system, but for now it has to work the way it is, unfortunatly.
second: Of course I think it's great, that we get real Dell Support here, but this time I am also greatful for hints of everyone if you maybe have experienced or even solved a similiar problem. But it will be a long post...
So, we are a stundent dorm here with homogeneous 5500 series setup, all users in VLAN 1, no ACLs or routings on the switches, nothing.
But as a dorm, we don't have any access to the clientcomputers and if they don't want to cooperate we have nothing.
To some ports it happens now, that the Link goes Down and some seconds later Up again several times but not all the time.
Of course there can be a lot of reasons like a restart, but some users experience problems even when actively stream-ing/discording/gaming/whatever and we know a lot of ppl don't even complain, so we don't know the real scale of the problem.
Of course there can be a lot of reasons like a restart, but some users experience problems even when actively stream-ing/discording/gaming/whatever and we know a lot of ppl don't even complain, so we don't know the real scale of the problem.
We couldn't find any pattern. Not to time of day, not to repetition (sometimes it's 5 minutes of down and ups sometimes like 30 minutes or more, but there is always seconds between the downup, not multiple per second like real flapping), also couldn't find a pattern to the ports plug a user to another port and it happens there too), or to whom it happens (sometimes old laptops sometimes really new once, some really old AP or even a COM-to-LAN bridge). Some cases are directly connected, some have a accesspoint between the switchport and their PC (in their case it's probably the APs fault, but the same model for other users works).
It happens on all switches, Firmware is updated everywhere.
There might be, but I am not sure yet, a cluster of cases on ports which report "remote peer true" for greenethernet. Thus we tried it without EEE, with EEE w/o LLDP, with both active, but there was no difference. With a cooperative user we tried deactivating all greenethernet things on the PCs NIC side, still happens.
With STP, w/o, still hapens.
Even in debug output, there is no reason written in the logs.
It could be a coincident, but it feels like after applying a different port setting it takes some hours, before it happens again. But not sure.
With STP, w/o, still hapens.
Even in debug output, there is no reason written in the logs.
It could be a coincident, but it feels like after applying a different port setting it takes some hours, before it happens again. But not sure.
Thank you for reading up(down?) to here.
Has anyone ever heard of something like this or some smart idea, how to narrow down the problem more?
No Events found!
DELL-Josh Cr
Moderator
•
9.6K Posts
•
42.2K Points
0
April 20th, 2021 10:00
Hi,
It sounds like you have done a lot of the troubleshooting steps already. Is it just one switch that is having the issue or is it multiple 5500s? Is there a lot of traffic on there? Some of these older switches would have weird issues if the traffic load was higher than what the switch could handle. You may want to try port mirroring and monitoring with wireshark to see if it shows anything during the drops. Page 411 https://dell.to/3xlHiYG
kumasan
8 Posts
0
April 20th, 2021 10:00
Hi Josh, thanks for the reply.
Yes, it happens to my knowledge on all 5 5548 and 1 5524P. Trafficwise the switches are bored I think. Even the uplink ports barely reach 100M if at all. And the Com-To-LAN Bridge where I have seen it happening too, doesn't even do 1k I would guess. Or that is what munin says.
Yes, port mirroring I actually tried already for some days, but right in that period when port mirroring was active, the port..flapping (can't think of a better word) didn't happen.
Could be coicident, but the problem is, that it happens for some minutes, but it can be hours if not days until it happens again and we don't know wether it's the tenants not using their computer or just the problem not showing.
But I will try a capture again. Maybe until then there are also other ideas
kumasan
8 Posts
0
April 22nd, 2021 07:00
So, for your reference the show log (trimmed to link down/up for port 5)
22-Apr-2021 08:21:36 :%LINK-W-Down: gi1/0/5
22-Apr-2021 08:21:34 :%LINK-I-Up: gi1/0/5
22-Apr-2021 08:21:32 :%LINK-W-Down: gi1/0/5
22-Apr-2021 08:21:30 :%LINK-I-Up: gi1/0/5
22-Apr-2021 08:21:24 :%LINK-W-Down: gi1/0/5
22-Apr-2021 08:20:36 :%LINK-I-Up: gi1/0/5
22-Apr-2021 08:20:33 :%LINK-W-Down: gi1/0/5
22-Apr-2021 08:20:00 :%LINK-I-Up: gi1/0/5
22-Apr-2021 08:19:57 :%LINK-W-Down: gi1/0/5
22-Apr-2021 08:19:56 :%LINK-I-Up: gi1/0/5
22-Apr-2021 08:19:53 :%LINK-W-Down: gi1/0/5
22-Apr-2021 08:19:52 :%LINK-I-Up: gi1/0/5
22-Apr-2021 08:19:49 :%LINK-W-Down: gi1/0/5
22-Apr-2021 08:19:49 :%LINK-I-Up: gi1/0/5
22-Apr-2021 08:19:47 :%LINK-W-Down: gi1/0/5
22-Apr-2021 08:19:46 :%LINK-I-Up: gi1/0/5
22-Apr-2021 08:19:44 :%LINK-W-Down: gi1/0/5
22-Apr-2021 08:19:43 :%LINK-I-Up: gi1/0/5
22-Apr-2021 08:19:40 :%LINK-W-Down: gi1/0/5
22-Apr-2021 08:19:39 :%LINK-I-Up: gi1/0/5
22-Apr-2021 08:19:33 :%LINK-W-Down: gi1/0/5
22-Apr-2021 08:18:23 :%LINK-I-Up: gi1/0/5
22-Apr-2021 08:18:20 :%LINK-W-Down: gi1/0/5
22-Apr-2021 08:18:19 :%LINK-I-Up: gi1/0/5
22-Apr-2021 08:18:12 :%LINK-W-Down: gi1/0/5
22-Apr-2021 08:17:57 :%LINK-I-Up: gi1/0/5
22-Apr-2021 08:17:51 :%LINK-W-Down: gi1/0/5
22-Apr-2021 08:17:49 :%LINK-I-Up: gi1/0/5
22-Apr-2021 08:17:47 :%LINK-W-Down: gi1/0/5
22-Apr-2021 08:17:44 :%LINK-I-Up: gi1/0/5
22-Apr-2021 08:17:41 :%LINK-W-Down: gi1/0/5
22-Apr-2021 08:17:08 :%LINK-I-Up: gi1/0/5
22-Apr-2021 08:17:06 :%LINK-W-Down: gi1/0/5
22-Apr-2021 08:17:05 :%LINK-I-Up: gi1/0/5
22-Apr-2021 08:17:02 :%LINK-W-Down: gi1/0/5
22-Apr-2021 08:17:01 :%LINK-I-Up: gi1/0/5
22-Apr-2021 08:16:58 :%LINK-W-Down: gi1/0/5
22-Apr-2021 08:16:57 :%LINK-I-Up: gi1/0/5
22-Apr-2021 08:16:54 :%LINK-W-Down: gi1/0/5
22-Apr-2021 08:16:53 :%LINK-I-Up: gi1/0/5
22-Apr-2021 08:16:50 :%LINK-W-Down: gi1/0/5
22-Apr-2021 08:16:30 :%LINK-I-Up: gi1/0/5
22-Apr-2021 08:16:27 :%LINK-W-Down: gi1/0/5
22-Apr-2021 08:16:26 :%LINK-I-Up: gi1/0/5
22-Apr-2021 08:16:20 :%LINK-W-Down: gi1/0/5
22-Apr-2021 08:15:58 :%LINK-I-Up: gi1/0/5
22-Apr-2021 08:15:56 :%LINK-W-Down: gi1/0/5
22-Apr-2021 08:15:55 :%LINK-I-Up: gi1/0/5
22-Apr-2021 08:15:53 :%LINK-W-Down: gi1/0/5
22-Apr-2021 08:15:51 :%LINK-I-Up: gi1/0/5
22-Apr-2021 08:15:49 :%LINK-W-Down: gi1/0/5
22-Apr-2021 08:15:47 :%LINK-I-Up: gi1/0/5
22-Apr-2021 08:15:45 :%LINK-W-Down: gi1/0/5
22-Apr-2021 08:15:43 :%LINK-I-Up: gi1/0/5
22-Apr-2021 08:15:40 :%LINK-W-Down: gi1/0/5
22-Apr-2021 08:15:39 :%LINK-I-Up: gi1/0/5
22-Apr-2021 08:15:36 :%LINK-W-Down: gi1/0/5
22-Apr-2021 08:15:08 :%LINK-I-Up: gi1/0/5
22-Apr-2021 08:15:05 :%LINK-W-Down: gi1/0/5
22-Apr-2021 08:15:03 :%LINK-I-Up: gi1/0/5
22-Apr-2021 08:14:57 :%LINK-W-Down: gi1/0/5
22-Apr-2021 08:08:33 :%LINK-W-Down: gi1/0/5
22-Apr-2021 08:08:32 :%LINK-I-Up: gi1/0/5
22-Apr-2021 08:08:31 :%LINK-W-Down: gi1/0/5
22-Apr-2021 08:08:30 :%LINK-I-Up: gi1/0/5
22-Apr-2021 08:08:23 :%LINK-W-Down: gi1/0/5
22-Apr-2021 08:08:22 :%LINK-I-Up: gi1/0/5
22-Apr-2021 08:08:19 :%LINK-W-Down: gi1/0/5
22-Apr-2021 08:08:18 :%LINK-I-Up: gi1/0/5
22-Apr-2021 08:08:15 :%LINK-W-Down: gi1/0/5
22-Apr-2021 08:08:15 :%LINK-I-Up: gi1/0/5
22-Apr-2021 08:08:13 :%LINK-W-Down: gi1/0/5
22-Apr-2021 08:08:12 :%LINK-I-Up: gi1/0/5
22-Apr-2021 08:08:09 :%LINK-W-Down: gi1/0/5
22-Apr-2021 08:08:08 :%LINK-I-Up: gi1/0/5
22-Apr-2021 08:08:05 :%LINK-W-Down: gi1/0/5
22-Apr-2021 08:08:04 :%LINK-I-Up: gi1/0/5
22-Apr-2021 08:08:01 :%LINK-W-Down: gi1/0/5
22-Apr-2021 08:07:49 :%LINK-I-Up: gi1/0/5
22-Apr-2021 08:07:39 :%LINK-W-Down: gi1/0/5
22-Apr-2021 08:02:21 :%LINK-W-Down: gi1/0/5
22-Apr-2021 08:02:20 :%LINK-I-Up: gi1/0/5
22-Apr-2021 08:02:13 :%LINK-W-Down: gi1/0/5
22-Apr-2021 08:02:12 :%LINK-I-Up: gi1/0/5
22-Apr-2021 08:02:09 :%LINK-W-Down: gi1/0/5
22-Apr-2021 08:01:53 :%LINK-I-Up: gi1/0/5
22-Apr-2021 08:01:47 :%LINK-W-Down: gi1/0/5
22-Apr-2021 08:01:46 :%LINK-I-Up: gi1/0/5
22-Apr-2021 08:01:33 :%LINK-W-Down: gi1/0/5
22-Apr-2021 07:30:43 :%LINK-W-Down: gi1/0/5
22-Apr-2021 07:30:42 :%LINK-I-Up: gi1/0/5
22-Apr-2021 07:30:39 :%LINK-W-Down: gi1/0/5
22-Apr-2021 07:30:38 :%LINK-I-Up: gi1/0/5
22-Apr-2021 07:30:35 :%LINK-W-Down: gi1/0/5
22-Apr-2021 07:30:34 :%LINK-I-Up: gi1/0/5
22-Apr-2021 07:30:31 :%LINK-W-Down: gi1/0/5
22-Apr-2021 07:30:31 :%LINK-I-Up: gi1/0/5
22-Apr-2021 07:30:28 :%LINK-W-Down: gi1/0/5
22-Apr-2021 07:30:26 :%LINK-I-Up: gi1/0/5
22-Apr-2021 07:30:23 :%LINK-W-Down: gi1/0/5
22-Apr-2021 07:30:21 :%LINK-I-Up: gi1/0/5
22-Apr-2021 07:30:18 :%LINK-W-Down: gi1/0/5
22-Apr-2021 07:29:22 :%LINK-I-Up: gi1/0/5
22-Apr-2021 07:29:12 :%LINK-W-Down: gi1/0/5
22-Apr-2021 06:42:22 :%LINK-W-Down: gi1/0/5
22-Apr-2021 06:42:21 :%LINK-I-Up: gi1/0/5
22-Apr-2021 06:42:18 :%LINK-W-Down: gi1/0/5
22-Apr-2021 06:42:17 :%LINK-I-Up: gi1/0/5
22-Apr-2021 06:42:10 :%LINK-W-Down: gi1/0/5
22-Apr-2021 06:41:54 :%LINK-I-Up: gi1/0/5
22-Apr-2021 06:41:48 :%LINK-W-Down: gi1/0/5
22-Apr-2021 06:41:36 :%LINK-I-Up: gi1/0/5
22-Apr-2021 06:41:27 :%LINK-W-Down: gi1/0/5
22-Apr-2021 06:41:26 :%LINK-I-Up: gi1/0/5
22-Apr-2021 06:41:23 :%LINK-W-Down: gi1/0/5
22-Apr-2021 06:41:22 :%LINK-I-Up: gi1/0/5
22-Apr-2021 06:41:19 :%LINK-W-Down: gi1/0/5
22-Apr-2021 06:41:18 :%LINK-I-Up: gi1/0/5
22-Apr-2021 06:41:15 :%LINK-W-Down: gi1/0/5
22-Apr-2021 06:41:14 :%LINK-I-Up: gi1/0/5
22-Apr-2021 06:41:11 :%LINK-W-Down: gi1/0/5
22-Apr-2021 06:40:58 :%LINK-I-Up: gi1/0/5
22-Apr-2021 06:40:56 :%LINK-W-Down: gi1/0/5
22-Apr-2021 06:40:55 :%LINK-I-Up: gi1/0/5
22-Apr-2021 06:40:52 :%LINK-W-Down: gi1/0/5
22-Apr-2021 06:40:51 :%LINK-I-Up: gi1/0/5
22-Apr-2021 06:40:48 :%LINK-W-Down: gi1/0/5
In the IO Graph over the captured time, one can see the link down phase pretty well, as packets per second drop to zero.
Unfortunatly, the only interesting packets I found where
src "Switch", dst "Nearest-non-TPMR-brdge", protocol "eap"
but those come only after a three second break in packets, so it's the link up I guess. The last packets before the previously stated one are mostly mDNS packets, but always from another source + mDNS is in the capture 44% backgroundnoise.
Is there a way to tell, wether the switch or the client takes the link down?
(Although we have that problem with multiple pretty different clients)
DELL-Marco B
Moderator
•
4K Posts
0
April 22nd, 2021 07:00
Hello,
I will check with Josh the troubleshooting about this issue, and i will let you know.
Thanks
Marco
DELL-Josh Cr
Moderator
•
9.6K Posts
•
42.2K Points
0
April 22nd, 2021 10:00
Thanks for sharing what you found. Doesn’t seem like there is anything in the traffic that is causing it. Can you run a show tech-support and see if there are any spanning-tree errors?
kumasan
8 Posts
0
April 22nd, 2021 12:00
---------------- ------------ ------------ ------------ ------------
gi1/0/2 3840678478 10262668 344617 463531314957
1
gi1/0/3 168078 0 230 28054850
gi1/0/4 0 0 0 0
gi1/0/5 657358 6046 1920 540820566
gi1/0/7 0 23 444 74467
gi1/0/8 103954 23850 5379 20291887
gi1/0/9 0 0 0 0
gi1/0/10 2346465 147161 2281 355127674
gi1/0/11 608109 0 2174 40291232
gi1/0/12 0 0 0 0
gi1/0/13 1503530 0 2173 98151587
gi1/0/14 11525250 23818 117701 2125311422
gi1/0/15 1393624 0 2173 90896460
gi1/0/16 0 0 0 0
gi1/0/17 19360748 17086 4240 21140050868
gi1/0/18 233794001 8377717 1266396 57555538691
gi1/0/19 29168929 2141038 77471 9735950188
gi1/0/20 316499305 8246739 1398501 140946585365
gi1/0/21 793921900 7798437 1127048 531706462035
gi1/0/22 194752516 9319840 893776 76987023499
gi1/0/23 0 0 0 0
gi1/0/24 455114652 11631208 4319452 179607889005
te1/0/1 0 0 0 0
te1/0/2 0 0 0 0
---------------- ------------ ------------ ------------ ------------
gi1/0/2 1898282446 47775759 9227038 955356910611
gi1/0/3 416950 58026898 9457817 10826385574
gi1/0/4 0 0 0 0
gi1/0/5 947818 57365911 9357668 11052084702
gi1/0/7 787059 8915904 1160483 2259839799
gi1/0/8 353351 58012292 9472264 10829088165
gi1/0/9 0 0 0 0
gi1/0/10 2913616 57867042 9454895 11724451932
gi1/0/11 412186 208877 91220 47617935
gi1/0/12 0 0 0 0
gi1/0/13 964626 208874 91221 88486216
gi1/0/14 17918699 58006942 9453166 12430738103
gi1/0/15 918150 208873 91221 84480688
gi1/0/16 0 0 0 0
gi1/0/17 16949625 58018966 9567146 15623863279
gi1/0/18 699013575 49673941 8305507 881598683916
gi1/0/19 68546577 54840614 9418982 83668981191
gi1/0/20 724806813 49804808 8173254 866864088832
gi1/0/21 1109076029 50253153 8444818 127595766604
2
gi1/0/22 420205323 48731986 8678074 488050526704
gi1/0/23 0 0 0 0
gi1/0/24 940114152 46420473 5252355 114609170179
8
te1/0/1 0 0 0 0
te1/0/2 0 0 0 0
------------------ show interfaces counters errors ------------------
Port Align-Err FCS-Err Xmit-Err Rcv-Err OutDiscards
---------------- ---------- -------- --------- -------- ------------
gi1/0/2 0 0 0 0 0
gi1/0/3 0 0 0 0 0
gi1/0/4 0 0 0 0 0
gi1/0/7 0 0 0 0 0
gi1/0/8 0 0 0 0 0
gi1/0/9 0 0 0 0 0
gi1/0/10 0 0 0 0 0
gi1/0/11 0 0 0 0 0
gi1/0/12 0 0 0 0 0
gi1/0/13 0 0 0 0 0
gi1/0/14 0 0 0 0 0
gi1/0/15 0 0 0 0 0
gi1/0/16 0 0 0 0 0
gi1/0/17 0 0 0 0 0
gi1/0/18 0 0 0 0 0
gi1/0/19 0 0 0 0 0
gi1/0/20 0 0 0 0 0
gi1/0/21 0 0 0 0 0
gi1/0/22 0 0 0 0 0
gi1/0/23 0 0 0 0 0
gi1/0/24 0 0 0 0 0
te1/0/1 0 0 0 0 0
te1/0/2 0 0 0 0 0
If it is relevant, please keep in mind, that there is an acecss point over PoE at port 5, and yes, there where TCP Errors in the capture too, but not correlated.
Or are those errors not TCP related but something different switch internal and our culprit?
gi1/0/22 0 300 0 698 0 (comment 41)
flipped yesterday for 30 minutes. (Normal user with none of our APs)
but not a single link up or down in the last 10 days so can't say.
but in the last 3 weeks just normal behaviour
DELL-Josh Cr
Moderator
•
9.6K Posts
•
42.2K Points
0
April 22nd, 2021 13:00
The errors are usually caused by collisions or a duplex mismatch, which could easily happen with going from wired to wireless. Are the other ports with fewer errors also connected to APs?
kumasan
8 Posts
0
April 22nd, 2021 14:00
The other ports above don't have one of our official APs, but I also can't see macs that look like an AP vendor.
Btw, of our official APs we have more than 60, but the errors I posted above where the only big numbers. No other AP showed those errors.
DELL-Josh Cr
Moderator
•
9.6K Posts
•
42.2K Points
0
April 22nd, 2021 17:00
Wifi isn’t full duplex, so maybe that is why the AP has errors. Which model AP is connected? Maybe there is some compatibility issue.
kumasan
8 Posts
0
April 23rd, 2021 02:00
That one is a TP-Link TL-WR1043ND HW v1.x with ddwrt, but we have 2 more of those and those work fine.
And it wouldn't explain, why some users had that problem while directly connected via cable.
DELL-Josh Cr
Moderator
•
9.6K Posts
•
42.2K Points
0
April 23rd, 2021 10:00
That’s true, that the direct connected users shouldn’t be affected then. There are not any errors on the switches that the 5500s are uplinked too right? I don’t think it is a hardware issue since it is happening on all of the switches. Can you swap any of the switches with a different model and see if it still happens?
matze0409
1 Message
0
April 25th, 2023 03:00
Someone has solved this Problems? Becuase we have exact the same problem with N2248PX-ON Switches from Dell