pk2016
1 Copper

Force10 S4810 - clish constantly crashing

Hello!

S4810#show version
Dell Real Time Operating System Software
Dell Operating System Version: 2.0
Dell Application Software Version: 9.11(2.8)
Copyright (c) 1999-2017 by Dell Inc. All Rights Reserved.
Build Time: Sat Jan 13 04:34:59 2018

After some time 2 ( TWO! ) of those devices stopped to accept telnet connection and even console connection!

Here is what we see on console port

 

con0 now available


Press RETURN to get started.

>trap: pid 26358.1 (clish): DTLB exception in user mode
trapframe 0xfdfa3f28 (exc=300 srr0/1=0x65996098/0x2d230 esr/dear=0x800000/0x28)
lr =65996088 ctr=00000000 cr =24002042 xer=00000000
r00=65996088 r01=651aff10 r02=01814764 r03=00000000
r04=00000000 r05=00000000 r06=00000000 r07=65b387ec
r08=00001770 r09=00000000 r10=00000000 r11=613efec0
r12=00000000 r13=01824a54 r14=00000000 r15=6fb48250
r16=6fafbfa4 r17=65b387cc r18=6fb30b50 r19=65b387ec
r20=00000000 r21=65aef90c r22=6fb3c6d0 r23=00000000
r24=000066f6 r25=6fafbf90 r26=41a7d659 r27=54000000
r28=00000000 r29=00000000 r30=65af15a8 r31=00000000

 

 

Here is what we see on syslog server from one of those devices

 

Jul 11 16:34:29 S4810 xx.xx.xx.xx %S4810:0 %KERN-4-INT: File table is 30630.000000ull
Jul 11 16:35:00 S4810 xx.xx.xx.xx %S4810:0 %KERN-4-INT: File table is 30630.000000ull
Jul 11 16:35:11 S4810 xx.xx.xx.xx %S4810:0 %KERN-3-INT: file: table is full - increase kern.maxfiles or MAXFILES
Jul 11 16:35:11 S4810 xx.xx.xx.xx %S4810:0 %POLLMGR-2-USER_FLASH_STATE: Internal flash disk removed from 'flash:'
Jul 11 16:35:12 S4810 xx.xx.xx.xx %S4810:0 %POLLMGR-2-USER_FLASH_STATE: Internal flash disk inserted in 'flash:'
Jul 11 16:35:16 S4810 xx.xx.xx.xx %S4810:0 %POLLMGR-2-USER_FLASH_STATE: Internal flash disk removed from 'flash:'
Jul 11 16:35:17 S4810 xx.xx.xx.xx %S4810:0 %POLLMGR-2-USER_FLASH_STATE: Internal flash disk inserted in 'flash:'
Jul 11 16:35:19 S4810 xx.xx.xx.xx %S4810:0 %POLLMGR-2-USER_FLASH_STATE: Internal flash disk removed from 'flash:'
Jul 11 16:35:19 S4810 xx.xx.xx.xx %S4810:0 %POLLMGR-2-USER_FLASH_STATE: Internal flash disk inserted in 'flash:'
Jul 11 16:35:20 S4810 xx.xx.xx.xx %S4810:0 %POLLMGR-2-USER_FLASH_STATE: Internal flash disk removed from 'flash:'
Jul 11 16:35:21 S4810 xx.xx.xx.xx %S4810:0 %POLLMGR-2-USER_FLASH_STATE: Internal flash disk inserted in 'flash:'
Jul 11 16:35:22 S4810 xx.xx.xx.xx %S4810:0 %KERN-3-INT: file: table is full - increase kern.maxfiles or MAXFILES#012 - repeated 593 times

 

 

 

0 Kudos
7 Replies
Moderator
Moderator

Re: Force10 S4810 - clish constantly crashing

I am not finding much info on these messages. Did these message seem to occur after a recent change to the switch? Do you have two switches giving these messages? Are these two stacked or directly connected together? Is the switch still operational and passing traffic?

Daniel Covey
Dell EMC| Enterprise Support Services
Get support on Twitter:@DellCaresPRO
Download our QRL app:iOS, Android, Windows
Dell Networking Resources

0 Kudos
pk2016
1 Copper

Re: Force10 S4810 - clish constantly crashing

Hello!

Those are two devices at different locations. No stacks, just standalone switches. Simplest configs ever - just vlans and LACP LAGs, no L3 routing, just in-band management and snmp.

No changes recently were made. I think those issues are related to our telnet script grabbing signal levels from transceivers. But it is very strange anyhow.

Still passing traffic and graphing graphs by snmp but no management, e.g. no telnet, no ssh, no console.

Still see some syslog messages on syslog server. 

0 Kudos
Moderator
Moderator

Re: Force10 S4810 - clish constantly crashing

Have you done a reboot of the switch? A reboot may allow the switch to be accessed again.

Daniel Covey
Dell EMC| Enterprise Support Services
Get support on Twitter:@DellCaresPRO
Download our QRL app:iOS, Android, Windows
Dell Networking Resources

0 Kudos
pk2016
1 Copper

Re: Force10 S4810 - clish constantly crashing

Hello! Reboot was performed, and switch is accessible now.

Buuut ..

Is it really enterprise or telecom firmware with those bugs?

0 Kudos
Moderator
Moderator

Re: Force10 S4810 - clish constantly crashing

Glad to hear a reboot got the switches accessible again. Just to confirm, management to the switch is lost after running a script that polls the switch? Outside of the script, the switch operates as expected?

Daniel Covey
Dell EMC| Enterprise Support Services
Get support on Twitter:@DellCaresPRO
Download our QRL app:iOS, Android, Windows
Dell Networking Resources

0 Kudos
pk2016
1 Copper

Re: Force10 S4810 - clish constantly crashing

I think so.

No, it is not the only bug, but it is most annoying 

0 Kudos
Moderator
Moderator

Re: Force10 S4810 - clish constantly crashing

What version firmware is on the second image of the switch? Have you tried booting to that image? To see if there is a difference in behavior.

Daniel Covey
Dell EMC| Enterprise Support Services
Get support on Twitter:@DellCaresPRO
Download our QRL app:iOS, Android, Windows
Dell Networking Resources

0 Kudos