Connectrix B-Series Brocade: How to recover from hung state on Management Ethernet port, with no access using webtools, telnet, or SSH

How to recover from hung state on Management Ethernet port, with no access using webtools, telnet, or SSH.

Brocade Director Class switch is not accessible via Chassis IP or Active CP IP, with no access using webtools, telnet, or SSH.
Brocade Departmental Switch is not accessible via switch IP, with no access via webtools, telnet, or SSH.

eth0

ifconfig

mgmt

According to Brocade L3 there are no known code issues around the Ethernet software, but they commonly see network configuration errors that result in the PHY getting hung. The PHY is the hardware Ethernet chip, and if enough errors are recorded it can result in that chip hanging. Depending on the network environment, a switch may run for some time before the configuration problem results in enough errors to cause a problem.

Here is an example of the stats from a switch with eth0 in hung state:

/sbin/ifconfig:
eth0   Link encap:Ethernet  HWaddr 00:05:33:32:40:E3 
          inet addr:xx.xxa.x.xxx  Bcast:xx.xxx.xx.xxx  Mask:xxx.xxx.xxx.x
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:368707819 errors:0 dropped:0 overruns:0 frame:450
          TX packets:161607771 errors:1321348 dropped:0 overruns:197 carrier:1291486
          collisions:129622 txqueuelen:1000

As seen above, the RX stats are clean, but there are significant TX errors reported.

Warning: Do not perform "ifconfig eth0 down" command to bring down the Ethernet port as it may cause panic or reboot on departmental class switch or unexpected HAfailover on Director class switch. 

Workaround for both Director and Departmental switch:

NOTE:  The workarounds below are considered non-disruptive, however it is recommended that the action plan be carried out during a customer window in case an unexpected reboot occurs.

Departmental Switch:

1. Connect to the switch via serial cable.

2. Reset eth0 using one of the options below (options are listed in recommended order from Brocade).

a. Perform hareboot, which will reboot the processor, but not the switch (non-disruptive to fibre channel)

Example:

DS5000B> hareboot
HA Rebooting ...

b. Change IP address of the switch and then change it back to the original IP.

Example:

Take note of the original IP address:

DS5000B> ipaddrshow

SWITCH
Ethernet IP Address: xx.xx.xx.xx
Ethernet Subnetmask: xxx.xxx.xxx.x
Gateway IP Address: xx.xx.xx.x
DHCP: Off

Change the IP settings back to factory default

DS5000B> ipaddrset
Ethernet IP Address [xx.xx.xx.xx]:xx.xx.xx.xx
Ethernet Subnetmask [xxx.xxx.xxx.x]:xxx.x.x.x
Gateway IP Address [xx.xx.xx.x]:x.x.x.x
DHCP [Off]:

Now change it back to the original IP

DS5000B> ipaddrset
Ethernet IP Address [xx.xx.xx.xx]:xx.xx.xx.xx
Ethernet Subnetmask [xxx.x.x.x]:xxx.xxx.xxx.x
Gateway IP Address [x.x.x.x]:xx.xx.xx.x
DHCP [Off]:

c. Change the flow control.

Example:

Take note of the current flow control.

DS5000B:> ifmodeshow eth0
Link mode: 100 Mbit, half duplex, link ok
MAC Address: 00:05:1E:04:2C:4D

Change the following to the opposite of the above output, in our case change it to 100/Full duplex. 

Note: As per Brocade it is recommended that the eth0 interface be running at 100/Full Duplex (assuming the IP network switch can support it).

DS5000B> ifmodeset eth0

Exercise care when using this command. Forcing the link to an operating mode not supported by the network equipment to which it is attached may result in an inability to communicate with the system through its ethernet interface.

It is recommended that you only use this command from the serial console port.

Are you sure you really want to do this? (yes, y, no, n): [no] y
Proceed with caution.
Auto-negotiate (yes, y, no, n): [no]
Force 100 Mbps / Full Duplex (yes, y, no, n): [no] y
Committing configuration...done.

If the user does NOT know if their IP switch can support 100/Full Duplex, then at this point we can leave the eth0 interface at 100/Full Duplex and test the connectivity. If the connectivity fails, change it back to the original setting.

Change it back to the original duplex setting

DS5000B> ifmodeset eth0

Exercise care when using this command. Forcing the link to an operating mode not supported by the network equipment to which it is attached may result in an inability to communicate with the system through its ethernet interface.

It is recommended that you only use this command from the serial console port.

Are you sure you really want to do this? (yes, y, no, n): [no] y
Proceed with caution.
Auto-negotiate (yes, y, no, n): [no]
Force 100 Mbps / Full Duplex (yes, y, no, n): [no]
Force 100 Mbps / Half Duplex (yes, y, no, n): [no] y
Committing configuration...done.

3. Test Ethernet connectivity.

 Director Switch:

1. Connect to the Active CP via the serial cable.

2. Reset eth0 using one of the options below (options are listed in recommended order from Brocade).

a. Perform hafailover, which will reboot the processor on the current active CP card, but not the switch (non-disruptive to fibre channel).

Check to ensure HA status is enabled and synchronized, perform CP failover.

DCX> hashow
Local CP (Slot 7, CP1): Active, Warm Recovered
Remote CP (Slot 6, CP0): Standby, Healthy
HA enabled, Heartbeat Up, HA State synchronized

Warning: Only If HA status is enabled and synchronized, perform CP failover, otherwise do NOT attempt a failover otherwise you will reboot the switch.

DCX> hafailover
Local CP (Slot 7, CP1): Active, Warm Recovered
Remote CP (Slot 6, CP0): Standby, Healthy
HA enabled, Heartbeat Up, HA State synchronized
Warning: This command is being run on a redundant control processor(CP)
system, and this operation will cause the active CP to reset.
Therefore all existing telnet sessions are required to be restarted.

Are you sure you want to fail over to the standby CP [y/n]? y
Forcing Failover ...

b. Change IP address of the switch and then change it back to the original IP.

Example:

Take note of the original IP address:

DCX> ipaddrshow

SWITCH
Ethernet IP Address: xx.xx.xx.xx
Ethernet Subnetmask: xxx.xxx.xxx.x
Gateway IP Address: xx.xx.xx.x
DHCP: Off

Change the IP settings back to factory default on the active CP, in this case the active CP is CP0

DCX> ipaddrset -cp 0
Host Name [cp0]:
Ethernet IP Address [xx.xx.xx.xx]:xx.xx.xx.xx
Ethernet Subnetmask [xxx.xxx.xxx.x]:xxx.x.x.x
Gateway IP Address [xx.xx.xx.x]:x.x.x.x

Now change it back to the original IP.

DCX> ipaddrset -cp 0
Host Name [cp0]:
Ethernet IP Address [xx.xx.xx.xx]:xx.xx.xx.xx
Ethernet Subnetmask [xxx.x.x.x]:xxx.xxx.xxx.x
Gateway IP Address [x.x.x.x]:xx.xx.xx.x

c. Change the flow control.

Take note of the current flow control.

DCX> ifmodeshow eth0
Link mode: 100 Mbit, half duplex, link ok
MAC Address: 00:05:1E:04:2C:4D

Change the following to the opposite of the above output, in our case change it to 100/Full duplex. 

Note: As per Brocade it is recommended that the eth0 interface be running at 100/Full Duplex (assuming the IP network switch can support it).

DCX> ifmodeset eth0

Exercise care when using this command.  Forcing the link to an operating mode not supported by the network equipment to which it is attached may result in an inability to communicate with the system through its ethernet interface.

It is recommended that you only use this command from the serial console port.

Are you sure you really want to do this? (yes, y, no, n): [no] y
Proceed with caution.
Auto-negotiate (yes, y, no, n): [no]
Force 100 Mbps / Full Duplex (yes, y, no, n): [no] y
Committing configuration...done.

If the customer does NOT know if their IP switch can support 100/Full Duplex, then at this point we can leave the eth0 interface at 100/Full Duplex and test the connectivity.  If the connectivity fails change it back to the original setting.

Change it back to the original duplex setting.

DCX> ifmodeset eth0

Exercise care when using this command.  Forcing the link to an operating mode not supported by the network equipment to which it is attached may result in an inability to communicate with the system through its ethernet interface.

It is recommended that you only use this command from the serial console port.

Are you sure you really want to do this? (yes, y, no, n): [no] y
Proceed with caution.
Auto-negotiate (yes, y, no, n): [no]
Force 100 Mbps / Full Duplex (yes, y, no, n): [no]
Force 100 Mbps / Half Duplex (yes, y, no, n): [no] y
Committing configuration...done.

3. Test Ethernet connectivity.