How to recover DELL EMC Networking N2000 and N3000 from inconsistent state while downgrading firmware

This article explains how to recover the switch from inconsistent state while doing a downgrade of firmware version in Dell N2000 and N3000.
 

Requirement

Console access to the switch is a must.  You will need physical access to the switch console port in order to complete the process.  You will need a personal computer with a serial port or a USB to serial adapter and the serial to RJ45 cable that comes with the switch.

Cause

The switch is stuck in an inconsistent state due to incompatible CPLD version.
A switch goes into an inconsistent state, when the CPLD installed is a newer version than the firmware, to be downgraded.  Anytime you downgrade the firmware you must consider the CPLD version and downgrade it if needed.
For example, firmware version 6.0.1.3 requires CPLD version 13.
If a customer is downgrading the switch from a higher firmware version 6.6.0.2 with CPLD 20 directly to 6.0.1.3, without following the proper downgrade procedure, the switch may get stuck in an inconsistent state.
You can find the CPLD version compatible for the firmware version in the release notes provided with the corresponding firmware download file.

 

How to know if a switch is stuck in an inconsistent state

While booting the switch a log message is seen mentioning the "system is restarting due to inconsistent state".

Sample console logs

Below is a sample console log of switch while booting which shows the switch is in an inconsistent state (Omitting complete boot sequence for brevity).

Extracting Operational Code from .stk file...2
done.
Loading Operational Code...done.
Decompressing Operational Code...done.
Uncompressing apps.lzma
SyncDB Running...
File: stm_api.c, Line: 62, Error 0 (0x0)
<185> May 14 01:38:40 0.0.0.0-0 General[fp_main_task]: stm_api.c(62) 1 %% Error 0 (0x0)
DMA pool size: 16777216
AXI unit 0: Dev 0xb342, Rev 0x01, Chip BCM56342_A0, Driver BCM56340_A0
In hpcLocalUnitTypeIdGet:611, could not find the board
In sysapiHpcStmUnitDbEntryLocalUnitIndexGet:342, could not find the board
The system is restarting due to the inconsistent state 0 in file: stm_api.c line 62                                                                                                                       
 <184> May 14 01:38:43 0.0.0.0-1 LOG[fp_main_task]: log_cnfgr.c(1357) 3 %% The system is restarting due to inconsistent state.                                 
Can't get unit number! 
Switching software SIGSEGV Handler
This build was configured to copy this crash information to a file.
Symbols already loaded.
starting pid 12
syncing filesystems....This may take a few moments
Rebooting system!
The system is going down NOW!
Sent SIGTERM to all processes
Sent SIGKILL to all processes
Requesting system reboot

 

Solution

Boot the switch in correct firmware version (compatible with the CPLD version) and proceed with proper downgrade procedure.

Boot the switch in backup partition. There is a chance that customer has just copied the older firmware to backup partition and is trying to reboot switch in backup partition.

Dell Networking Boot Options
============================
Select a menu option within 3 seconds or the Operational Code will start automatically...
1 - Start Operational Code
2 - Display Boot Menu


When the boot menu appears, select 2 "Display Boot Menu".

Select (1, 2) #2
Boot Main Menu
==============
1 - Start Operational Code
2 - Select Baud Rate
3 - Retrieve Logs
4 - Load New Operational Code
5 - Display Operational Code Details
9 - Reboot
10 - Restore Configuration to Factory Defaults
11 - Activate Backup Image
12 - Start Password Recovery

From Boot Main Menu select 11 "Activate Backup Image".

Enter Choice# 11

After this, reboot the switch with option 9 "Reboot".

Boot Main Menu
==============
1 - Start Operational Code
2 - Select Baud Rate
3 - Retrieve Logs
4 - Load New Operational Code
5 - Display Operational Code Details
9 - Reboot
10 - Restore Configuration to Factory Defaults
11 - Activate Backup Image
12 - Start Password Recovery

Enter Choice# 9

Option 2
If the 1^st option did not work, try to load firmware using TFTP/XMODEM. Follow the procedure in upgrade guide for switch recovery.

• If the previous firmware version is known, upgrade to that version.
• If the firmware version is unknown, upgrade it to the latest firmware version available

Once the switch recovers follow the correct downgrade procedure provided in the upgrade guide given with the firmware download file. Upgrade to intermediate firmware release N3000_N2000vB.6.3.2.stk and downgrade CPLD to 13.



 

Boot the switch in backup partition. There is a chance that customer has just copied the older firmware to backup partition and is trying to reboot switch in backup partition.

Option 2
If the 1^st option did not work, try to load firmware using TFTP/XMODEM. Follow the procedure in upgrade guide for switch recovery.