PowerVault MD3200 lockdown recovery and firmware upgrade problem

Hello @spaxton,

Ok, so the error message says "The storage array is in Recovery Mode". This normally is caused by a lockdown status.

What you can do is to use a serial cable and connect the serial cable to the Controller. Use putty to see output. Reboot it and capture the boot text. That will offer more information about the error.

You can also use the command lemClearLockdown via the serial cable connected to the affected controller and that should clear the error.  

Regards.

Hello ,

After a long time after I gave up on this, I decided to try searching the web and, believe it or not, this post was offered among the search results. I gave up back then because it looked too complicated to me...

Unfortunately, I wasn't able to do this because these controllers won't accept commands. I get some response when I press CTRL + Break but pressing S gives only black background terminal. Please check my screenshots from both controllers. They give different errors and both of them does not respond when I enter S. Serial port was configured 115200-8-n-1.

What am I missing? Can You help?

Best Regards,

Hello spaxton,

Do you know if the firmware is the same on both?

Have you updated the firmware?

Can you try unseating one of the controller and try access the console with just the other controller?

Please let me know how it goes.

Hello Spaxton,

I'm sorry to see that. Unfortunately there are a lot of factors for this particular issue. At this point would need to engage backline support.  We can only use the shelluser login on the backline support.  Front line techs such as myself do not have access to these. They will have to figure out which controller has the corrupt db lockdown on it and then have to do specific shell user commands in a specific order based on what is revealed.  It is recommended to call in 1-800-945-3355 (for US/Canada) to set up a normal SR case.

Are you still under warranty?  

What country are you located?

Hello Charles,

Thank You very much for replying,

Both firmware are the same. I did not upgrade the firmware

I tried unseating and accessing just one of them, but it reads the same state as on pictures.

Do you have any other ideas?

Respectfully, 

Spaxton

Charles, thank You very much!

The device warranty has expired and I am located in Montenegro. The device was bought on auction with a bunch of servers with it. Some other devices in the bunch were also faulty but it was easy to solve. This one would be great to complete a virtualization platform for testing purposes but it seems I will have to bin it if no support can be given.

Anyway, thanks again.

With kind regards from Montenegro...

By the way...

I don't get it why I can't just reset it to factory defaults? I don't care about the previous configuration.

Ok, Many thanks for help. I will try the support link.

Hello Spaxton,

I provided the recommended guidance, but if you want, you can try to delete things in MSDM to see if it will allow you access, but typically it locks down the controllers so you can't make changes until it is cleared. It will require backend support engaged on a screen share session to work through it.

You may also contact support https://dell.to/2QuvDGm  and inquire about Post Standard Support.

Hello Charles,

I've managed to power on these controllers normally. It worked everything as it should and I started creating virtual drives. Today I powered it on and left it working but the UPS went out and the controller was powered off. Now when I power it on, one of the controllers has serious errors and I cannot enter the menu via the terminal.

can You tell me what could this mean?

-=<###>=-
Instantiating /ram as rawFs, device = 0x1
Formatting /ram for DOSFS
Instantiating /ram as rawFs, device = 0x1
Formatting...Retrieved old volume params with %38 confidence:
Volume Parameters: FAT type: FAT32, sectors per cluster 0
0 FAT copies, 0 clusters, 0 sectors per FAT
Sectors reserved 0, hidden 0, FAT sectors 0
Root dir entries 0, sysId (null) , serial number 10000
Label:" " ...
Disk with 1024 sectors of 512 bytes will be formatted with:
Volume Parameters: FAT type: FAT12, sectors per cluster 1
2 FAT copies, 1010 clusters, 3 sectors per FAT
Sectors reserved 1, hidden 0, FAT sectors 6
Root dir entries 112, sysId VXDOS12 , serial number 10000
Label:" " ...
Instantiating /ram as rawFs, device = 0x1
OK.

RTC Error: Real-time clock device is not working

Adding 13884 symbols for standalone.
Length: 0x13c Bytes
Version ver03.0A

Reset, Power-Up Diagnostics - Loop 1 of 1
3600 Processor DRAM
01 Data lines Passed
02 Address lines Passed
3300 NVSRAM
01 Data lines Passed
4410 Ethernet 82574 1
01 Register read Passed
02 Register address lines Passed
6D40 Bobcat
02 Flash Test Passed
3700 PLB SRAM
01 Data lines Passed
02 Address lines Passed
6D50 LSISAS2008 IOC 1
01 Register Read Test Passed
02 Register Address Lines Test Passed
03 Register Data Lines Test Passed
3900 Real-Time Clock
01 RT Clock Tick Passed
Diagnostic Manager exited normally.

Current date: 05/18/21 time: 07:56:56

Send for Service Interface or baud rate change
05/18/21-18:09:33 (tRAID): NOTE: Set Powerup State
05/18/21-18:09:33 (tRAID): SOD Sequence is Normal, 0 on controller A
05/18/21-18:09:34 (tRAID): NOTE: Turning on tray summary fault LED
05/18/21-18:09:34 (tRAID): NOTE: SODRebootLoop- Limit:5 Cnt:3
05/18/21-18:09:34 (tRAID): NOTE: Installed Protocols:
05/18/21-18:09:34 (tRAID): NOTE: Required Protocols:

Press within 5 seconds: for Service Interface, for baud rate

VxWorks login: 05/18/21-18:09:35 (tRAID): NOTE: DSM: Current revision 7
05/18/21-18:09:35 (tRAID): NOTE: SYMBOL: SYMbolAPI registered.
05/18/21-18:09:36 (tRAID): NOTE: RCBBitmapManager total RPA size = 1778384896
05/18/21-18:09:36 (tRAID): NOTE: init: ioc: 0, PLVersion: 11-075-03-00
05/18/21-18:09:36 (tRAID): NOTE: Initiating Host channel: ioc:1 bringup
05/18/21-18:09:36 (tRAID): NOTE: IOC HW detected, index:0 type:host chan:0 chipID:72 rev:3
Bad File CRC after decompression
05/18/21-18:09:37 (tRAID): WARN: sasReadImageFromFFS: Failed to get IOC firmware from flash
05/18/21-18:09:37 (tRAID): ERROR: Cannot Download ioc:1 chipType:Host channel:0 error:ffffffff
05/18/21-18:09:37 (tRAID): WARN: mpt_do_ioc_recovery: ioc:1 NOT READY WARNING
05/18/21-18:09:37 (tRAID): WARN: *** SAS IOC:1 init Attempt:0 FAILED mpt_do_ioc_recovery or NVRAM update status:0xffffffff DoorBell:x00000000 retry it
05/18/21-18:09:37 (tRAID): NOTE: Initiating Host channel: ioc:1 bringup
05/18/21-18:09:37 (tRAID): NOTE: IOC HW detected, index:0 type:host chan:0 chipID:72 rev:3
Bad File CRC after decompression
05/18/21-18:09:37 (tRAID): WARN: sasReadImageFromFFS: Failed to get IOC firmware from flash
05/18/21-18:09:37 (tRAID): ERROR: Cannot Download ioc:1 chipType:Host channel:0 error:ffffffff
05/18/21-18:09:37 (tRAID): WARN: mpt_do_ioc_recovery: ioc:1 NOT READY WARNING
05/18/21-18:09:37 (tRAID): WARN: *** SAS IOC:1 init Attempt:1 FAILED mpt_do_ioc_recovery or NVRAM update status:0xffffffff DoorBell:x00000000 retry it
05/18/21-18:09:37 (tRAID): NOTE: Initiating Host channel: ioc:1 bringup
05/18/21-18:09:37 (tRAID): NOTE: IOC HW detected, index:0 type:host chan:0 chipID:72 rev:3
Bad File CRC after decompression
05/18/21-18:09:37 (tRAID): WARN: sasReadImageFromFFS: Failed to get IOC firmware from flash
05/18/21-18:09:37 (tRAID): ERROR: Cannot Download ioc:1 chipType:Host channel:0 error:ffffffff
05/18/21-18:09:37 (tRAID): WARN: mpt_do_ioc_recovery: ioc:1 NOT READY WARNING
05/18/21-18:09:37 (tRAID): WARN: *** SAS IOC:1 init Attempt:2 FAILED mpt_do_ioc_recovery or NVRAM update status:0xffffffff DoorBell:x00000000 retry it
05/18/21-18:09:37 (tRAID): NOTE: Initiating Host channel: ioc:1 bringup
05/18/21-18:09:37 (tRAID): NOTE: IOC HW detected, index:0 type:host chan:0 chipID:72 rev:3
Bad File CRC after decompression
05/18/21-18:09:37 (tRAID): WARN: sasReadImageFromFFS: Failed to get IOC firmware from flash
05/18/21-18:09:37 (tRAID): ERROR: Cannot Download ioc:1 chipType:Host channel:0 error:ffffffff
05/18/21-18:09:37 (tRAID): WARN: mpt_dOC:1 did not initialize properly after 305/18/21-18:09:37 (tRAID): PANIC: SAS IOC:1 did not initialize 05/18/21-18:09:37 (tRAID): WARN: *** SAS IOC:1 init Attempt:3 FAILED mpt_do_ioc_recovery or NVRAM update status:0xffffffff DoorBell:x00000000 retry it
properly after 3 retries of mpt_do_ioc_recovery r:0xffffffff

-=<###>=-
Instantiating /ram as rawFs, device = 0x1
Formatting /ram for DOSFS
Instantiating /ram as rawFs, device = 0x1
Formatting...Retrieved old volume params with %38 confidence:
Volume Parameters: FAT type: FAT32, sectors per cluster 0
0 FAT copies, 0 clusters, 0 sectors per FAT
Sectors reserved 0, hidden 0, FAT sectors 0
Root dir entries 0, sysId (null) , serial number 10000
Label:" " ...
Disk with 1024 sectors of 512 bytes will be formatted with:
Volume Parameters: FAT type: FAT12, sectors per cluster 1
2 FAT copies, 1010 clusters, 3 sectors per FAT
Sectors reserved 1, hidden 0, FAT sectors 6
Root dir entries 112, sysId VXDOS12 , serial number 10000
Label:" " ...
Instantiating /ram as rawFs, device = 0x1
OK.

RTC Error: Real-time clock device is not working

Adding 13884 symbols for standalone.
Length: 0x13c Bytes
Version ver03.0A

I also managed to make some reset but one controller remains to be in fault. The display output is now like this:

Reset, Power-Up Diagnostics - Loop 1 of 1
3600 Processor DRAM
01 Data lines Passed
02 Address lines Passed
3300 NVSRAM
01 Data lines Passed
4410 Ethernet 82574 1
01 Register read Passed
02 Register address lines Passed
6D40 Bobcat
02 Flash Test Passed
3700 PLB SRAM
01 Data lines Passed
02 Address lines Passed
6D50 LSISAS2008 IOC 1
01 Register Read Test Passed
02 Register Address Lines Test Passed
03 Register Data Lines Test Passed
3900 Real-Time Clock
01 RT Clock Tick Passed
Diagnostic Manager exited normally.

Current date: 05/18/21 time: 08:32:02

Send for Service Interface or baud rate change
05/18/21-18:44:39 (tRAID): NOTE: Set Powerup State
05/18/21-18:44:39 (tRAID): SOD Sequence is Normal, 0 on controller A
05/18/21-18:44:40 (tRAID): NOTE: Turning on tray summary fault LED
05/18/21-18:44:40 (tRAID): NOTE: SODRebootLoop- Limit:5 Cnt:1
05/18/21-18:44:40 (tRAID): NOTE: Installed Protocols:
05/18/21-18:44:40 (tRAID): NOTE: Required Protocols:

After this it stops the network interface and it is not possible to enter any commands.

Hello spaxton,

I'm sorry to see that. Did you get in touch with your regional support for a backline engineer (L3 as we know them) to work with you on unlocking the controllers?  It was explained to me a couple times by my L3 because I asked a couple of times if there was anything we could do and the explanation was: L3 will have to figure out which controller has the corrupt db lockdown on it and then have to do specific shell user commands in a specific order based on what is revealed.

I asked my engineer to take a look at this information  and it appears to be boot looping controller due to internal communication NVRAM issue possibly tied to the corrupt DB lockdown but would not know for sure until the engineer can check the status of the hardware and the corrupt db lockdown.

If you have a method you used to get them to boot normally you may go through the process again and check results. I'd recommend to call into support if it happens again.

Hello Charles,

Thanks for reply.

The method I used to clear the lockdown at first moment was this:

Instead of typing when prompted, it was needed to hit ESC after what the system will prompt vxWorks login. By entering shellUsr as username and DF4m/2> as a password, the new shell appears where I entered lemClearLockdown command and everything worked after that. MSDM displayed that the controller needed some attention but I didn't have much time last night to check why and I was very excited that it works. I think I made a mistake because I didn't use the command sysWipeAllConfigData and then start from scratch but I entered this command on a working controller today after the problem appeared and then the other one also started again but still displaying errors.

So,

Press Ctrl-Break
Instead of hitting S press Esc
then it will prompt vxWorks login:
Username: shellUsr
Passsword: DF4m/2>

How to find local L3 support for DeLL?

Respectfully,

I don’t see direct support there but it has partner contacts. https://dell.to/3hyiPJZ

Thanks for the info. I don't think they can resolve it. I know this company and I will ask them.

By the way, the controller which reported corrupted database still works. The one which was it reboot lockdown is now again in reboot lockdown because it has some bootloop problem.