PowerVault

Last reply by 07-06-2022 Unsolved
Start a Discussion
2 Bronze
2 Bronze
3602

PowerVault MD3200 lockdown recovery and firmware upgrade problem

Hello friends,

I am facing a problem with a MD3200 storage. This unit was not used for some time and all the drives used in it before were removed. I want to use it now but the array is in a problem. It reports "The storage array is in Recovery Mode" . I wanted to clear the configuration but the only option I have there is to upgrade the firmware and when I try to upgrade, it says the firmware in incompatible. Failure number 434.

I tried several versions of firmwares. Current firmware is 7.84.44.60. Whichever firmware I try it says it is not compatible. I have unziped all before trying to uploading to the controller.

What should I do? 

Best Regards.

Replies (29)
1187

Hello Charles,

 

I've managed to power on these controllers normally. It worked everything as it should and I started creating virtual drives. Today I powered it on and left it working but the UPS went out and the controller was powered off. Now when I power it on, one of the controllers has serious errors and I cannot enter the menu via the terminal.

 

can You tell me what could this mean?

 

-=<###>=-
Instantiating /ram as rawFs, device = 0x1
Formatting /ram for DOSFS
Instantiating /ram as rawFs, device = 0x1
Formatting...Retrieved old volume params with %38 confidence:
Volume Parameters: FAT type: FAT32, sectors per cluster 0
0 FAT copies, 0 clusters, 0 sectors per FAT
Sectors reserved 0, hidden 0, FAT sectors 0
Root dir entries 0, sysId (null) , serial number 10000
Label:" " ...
Disk with 1024 sectors of 512 bytes will be formatted with:
Volume Parameters: FAT type: FAT12, sectors per cluster 1
2 FAT copies, 1010 clusters, 3 sectors per FAT
Sectors reserved 1, hidden 0, FAT sectors 6
Root dir entries 112, sysId VXDOS12 , serial number 10000
Label:" " ...
Instantiating /ram as rawFs, device = 0x1
OK.

RTC Error: Real-time clock device is not working

Adding 13884 symbols for standalone.
Length: 0x13c Bytes
Version ver03.0A

 


Reset, Power-Up Diagnostics - Loop 1 of 1
3600 Processor DRAM
01 Data lines Passed
02 Address lines Passed
3300 NVSRAM
01 Data lines Passed
4410 Ethernet 82574 1
01 Register read Passed
02 Register address lines Passed
6D40 Bobcat
02 Flash Test Passed
3700 PLB SRAM
01 Data lines Passed
02 Address lines Passed
6D50 LSISAS2008 IOC 1
01 Register Read Test Passed
02 Register Address Lines Test Passed
03 Register Data Lines Test Passed
3900 Real-Time Clock
01 RT Clock Tick Passed
Diagnostic Manager exited normally.


Current date: 05/18/21 time: 07:56:56

Send <BREAK> for Service Interface or baud rate change
05/18/21-18:09:33 (tRAID): NOTE: Set Powerup State
05/18/21-18:09:33 (tRAID): SOD Sequence is Normal, 0 on controller A
05/18/21-18:09:34 (tRAID): NOTE: Turning on tray summary fault LED
05/18/21-18:09:34 (tRAID): NOTE: SODRebootLoop- Limit:5 Cnt:3
05/18/21-18:09:34 (tRAID): NOTE: Installed Protocols: <MTPs: SAS USB > <ITPs: RDMA ISER IPOIB > <STPs: FCP RmtDMA iSCSI SAS >
05/18/21-18:09:34 (tRAID): NOTE: Required Protocols: <MTPs: SAS > <ITPs: UNK > <STPs: SAS >

Press within 5 seconds: <S> for Service Interface, <BREAK> for baud rate

VxWorks login: 05/18/21-18:09:35 (tRAID): NOTE: DSM: Current revision 7
05/18/21-18:09:35 (tRAID): NOTE: SYMBOL: SYMbolAPI registered.
05/18/21-18:09:36 (tRAID): NOTE: RCBBitmapManager total RPA size = 1778384896
05/18/21-18:09:36 (tRAID): NOTE: init: ioc: 0, PLVersion: 11-075-03-00
05/18/21-18:09:36 (tRAID): NOTE: Initiating Host channel: ioc:1 bringup
05/18/21-18:09:36 (tRAID): NOTE: IOC HW detected, index:0 type:host chan:0 chipID:72 rev:3
Bad File CRC after decompression
05/18/21-18:09:37 (tRAID): WARN: sasReadImageFromFFS: Failed to get IOC firmware from flash
05/18/21-18:09:37 (tRAID): ERROR: Cannot Download ioc:1 chipType:Host channel:0 error:ffffffff
05/18/21-18:09:37 (tRAID): WARN: mpt_do_ioc_recovery: ioc:1 NOT READY WARNING
05/18/21-18:09:37 (tRAID): WARN: *** SAS IOC:1 init Attempt:0 FAILED mpt_do_ioc_recovery or NVRAM update status:0xffffffff DoorBell:x00000000 retry it
05/18/21-18:09:37 (tRAID): NOTE: Initiating Host channel: ioc:1 bringup
05/18/21-18:09:37 (tRAID): NOTE: IOC HW detected, index:0 type:host chan:0 chipID:72 rev:3
Bad File CRC after decompression
05/18/21-18:09:37 (tRAID): WARN: sasReadImageFromFFS: Failed to get IOC firmware from flash
05/18/21-18:09:37 (tRAID): ERROR: Cannot Download ioc:1 chipType:Host channel:0 error:ffffffff
05/18/21-18:09:37 (tRAID): WARN: mpt_do_ioc_recovery: ioc:1 NOT READY WARNING
05/18/21-18:09:37 (tRAID): WARN: *** SAS IOC:1 init Attempt:1 FAILED mpt_do_ioc_recovery or NVRAM update status:0xffffffff DoorBell:x00000000 retry it
05/18/21-18:09:37 (tRAID): NOTE: Initiating Host channel: ioc:1 bringup
05/18/21-18:09:37 (tRAID): NOTE: IOC HW detected, index:0 type:host chan:0 chipID:72 rev:3
Bad File CRC after decompression
05/18/21-18:09:37 (tRAID): WARN: sasReadImageFromFFS: Failed to get IOC firmware from flash
05/18/21-18:09:37 (tRAID): ERROR: Cannot Download ioc:1 chipType:Host channel:0 error:ffffffff
05/18/21-18:09:37 (tRAID): WARN: mpt_do_ioc_recovery: ioc:1 NOT READY WARNING
05/18/21-18:09:37 (tRAID): WARN: *** SAS IOC:1 init Attempt:2 FAILED mpt_do_ioc_recovery or NVRAM update status:0xffffffff DoorBell:x00000000 retry it
05/18/21-18:09:37 (tRAID): NOTE: Initiating Host channel: ioc:1 bringup
05/18/21-18:09:37 (tRAID): NOTE: IOC HW detected, index:0 type:host chan:0 chipID:72 rev:3
Bad File CRC after decompression
05/18/21-18:09:37 (tRAID): WARN: sasReadImageFromFFS: Failed to get IOC firmware from flash
05/18/21-18:09:37 (tRAID): ERROR: Cannot Download ioc:1 chipType:Host channel:0 error:ffffffff
05/18/21-18:09:37 (tRAID): WARN: mpt_dOC:1 did not initialize properly after 305/18/21-18:09:37 (tRAID): PANIC: SAS IOC:1 did not initialize 05/18/21-18:09:37 (tRAID): WARN: *** SAS IOC:1 init Attempt:3 FAILED mpt_do_ioc_recovery or NVRAM update status:0xffffffff DoorBell:x00000000 retry it
properly after 3 retries of mpt_do_ioc_recovery r:0xffffffff

-=<###>=-
Instantiating /ram as rawFs, device = 0x1
Formatting /ram for DOSFS
Instantiating /ram as rawFs, device = 0x1
Formatting...Retrieved old volume params with %38 confidence:
Volume Parameters: FAT type: FAT32, sectors per cluster 0
0 FAT copies, 0 clusters, 0 sectors per FAT
Sectors reserved 0, hidden 0, FAT sectors 0
Root dir entries 0, sysId (null) , serial number 10000
Label:" " ...
Disk with 1024 sectors of 512 bytes will be formatted with:
Volume Parameters: FAT type: FAT12, sectors per cluster 1
2 FAT copies, 1010 clusters, 3 sectors per FAT
Sectors reserved 1, hidden 0, FAT sectors 6
Root dir entries 112, sysId VXDOS12 , serial number 10000
Label:" " ...
Instantiating /ram as rawFs, device = 0x1
OK.

RTC Error: Real-time clock device is not working

Adding 13884 symbols for standalone.
Length: 0x13c Bytes
Version ver03.0A

1173

I also managed to make some reset but one controller remains to be in fault. The display output is now like this:

 

Reset, Power-Up Diagnostics - Loop 1 of 1
3600 Processor DRAM
01 Data lines Passed
02 Address lines Passed
3300 NVSRAM
01 Data lines Passed
4410 Ethernet 82574 1
01 Register read Passed
02 Register address lines Passed
6D40 Bobcat
02 Flash Test Passed
3700 PLB SRAM
01 Data lines Passed
02 Address lines Passed
6D50 LSISAS2008 IOC 1
01 Register Read Test Passed
02 Register Address Lines Test Passed
03 Register Data Lines Test Passed
3900 Real-Time Clock
01 RT Clock Tick Passed
Diagnostic Manager exited normally.


Current date: 05/18/21 time: 08:32:02

Send <BREAK> for Service Interface or baud rate change
05/18/21-18:44:39 (tRAID): NOTE: Set Powerup State
05/18/21-18:44:39 (tRAID): SOD Sequence is Normal, 0 on controller A
05/18/21-18:44:40 (tRAID): NOTE: Turning on tray summary fault LED
05/18/21-18:44:40 (tRAID): NOTE: SODRebootLoop- Limit:5 Cnt:1
05/18/21-18:44:40 (tRAID): NOTE: Installed Protocols: <MTPs: SAS USB > <ITPs: RDMA ISER IPOIB > <STPs: FCP RmtDMA iSCSI SAS >
05/18/21-18:44:40 (tRAID): NOTE: Required Protocols: <MTPs: SAS > <ITPs: UNK > <STPs: SAS >

 

 

After this it stops the network interface and it is not possible to enter any commands.

 

1165

Hello spaxton,

 

I'm sorry to see that. Did you get in touch with your regional support for a backline engineer (L3 as we know them) to work with you on unlocking the controllers?  It was explained to me a couple times by my L3 because I asked a couple of times if there was anything we could do and the explanation was: L3 will have to figure out which controller has the corrupt db lockdown on it and then have to do specific shell user commands in a specific order based on what is revealed.

 

I asked my engineer to take a look at this information  and it appears to be boot looping controller due to internal communication NVRAM issue possibly tied to the corrupt DB lockdown but would not know for sure until the engineer can check the status of the hardware and the corrupt db lockdown.

 

If you have a method you used to get them to boot normally you may go through the process again and check results. I'd recommend to call into support if it happens again.


Dell -Charles R
Social Media and Communities Professional
Dell Technologies | Enterprise Support Services
#IWork4Dell

Did I answer your query? Please click on ‘Accept as Solution’. ‘Kudo’ the posts you like!

1162

Hello Charles,

 

Thanks for reply.

The method I used to clear the lockdown at first moment was this:

 

Instead of typing <S> when prompted, it was needed to hit ESC after what the system will prompt vxWorks login. By entering shellUsr as username and DF4m/2> as a password, the new shell appears where I entered lemClearLockdown command and everything worked after that. MSDM displayed that the controller needed some attention but I didn't have much time last night to check why and I was very excited that it works. I think I made a mistake because I didn't use the command sysWipeAllConfigData and then start from scratch but I entered this command on a working controller today after the problem appeared and then the other one also started again but still displaying errors.

So,

Press Ctrl-Break
Instead of hitting S press Esc
then it will prompt vxWorks login:
Username: shellUsr
Passsword: DF4m/2>

 

How to find local L3 support for DeLL?

 

Respectfully,

1160

I don’t see direct support there but it has partner contacts. https://dell.to/3hyiPJZ


Thanks,

DELL-Josh Cr
Social Media and Communities Professional
Dell Technologies | Enterprise Support Services
#IWork4Dell

Did I answer your query? Please click on ‘Accept as Solution’. ‘Kudo’ the posts you like!

1155

Thanks for the info. I don't think they can resolve it. I know this company and I will ask them.

By the way, the controller which reported corrupted database still works. The one which was it reboot lockdown is now again in reboot lockdown because it has some bootloop problem.

1043

Hello Josh,

 

As I said, no local service is able to resolve the issue. Can you help me by providing all possible commands for vx console?

Best Regards.

1038

Hi,

Unfortunately, I am not able to give the commands as they are confidential, you may just need to replace the controllers.


Thanks,

DELL-Josh Cr
Social Media and Communities Professional
Dell Technologies | Enterprise Support Services
#IWork4Dell

Did I answer your query? Please click on ‘Accept as Solution’. ‘Kudo’ the posts you like!

1034

Hi Josh. Thanks for the info.

I bought another controller already but I want this one alive as well. Is there any paid remote service which could help with this?

Best Regards.

1028

Hello spaxton,

 

You can contact Dell in your region and inquire about Post Standard Support and they can give you a quote for  it.


Dell -Charles R
Social Media and Communities Professional
Dell Technologies | Enterprise Support Services
#IWork4Dell

Did I answer your query? Please click on ‘Accept as Solution’. ‘Kudo’ the posts you like!

Latest Solutions
Top Contributor