Unsolved
This post is more than 5 years old
9 Posts
0
6569
Dell PowerVault MD3400 warning massege: "controller is locked down due to detected corruption in the primary database..."
Hello dear friends! I have Dell PowerVault MD3400 with two controllers. One of them has a mistake.
MDSM doesn't see it, though one of controllers gets ip adress from DHCP server.
I need your ideas and advises.
Here is a log from putty connection with serial cable:
Reset, Power-Up Diagnostics - Loop 1 of 1
3600 Processor DRAM
01 Data lines Passed
02 Address lines Passed
3300 NVSRAM
01 Data lines Passed
3900 Real-Time Clock
01 RT Clock Tick Passed
Diagnostic Manager exited normally.
08/21/17-11:41:53 (tNetCfgInit): NOTE: eth0: LinkDown event
08/21/17-11:41:53 (tNetCfgInit): NOTE: eth1: LinkDown event
08/21/17-11:41:53 (tNetCfgInit): NOTE: Network Ready
08/21/17-11:41:55 (tNet0): NOTE: eth1: LinkUp event
Current date: 08/21/17 time: 19:37:38
Send for Service Interface or baud rate change
08/21/17-11:42:10 (tRAID): NOTE: Set Powerup State
08/21/17-11:42:11 (tRAID): SOD Sequence is Normal, 0 on controller B
08/21/17-11:42:11 (tRAID): NOTE: loading flash file: ECTDell
08/21/17-11:42:11 (tRAID): NOTE: unloading flash file: ECTDell
08/21/17-11:42:11 (tRAID): NOTE: Turning on tray summary fault LED
08/21/17-11:42:11 (tRAID): NOTE: Installed Protocols:
08/21/17-11:42:11 (tRAID): NOTE: Required Protocols:
08/21/17-11:42:13 (tRAID): NOTE: DSM: Current revision 7
08/21/17-11:42:13 (tRAID): NOTE: SYMBOL: SYMbolAPI registered.
08/21/17-11:42:14 (tRAID): NOTE: RCBBitmapManager total RPA size = 3656384512
08/21/17-11:42:14 (tRAID): NOTE: PCI DevIdl 0x1 is GEN3 MPT Adapter - Host side
08/21/17-11:42:14 (tRAID): NOTE: PCI DevIdl 0x2 is GEN3 MPT Adapter - Host side
08/21/17-11:42:14 (tRAID): NOTE: init: ioc: 0, PLVersion: 02-075-00-00
08/21/17-11:42:14 (tRAID): NOTE: init call sasConfigForTbm
Change TBM to Ext Mode ...
08/21/17-11:42:15 (tRAID): NOTE: Enable TBM Ext Mode OK
Cable Management is NOT initialized. Code: x00000000
08/21/17-11:42:15 (tRAID): NOTE: Initiating Host channel: ioc:1 bringup
08/21/17-11:42:15 (tRAID): WARN: Check/Do IOC FW download chanIdx 0 isHost=1
08/21/17-11:42:15 (tRAID): NOTE: IOC HW detected, index:0 type:host chan:0 chipID:97 rev:2
08/21/17-11:42:17 (IOSched): NOTE: SAS Expander Added: expDevHandle:x9 enclHandle:x2 numPhys:30 port:0 ioc:0 channel:1 numEntries:22
08/21/17-11:42:18 (tRAID): NOTE: IOC:1 Firmware Version: 02-75-00-00
08/21/17-11:42:19 (tRAID): NOTE: setIOCPhySettings Write the page
08/21/17-11:42:20 (tRAID): NOTE: Configuring IOC Diagnostic Trace for ioc:1
08/21/17-11:42:20 (tRAID): NOTE: Diagnostic Trace Post message successful
08/21/17-11:42:20 (tRAID): NOTE: IOC Snapshot Post message successful
08/21/17-11:42:20 (tRAID): NOTE: Initiating Host channel: ioc:2 bringup
08/21/17-11:42:20 (tRAID): WARN: Check/Do IOC FW download chanIdx 2 isHost=1
08/21/17-11:42:20 (tRAID): NOTE: IOC HW detected, index:2 type:host chan:2 chipID:97 rev:2
08/21/17-11:42:22 (tRAID): NOTE: IOC:2 Firmware Version: 02-75-00-00
08/21/17-11:42:24 (tRAID): NOTE: setIOCPhySettings Write the page
08/21/17-11:42:25 (tRAID): NOTE: Configuring IOC Diagnostic Trace for ioc:2
08/21/17-11:42:25 (tRAID): NOTE: Diagnostic Trace Post message successful
08/21/17-11:42:25 (tRAID): NOTE: IOC Snapshot Post message successful
08/21/17-11:42:26 (tSasExpChk): NOTE: Local Expander Firmware Version: 00.00.90.10
08/21/17-11:42:26 (tSasExpChk): NOTE: Enabling this controller's expander phys, channel=1
08/21/17-11:42:26 (sasEnPhys): NOTE: sasEnableDrivePhysTask: 12 phys enabled on channel 1
08/21/17-11:42:26 (tSasDiscCom): NOTE: SAS: Initial Discovery Complete Time: 43 seconds since last power on/reset, 15 seconds since sas instantiated
08/21/17-11:42:26 (tRAID): NOTE: usbd: discovered device, drive:0
08/21/17-11:42:26 (tRAID): NOTE: CrushMemoryPoolMgr: platform and memory (CPU MemSize:4096), adjusted allocating size to 65536 for CStripes
08/21/17-11:42:27 (tRAID): ERROR: FTDC metadata reinit: invalid, moved or RPA size changed
08/21/17-11:42:27 (tRAID): NOTE: DQ FTDC: Reinitialized DQ Trace History metadata: signature was invalid
08/21/17-11:42:28 (tRAID): SOD: Instantiation Phase Complete
08/21/17-11:42:27 (tRAID): NOTE: FTDC: Deferring format string copies due to pending restore
08/21/17-11:42:27 (tRAID): WARN: sodLockdownCheck: sodSequence update: CtlrLockdown
08/21/17-11:42:27 (tRAID): WARN: No attempt made to open Inter-Controller Communication Channels 0-0
08/21/17-11:42:27 (tRAID): NOTE: LockMgr Role is Master
08/21/17-11:42:28 (tRAID): NOTE: WWN baseName 000200a0-98558894 (valid==>SigMatch)
08/21/17-11:42:28 (tRAID): NOTE: spmEarlyData: No data available
08/21/17-11:42:29 (tRAID): SOD: Pre-Initialization Phase Complete
08/21/17-11:42:51 (ssmSpin): NOTE: FTDC: Freeze for pending restore from backup
08/21/17-11:42:53 (tRAID): WARN: CmgrLockdownException: CORRUPT DBM DATABASE DETECTED - LOCKDOWN Type=6
08/21/17-11:42:53 (tRAID): WARN: ACS: testCommunication: Domi Exception caught: IconSendInfeasibleException Error
08/21/17-11:42:53 (tRAID): NOTE: ACS: autoCodeSync(): Process start. Comm Status: 0
08/21/17-11:42:53 (tRAID): WARN: ACS: autoCodeSync(): Skipped since alt not communicating.
08/21/17-11:42:54 (tRAID): SOD: Code Synchronization Initialization Phase Complete
08/21/17-11:42:54 (tRAID): WARN: Lockdown case-DB CORRUPT. NVPS did not update from Sstor.
08/21/17-11:42:55 (tRAID): SOD: Initialization Phase Complete
==============================================
Title: Disk Array Controller
Copyright 2013-2014 NetApp, Inc. All Rights Reserved.
Name: RC
Version: 08.10.05.60
Date: 02/21/2014
Time: 13:48:14 CST
Models: 2701
Manager: devmgr.v1110api36.Manager
==============================================
08/21/17-11:42:54 (tRAID): NOTE: SYMbol available
08/21/17-11:42:54 (tRAID): WARN: *********************************************************
08/21/17-11:42:54 (tRAID): WARN: *********************************************************
08/21/17-11:42:54 (tRAID): WARN: ** **
08/21/17-11:42:54 (tRAID): WARN: ** WARNING!!!!!! **
08/21/17-11:42:54 (tRAID): WARN: ** **
08/21/17-11:42:54 (tRAID): WARN: ** This controller is locked down due to detected **
08/21/17-11:42:54 (tRAID): WARN: ** corruption in the primary database. **
08/21/17-11:42:54 (tRAID): WARN: ** In order to prevent any configuration loss, the **
08/21/17-11:42:54 (tRAID): WARN: ** backed up database must be restored. **
08/21/17-11:42:54 (tRAID): WARN: ** **
08/21/17-11:42:54 (tRAID): WARN: ** **
08/21/17-11:42:54 (tRAID): WARN: *********************************************************
08/21/17-11:42:54 (tRAID): WARN: *********************************************************
08/21/17-11:42:55 (tRAID): SOD: sodComplete Notification Complete
08/21/17-11:42:56 (tRAID): sodMain CtlrLockdown sequence finished, elapsed time = 45 seconds
08/21/17-11:42:56 (tRAID): sodMain complete
08/21/17-11:42:56 (tRAID): WARN: SOD lockdown sequence complete - suspending...
DELL-Sam L
Moderator
Moderator
•
6.9K Posts
0
August 22nd, 2017 11:00
Hello Hasan Y,
Did you recently do a firmware upgrade on your MD3400? Was anything replaced recently? Was there a power outage recently? I will send you a private message & if you can reply back with the service tag of your MD3400 I can look into this & see what is going to need to be done to resolve your issue.
Please let us know if you have any other questions.
Hasan Y.
9 Posts
0
August 22nd, 2017 22:00
Hello Sam! Thanks for your answer.
I just pulled out all 12HDDs for other project and put 12 other new empty hard drives like Dell 2TB. When I turned on, controller switched in Lock-Down mode.
Where can I write my ST number? Will u give me your e-mail or write me a private massege?
DELL-Sam L
Moderator
Moderator
•
6.9K Posts
0
August 23rd, 2017 10:00
Hello Hasan Y,
I sent an email to the address that you used in your profile. If you can reply back with your service tag that would be great.
Please let us know if you have any other questions.
Hasan Y.
9 Posts
0
August 23rd, 2017 21:00
Hello, Sam. Sorry, I didn't check my mail yesterday. I have replyed to your massege today.
Good luck!
Kashifsmb
5 Posts
0
May 19th, 2018 02:00
Dear Sam
i have same problem can you plese help below error i have seen on bootup
Instantiating /ram as rawFs, device = 0x1
Formatting /ram for DOSFS
Instantiating /ram as rawFs, device = 0x1
Formatting...Retrieved old volume params with %38 confidence:
Volume Parameters: FAT type: FAT32, sectors per cluster 0
0 FAT copies, 0 clusters, 0 sectors per FAT
Sectors reserved 0, hidden 0, FAT sectors 0
Root dir entries 0, sysId (null) , serial number f10000
Label:" " ...
Disk with 1024 sectors of 512 bytes will be formatted with:
Volume Parameters: FAT type: FAT12, sectors per cluster 1
2 FAT copies, 1010 clusters, 3 sectors per FAT
Sectors reserved 1, hidden 0, FAT sectors 6
Root dir entries 112, sysId VXDOS12 , serial number f10000
Label:" " ...
RTC Error: Real-time clock device is not working
OK.
Adding 14607 symbols for standalone.
05/19/18-09:00:16 (tSystem): ERROR: FPGA FW is out of date
"Rhone03 rev17" currently in use
"Rhone03 rev20" available for update
05/19/18-09:00:20 (tNetCfgInit): NOTE: eth0: LinkUp event
05/19/18-09:00:20 (tNetCfgInit): NOTE: Network Ready
Current date: 05/19/18 time: 09:00:41
Send for Service Interface or baud rate change
05/19/18-09:00:41 (tRAID): SOD Sequence is Normal, 0 on controller A
05/19/18-09:00:41 (tRAID): NOTE: loading flash file: ECTDell
05/19/18-09:00:41 (tRAID): NOTE: unloading flash file: ECTDell
05/19/18-09:00:42 (tRAID): NOTE: I2C transaction returned 0x041aad00
05/19/18-09:00:42 (tRAID): NOTE: I2C transaction returned 0x041aad00
05/19/18-09:00:42 (tRAID): NOTE: I2C transaction returned 0x041aad00
05/19/18-09:00:42 (tRAID): NOTE: I2C transaction returned 0x041aad00
05/19/18-09:00:42 (tRAID): NOTE: I2C transaction returned 0x041aad00
05/19/18-09:00:42 (tRAID): NOTE: I2C transaction returned 0x041aad00
05/19/18-09:00:42 (tRAID): ERROR: Both midplane VPD ERROMS are bad
ERROR: Remap entry not found for ssid=0x0e.
05/19/18-09:00:42 (tRAID): NOTE: Turning on tray summary fault LED
05/19/18-09:00:42 (tRAID): NOTE: SODRebootLoop- Limit:5 Cnt:4
05/19/18-09:00:42 (tRAID): WARN: In getExpectedMdplVpdData, unknown SubsystemID of 14 was found - may result in controller lockdown
05/19/18-09:00:42 (tRAID): NOTE: I2C transaction returned 0x041aad00
05/19/18-09:00:42 (tRAID): ERROR: HCVH - Lockdown Controller: SBB VPD Checksum Validation failed
05/19/18-09:00:42 (tRAID): WARN: Suspending Ctlr: "SBB_Acess_Chksum_Failed"
05/19/18-09:00:42 (tRAID): NOTE: Flashing tray summary fault LED
05/19/18-09:00:42 (tRAID): WARN: Controller entering ClientErr lockdown state...
DELL-Sam L
Moderator
Moderator
•
6.9K Posts
0
May 21st, 2018 07:00
Hello Kashifsmb,
Based on the error that you have listed you will need to clear the lockdown using the VXshell. Now we can’t provide you the password to the VXshell as that is reserved to tech support only. If you are able to access the VXshell then you would need to run the "lemClearLockdown” to remove the lockdown and have the controller boot normally.
Please let us know if you have any other questions.
Kashifsmb
5 Posts
0
May 21st, 2018 21:00
Dear SAM,
i have run cmd lemclearlockdown but still same as below,
-> lemClearLockdown
-=<###>=-
Instantiating /ram as rawFs, device = 0x1
Formatting /ram for DOSFS
Instantiating /ram as rawFs, device = 0x1
Formatting...Retrieved old volume params with %38 confidence:
Volume Parameters: FAT type: FAT32, sectors per cluster 0
0 FAT copies, 0 clusters, 0 sectors per FAT
Sectors reserved 0, hidden 0, FAT sectors 0
Root dir entries 0, sysId (null) , serial number f10000
Label:" " ...
Disk with 1024 sectors of 512 bytes will be formatted with:
Volume Parameters: FAT type: FAT12, sectors per cluster 1
2 FAT copies, 1010 clusters, 3 sectors per FAT
Sectors reserved 1, hidden 0, FAT sectors 6
Root dir entries 112, sysId VXDOS12 , serial number f10000
Label:" " ...
RTC Error: Real-time clock device is not working
OK.
Adding 14607 symbols for standalone.
05/22/18-03:07:49 (tSystem): ERROR: FPGA FW is out of date
"Rhone03 rev17" currently in use
"Rhone03 rev20" available for update
05/22/18-03:07:53 (tNetCfgInit): NOTE: eth0: LinkUp event
05/22/18-03:07:53 (tNetCfgInit): NOTE: Network Ready
Current date: 05/22/18 time: 03:08:14
Send for Service Interface or baud rate change
05/22/18-03:08:14 (tRAID): SOD Sequence is Normal, 0 on controller A
05/22/18-03:08:14 (tRAID): NOTE: loading flash file: ECTDell
05/22/18-03:08:14 (tRAID): NOTE: unloading flash file: ECTDell
05/22/18-03:08:15 (tRAID): NOTE: I2C transaction returned 0x041aad00
05/22/18-03:08:15 (tRAID): NOTE: I2C transaction returned 0x041aad00
05/22/18-03:08:15 (tRAID): NOTE: I2C transaction returned 0x041aad00
05/22/18-03:08:15 (tRAID): NOTE: I2C transaction returned 0x041aad00
05/22/18-03:08:15 (tRAID): NOTE: I2C transaction returned 0x041aad00
05/22/18-03:08:15 (tRAID): NOTE: I2C transaction returned 0x041aad00
05/22/18-03:08:15 (tRAID): ERROR: Both midplane VPD ERROMS are bad
ERROR: Remap entry not found for ssid=0x0e.
05/22/18-03:08:15 (tRAID): NOTE: Turning on tray summary fault LED
05/22/18-03:08:15 (tRAID): WARN: In getExpectedMdplVpdData, unknown SubsystemID of 14 was found - may result in controller lockdown
05/22/18-03:08:15 (tRAID): NOTE: I2C transaction returned 0x041aad00
05/22/18-03:08:15 (tRAID): ERROR: HCVH - Lockdown Controller: SBB VPD Checksum Validation failed
05/22/18-03:08:15 (tRAID): WARN: Suspending Ctlr: "SBB_Acess_Chksum_Failed"
05/22/18-03:08:15 (tRAID): NOTE: Flashing tray summary fault LED
05/22/18-03:08:15 (tRAID): WARN: Controller entering ClientErr lockdown state...
DELL-Sam L
Moderator
Moderator
•
6.9K Posts
0
May 22nd, 2018 05:00
Hello Kashifsmb,
Can I get you to capture the entire boot process of the controller? Also, I will need to look at the support bundle from your MD as well. I will send you an email so that you can send me the logs.
Please let us know if you have any other questions.
Kashifsmb
5 Posts
0
June 26th, 2018 06:00
after i have send you email i dnt got any replay. awatting for your kind replay
DELL-Sam L
Moderator
Moderator
•
6.9K Posts
0
June 26th, 2018 06:00
Hello Kashifsmb,
I replied to your email asking for a support bundle from your MD3400 as well as the complete serial capture log from your controller as the partial log that you sent wasn’t enough to tell what was going on. I will send you another email so that I can get the support bundle & full Controller serial Capture.
Please let us know if you have any other questions.
Kashifsmb
5 Posts
0
June 27th, 2018 00:00
yah thank you i have received email can you please explain how to get support bundle by serial via putty.
DELL-Sam L
Moderator
Moderator
•
6.9K Posts
0
June 27th, 2018 08:00
Hello Kashifsmb,
To get the full controller boot up all you need to do is to start a terminal session & connect your serial cable to the controller. You don’t need to log in to shellusr as that is not required. You will want to remove the affected controller and wait about 1 minute then insert the controller back into its slot. When the controller is inserted you will see the controller start it’s bootup. If you see on the boot up “Start of Day Routine” message, then the controller is up & has logged in. If you don’t then I will need you to capture the log you get & send the file to me so that I can review it.
As for gather a support bundle you can do that either SMCLI or MDSM. Here is a link to the SMCLI guide and if you look on page 287 it explains how to gather the support bundle. https://topics-cdn.dell.com/pdf/powervault-md3400_cli-guide_en-us.pdf
Please let us know if you have any other questions.
Kashifsmb
5 Posts
0
July 2nd, 2018 06:00
Dear SAM,
please find below logs after wait1 minute and reiesert. i have just one controlor only.
Instantiating /ram as rawFs, device = 0x1
Formatting /ram for DOSFS
Instantiating /ram as rawFs, device = 0x1
Formatting...Retrieved old volume params with %38 confidence:
Volume Parameters: FAT type: FAT32, sectors per cluster 0
0 FAT copies, 0 clusters, 0 sectors per FAT
Sectors reserved 0, hidden 0, FAT sectors 0
Root dir entries 0, sysId (null) , serial number f10000
Label:" " ...
Disk with 1024 sectors of 512 bytes will be formatted with:
Volume Parameters: FAT type: FAT12, sectors per cluster 1
2 FAT copies, 1010 clusters, 3 sectors per FAT
Sectors reserved 1, hidden 0, FAT sectors 6
Root dir entries 112, sysId VXDOS12 , serial number f10000
Label:" " ...
RTC Error: Real-time clock device is not working
OK.
Adding 14607 symbols for standalone.
Reset, Power-Up Diagnostics - Loop 1 of 1
3600 Processor DRAM
01 Data lines Passed
02 Address lines Passed
3300 NVSRAM
01 Data lines Passed
4410 Ethernet 82574 1
01 Register read Passed
02 Register address lines Passed
6D40 Bobcat
02 Flash Test Passed
3700 PLB SRAM
01 Data lines Passed
02 Address lines Passed
7000 SE iSCSI BE2 1
01 Register Read Test Passed
02 Register Address Lines Test Passed
03 Register Data Lines Test Passed
3900 Real-Time Clock
01 RT Clock Tick Passed
Diagnostic Manager exited normally.
06/28/18-13:08:04 (tSystem): ERROR: FPGA FW is out of date
"Rhone03 rev17" currently in use
"Rhone03 rev20" available for update
06/28/18-13:08:09 (tNetCfgInit): NOTE: eth0: LinkUp event
06/28/18-13:08:09 (tNetCfgInit): NOTE: Network Ready
Current date: 06/28/18 time: 13:08:28
Send for Service Interface or baud rate change
06/28/18-13:08:28 (tRAID): SOD Sequence is Normal, 0 on controller A
06/28/18-13:08:29 (tRAID): NOTE: loading flash file: ECTDell
06/28/18-13:08:29 (tRAID): NOTE: unloading flash file: ECTDell
06/28/18-13:08:30 (tRAID): NOTE: I2C transaction returned 0x041aad00
06/28/18-13:08:30 (tRAID): NOTE: I2C transaction returned 0x041aad00
06/28/18-13:08:30 (tRAID): NOTE: I2C transaction returned 0x041aad00
06/28/18-13:08:30 (tRAID): NOTE: I2C transaction returned 0x041aad00
06/28/18-13:08:30 (tRAID): NOTE: I2C transaction returned 0x041aad00
06/28/18-13:08:30 (tRAID): NOTE: I2C transaction returned 0x041aad00
06/28/18-13:08:30 (tRAID): ERROR: Both midplane VPD ERROMS are bad
ERROR: Remap entry not found for ssid=0x0e.
06/28/18-13:08:30 (tRAID): NOTE: Turning on tray summary fault LED
06/28/18-13:08:30 (tRAID): NOTE: SODRebootLoop- Limit:5 Cnt:3
06/28/18-13:08:30 (tRAID): WARN: In getExpectedMdplVpdData, unknown SubsystemID of 14 was found - may result in controller lockdown
06/28/18-13:08:30 (tRAID): NOTE: I2C transaction returned 0x041aad00
06/28/18-13:08:30 (tRAID): ERROR: HCVH - Lockdown Controller: SBB VPD Checksum V alidation failed
06/28/18-13:08:30 (tRAID): WARN: Suspending Ctlr: "SBB_Acess_Chksum_Failed"
06/28/18-13:08:30 (tRAID): NOTE: Flashing tray summary fault LED
06/28/18-13:08:30 (tRAID): WARN: Controller entering ClientErr lockdown state.. .