MD3200 Lockout - Keyboard input not working

We have an MD3200 with a new controller. I cannot clear the lockdown because it never seems to finish booting. it gets as far as 

08/29/17-21:08:40 (tRAID): NOTE: SYMbol available
08/29/17-21:08:40 (tRAID): WARN: *********************************************************
08/29/17-21:08:40 (tRAID): WARN: *********************************************************
08/29/17-21:08:40 (tRAID): WARN: ** **
08/29/17-21:08:40 (tRAID): WARN: ** WARNING!!!!!! **
08/29/17-21:08:40 (tRAID): WARN: ** **
08/29/17-21:08:40 (tRAID): WARN: ** This controller is not running the correct firmware **
08/29/17-21:08:40 (tRAID): WARN: ** release. The controller will enter lockdown mode. **
08/29/17-21:08:40 (tRAID): WARN: ** In order to prevent any configuration loss, the **
08/29/17-21:08:40 (tRAID): WARN: ** controller must be loaded with x0.00.xx.xx **
08/29/17-21:08:40 (tRAID): WARN: ** firmware release. **
08/29/17-21:08:40 (tRAID): WARN: ** **
08/29/17-21:08:40 (tRAID): WARN: ** **
08/29/17-21:08:40 (tRAID): WARN: *********************************************************
08/29/17-21:08:40 (tRAID): WARN: *********************************************************
08/29/17-21:08:41 (tRAID): SOD: sodComplete Notification Complete
08/29/17-21:08:41 (tRAID): sodMain CtlrLockdown sequence finished, elapsed time = 43 seconds
08/29/17-21:08:41 (tRAID): sodMain complete
08/29/17-21:08:40 (tRAID): WARN: SOD lockdown sequence complete - suspending...

Then, each time I type a command, it just jumps to the next line. I cannot enter any commands. 

When I press ctrl + break I get the message 

Press within 5 seconds: for Service Interface, for baud rate

but entering S does nothing. Can anyone suggest something I haven't tried yet?

Thanks!

Hello Omega876,

Is this a single controller MD3200 or is it a dual controller MD3200? What is the current version of firmware on your MD3200? It sounds like the firmware on the replacement is not the same as the firmware on your MD3200.

Please let us know if you have any other questions.

Sam,

This is a dual controller MD3200 but I only have one controller plugged in. In my previous tests with different controllers (I have 3, one came with the unit, two are replacements) I am able to control it with console commands and even add it to MDSM with no issues (other than it tells me I am missing the other controller).

So for this test, I just wanted to test the controller alone without the influence of the other working controller, if that makes sense. 

How do I check the firmware of the MD3200?

Thanks.

Just an update,

I read somewhere that the controllers, if not on the same firmware, will try and update one to the other. I attempted to connect both controllers and ended up corrupting my working controller. So now I have one controller that freezes during boot and one controller that is locked out and unable to accept shell commands. 

Hello Omega876,

Yes you are correct, when a replacement controller is inserted, it will try to update to the current version of firmware that is running on active controller. When replacing a controller it is best to offline the controller in MDSM before removal.

You should be able to test the non-working controller in your MD3200, and see if it will copy the configuration to it. You should be able to insert just the non-working controller by itself & it should copy the configuration & firmware from your HDD.

Please let us know if you have any other questions.

Hi Sam,

Since we bought the unit used, I'm not sure if the controllers were set to offline. What I know, is when the MD3200 arrived, I couldn't manage the controllers at all from MDSM. I was able to interact with the Shell via a console cable, and enter commands, but I couldn't add both controllers to MDSM no matter what I tried. The controller we suspected was bad, we sent back. The replacement controller, when booted by itself shows the following. As you can see, none of my commands or typing are registered at the bottom.

=<###>=-
Instantiating /ram as rawFs, device = 0x1
Formatting /ram for DOSFS
Instantiating /ram as rawFs, device = 0x1
Formatting...Retrieved old volume params with %38 confidence:
Volume Parameters: FAT type: FAT32, sectors per cluster 0
0 FAT copies, 0 clusters, 0 sectors per FAT
Sectors reserved 0, hidden 0, FAT sectors 0
Root dir entries 0, sysId (null) , serial number 10000
Label:" " ...
Disk with 1024 sectors of 512 bytes will be formatted with:
Volume Parameters: FAT type: FAT12, sectors per cluster 1
2 FAT copies, 1010 clusters, 3 sectors per FAT
Sectors reserved 1, hidden 0, FAT sectors 6
Root dir entries 112, sysId VXDOS12 , serial number 10000
Label:" " ...

RTC Error: Real-time clock device is not working
OK.

Adding 13818 symbols for standalone.

Reset, Power-Up Diagnostics - Loop 1 of 1
3600 Processor DRAM
01 Data lines Passed
02 Address lines Passed
3300 NVSRAM
01 Data lines Passed
4410 Ethernet 82574 1
01 Register read Passed
02 Register address lines Passed
6D40 Bobcat
02 Flash Test Passed
3700 PLB SRAM
01 Data lines Passed
02 Address lines Passed
6D50 LSISAS2008 IOC 1
01 Register Read Test Passed
02 Register Address Lines Test Passed
03 Register Data Lines Test Passed
3900 Real-Time Clock
01 RT Clock Tick Passed
Diagnostic Manager exited normally.
09/05/17-19:41:22 (tNetCfgInit): NOTE: eth0: LinkUp event
09/05/17-19:41:22 (tNetCfgInit): NOTE: Acquiring network parameters for interface gei0 using DHCP
09/05/17-19:41:30 (ipdhcpc): NOTE: netCfgDhcpReplyCallback :: received OFFER on interface gei0, unit 0
09/05/17-19:41:31 (ipdhcpc): NOTE: DHCP server: 10.1.14.3
09/05/17-19:41:31 (ipdhcpc): WARN: **WARNING** The DHCP Server did not assign a permanent IP for gei0.
09/05/17-19:41:31 (ipdhcpc): WARN: Network access to this controller may eventually fail.
09/05/17-19:41:31 (ipdhcpc): NOTE: Client IP routers: 10.1.14.3
09/05/17-19:41:31 (ipdhcpc): NOTE: Client DNS name servers: 10.1.14.3
09/05/17-19:41:31 (ipdhcpc): NOTE: DNS domain name: routerf3bc84.com
09/05/17-19:41:31 (ipdhcpc): NOTE: Assigned IP address: 10.1.14.108
09/05/17-19:41:31 (ipdhcpc): NOTE: Assigned subnet mask: 255.255.255.0
09/05/17-19:41:31 (tNetReset): NOTE: Network Ready

Current date: 09/05/17 time: 10:05:53

Send for Service Interface or baud rate change
09/05/17-19:41:34 (tRAID): NOTE: Set Powerup State
09/05/17-19:41:34 (tRAID): SOD Sequence is Normal, 0 on controller A
09/05/17-19:41:35 (tRAID): NOTE: Turning on tray summary fault LED
09/05/17-19:41:35 (tRAID): NOTE: Installed Protocols:
09/05/17-19:41:35 (tRAID): NOTE: Required Protocols:
09/05/17-19:41:36 (tRAID): NOTE: SYMBOL: SYMbolAPI registered.
09/05/17-19:41:36 (tRAID): NOTE: RCBBitmapManager total RPA size = 1828716544
09/05/17-19:41:36 (tRAID): NOTE: HealthCheck: AltFwStableThresh set from 60 to 120
09/05/17-19:41:36 (tRAID): NOTE: init: ioc: 0, PLVersion: 06-000-05-00
09/05/17-19:41:37 (tRAID): NOTE: Initiating Host channel: ioc:1 bringup
09/05/17-19:41:37 (tRAID): NOTE: IOC HW detected, index:0 type:host chan:0 chipID:72 rev:3
09/05/17-19:41:38 (IOSched): NOTE: SAS Expander Added: expDevHandle:x11 enclHandle:x2 numPhys:25 port:2 ioc:0 channel:0
09/05/17-19:41:38 (tSasExpChk): NOTE: Local Expander Firmware Version: 00.00.80.04
09/05/17-19:41:39 (tRAID): NOTE: IOC Firmware Version: 06-00-05-00
09/05/17-19:41:40 (tRAID): WARN: SendIocInit -- Using SHELL Bus for HIC
09/05/17-19:41:43 (tRAID): SOD: Instantiation Phase Complete
09/05/17-19:41:42 (tRAID): WARN: No attempt made to open Inter-Controller Communication Channels 0-0
09/05/17-19:41:42 (tRAID): NOTE: LockMgr Role is Master
09/05/17-19:41:46 (tSasDiscCom): NOTE: SAS: Initial Discovery Complete Time: 30 seconds since last power on/reset, 10 seconds since sas instantiated
09/05/17-19:41:46 (tRAID): NOTE: WWN baseName 0006782b-cb4199af (valid==>SigMatch)
09/05/17-19:41:46 (tRAID): NOTE: spmEarlyData: No data available
09/05/17-19:41:47 (tRAID): SOD: Pre-Initialization Phase Complete
09/05/17-19:42:16 (ssmSpin): WARN: Drive 0x10000 incompatible DSM LockedOutStatus=6 Reason=2
09/05/17-19:42:16 (ssmSpin): WARN: Drive 0x10001 incompatible DSM LockedOutStatus=6 Reason=2
09/05/17-19:42:16 (ssmSpin): WARN: Drive 0x10002 incompatible DSM LockedOutStatus=6 Reason=2
09/05/17-19:42:16 (ssmSpin): WARN: Drive 0x10003 incompatible DSM LockedOutStatus=6 Reason=2
09/05/17-19:42:16 (ssmSpin): WARN: Drive 0x10004 incompatible DSM LockedOutStatus=6 Reason=2
09/05/17-19:42:16 (ssmSpin): WARN: Drive 0x10005 incompatible DSM LockedOutStatus=6 Reason=2
09/05/17-19:42:16 (ssmSpin): WARN: Drive 0x10006 incompatible DSM LockedOutStatus=6 Reason=2
09/05/17-19:42:16 (ssmSpin): WARN: Drive 0x10007 incompatible DSM LockedOutStatus=6 Reason=2
09/05/17-19:42:16 (ssmSpin): WARN: Drive 0x10008 incompatible DSM LockedOutStatus=6 Reason=2
09/05/17-19:42:16 (tRAID): WARN: dbm::RWFileSystem::initialize: Exception caught, ConstructorIOException: -16
09/05/17-19:42:16 (tRAID): ERROR: In PersistenceManager::initialize: catch DbmNoFileSystemException: recType: 84
09/05/17-19:42:16 (tRAID): ERROR: ADM Load Reservations failed with error (5) Exception
09/05/17-19:42:16 (tRAID): WARN: CmgrLockdownException: UP REVISION SSTOR - LOCKDOWN Type=3
09/05/17-19:42:16 (tRAID): WARN: sodLockdownCheck: sodSequence update: CtlrLockdown
09/05/17-19:42:16 (tRAID): NOTE: ACS: Icon ping to alternate failed: -2, resp: 0
09/05/17-19:42:16 (tRAID): NOTE: ACS: autoCodeSync(): Process start. Comm Mode: 0, Status: 0
09/05/17-19:42:16 (tRAID): WARN: ACS: autoCodeSync(): Skipped since alt not communicating.
09/05/17-19:42:17 (tRAID): SOD: Code Synchronization Initialization Phase Complete
09/05/17-19:42:16 (tRAID): WARN: Lockdown case-REV UP drives. NVPS did not update from Sstor.
09/05/17-19:42:17 (tRAID): SOD: Initialization Phase Complete
==============================================
Title: Disk Array Controller
Copyright 2008-2011 LSI Logic Corporation, All Rights Reserved.

Name: RC
Version: 07.80.41.60
Date: 08/30/2011
Time: 18:16:51 CDT
Models: 2660
Manager: devmgr.v1080api06.Manager
==============================================

09/05/17-19:42:16 (tRAID): NOTE: SYMbol available
09/05/17-19:42:16 (tRAID): WARN: *********************************************************
09/05/17-19:42:16 (tRAID): WARN: *********************************************************
09/05/17-19:42:16 (tRAID): WARN: ** **
09/05/17-19:42:16 (tRAID): WARN: ** WARNING!!!!!! **
09/05/17-19:42:16 (tRAID): WARN: ** **
09/05/17-19:42:16 (tRAID): WARN: ** This controller is not running the correct firmware **
09/05/17-19:42:16 (tRAID): WARN: ** release. The controller will enter lockdown mode. **
09/05/17-19:42:16 (tRAID): WARN: ** In order to prevent any configuration loss, the **
09/05/17-19:42:16 (tRAID): WARN: ** controller must be loaded with x0.00.xx.xx **
09/05/17-19:42:16 (tRAID): WARN: ** firmware release. **
09/05/17-19:42:16 (tRAID): WARN: ** **
09/05/17-19:42:16 (tRAID): WARN: ** **
09/05/17-19:42:16 (tRAID): WARN: *********************************************************
09/05/17-19:42:16 (tRAID): WARN: *********************************************************
09/05/17-19:42:17 (tRAID): SOD: sodComplete Notification Complete
09/05/17-19:42:17 (tRAID): sodMain CtlrLockdown sequence finished, elapsed time = 43 seconds
09/05/17-19:42:17 (tRAID): sodMain complete
09/05/17-19:42:17 (tRAID): WARN: SOD lockdown sequence complete - suspending...

lem
clear
lemClearLockdown

Hello Omega876,

So based on what you posted I can see that this the primary controller & it is not booting due to not being able to read the firmware. I am guessing that you are getting the same message on the other controller as well?

Have you tried to use just one controller & see if it boots fully? If you have and you are getting the same errors then what is needed is to clear the lockdown state.

In order to clear a lock down state you have to be in secure shell of the controller. Once in secure shell you can run the lemclearlockdown. You will need to know the password for secure shell to access the full menu. Dell doesn’t provide the secure shell password to customers.

Please let us know if you have any other questions.

Sam,

I am only having this issue on this controller. i was able to login to the secure Shell on controller 0 and clear the lockdown. I was also able to clear the config. Now controller 0 is working fine and I was able to add controller 0 MDSM.

The problem is still with this controller 1. This controller I was also able to login to the secure shell with the username and password. I ran the command lemClearLockdown and the unit begins to clear the lockdown. The issue is that no matter how many times I run this command it stays in lockdown because of the firmware message as you mentioned earlier.

Name: RC
Version: 07.80.41.60
Date: 08/30/2011
Time: 18:16:51 CDT
Models: 2660
Manager: devmgr.v1080api06.Manager
==============================================

09/06/17-19:34:01 (tRAID): NOTE: SYMbol available
09/06/17-19:34:01 (tRAID): WARN: *********************************************************
09/06/17-19:34:01 (tRAID): WARN: *********************************************************
09/06/17-19:34:01 (tRAID): WARN: ** **
09/06/17-19:34:01 (tRAID): WARN: ** WARNING!!!!!! **
09/06/17-19:34:01 (tRAID): WARN: ** **
09/06/17-19:34:01 (tRAID): WARN: ** This controller is not running the correct firmware **
09/06/17-19:34:01 (tRAID): WARN: ** release. The controller will enter lockdown mode. **
09/06/17-19:34:01 (tRAID): WARN: ** In order to prevent any configuration loss, the **
09/06/17-19:34:01 (tRAID): WARN: ** controller must be loaded with x0.00.xx.xx **
09/06/17-19:34:01 (tRAID): WARN: ** firmware release. **
09/06/17-19:34:01 (tRAID): WARN: ** **
09/06/17-19:34:01 (tRAID): WARN: ** **
09/06/17-19:34:01 (tRAID): WARN: *********************************************************
09/06/17-19:34:01 (tRAID): WARN: *********************************************************
09/06/17-19:34:02 (tRAID): SOD: sodComplete Notification Complete
09/06/17-19:34:02 (tRAID): sodMain CtlrLockdown sequence finished, elapsed time = 19 seconds
09/06/17-19:34:02 (tRAID): sodMain complete
09/06/17-19:34:01 (tRAID): WARN: SOD lockdown sequence complete - suspending...
Name: RC
Version: 07.80.41.60
Date: 08/30/2011
Time: 18:16:51 CDT
Models: 2660
Manager: devmgr.v1080api06.Manager
==============================================

09/06/17-19:34:01 (tRAID): NOTE: SYMbol available
09/06/17-19:34:01 (tRAID): WARN: *********************************************************
09/06/17-19:34:01 (tRAID): WARN: *********************************************************

My question: Is there a way to update the firmware manually without the controller being added to MDSM? If not, what are my options for fixing this?

Thanks!

Hello Omega876,

It looks like your controller could be set to simplex mode. What you will need to do is to run the following command from controller in slot 0 to put your MD3200 in duplex mode. From SMCLI run the following

SMcli.exe -n ArrayName -c “set storageArray redundancyMode=duplex;

Once the command finishes running then you will want to hot add the controller in slot 1 & it should bring it online.

If you can’t run the command from SMCLI then you can run the following from the serial port on controller 0.

setSimplexMode_MT 1

Please let us know if you have any other questions.