Locked MD3000

Hello belykh_k,

So based on the output that you posted it looks like the alt controller has an issues and is stuck in a boot loop. The controller that you are connected to via serial is the master controller. Is the output the same from the alternate controller as well?

Please let us know if you have any other questions.

Log from another controller, capruted like previous, waited for 5 mins after last line:

-=<###>=-
Attaching interface lo0... done

Adding 9767 symbols for standalone.
Error
07/20/17-04:26:31 (GMT) (tRootTask): NOTE: I2C transaction returned 0x0423fe00
07/20/17-04:26:37 (GMT) (tRootTask): ERROR: ne1617Retry timeout
07/20/17-04:26:37 (GMT) (tRootTask): NOTE: I2C transaction returned 0x04223409
07/20/17-04:26:38 (GMT) (tRootTask): ERROR: ne1617Retry timeout
07/20/17-04:26:38 (GMT) (tRootTask): NOTE: I2C transaction returned 0x04229c09
07/20/17-04:26:40 (GMT) (tRootTask): ERROR: Could not reconstruct VPD on midplan e EEPROMs

Reset, Power-Up Diagnostics - Loop 1 of 1
3600 Processor DRAM
01 Data lines Passed
02 Address lines Passed
3300 NVSRAM
01 Data lines Passed
5900 Ethernet 91c111 #1
01 Register read Passed
02 Register test Passed
3A00 NAND Flash
06 Bad Blocks Test Passed
2310 Application Accelerator Unit
01 AAU Register Test Passed
6D00 LSI SAS 1068 IOC--Base Board
01 IOC Register Read Test Passed
02 IOC Register Address Lines Test Passed
03 IOC Register Data Lines Test Passed
3900 Real-Time Clock
01 RT Clock Tick Passed
Diagnostic Manager exited normally.

Current date: 07/20/17 time: 08:58:51

Send for Service Interface or baud rate change
07/20/17-04:26:58 (GMT) (tRAID): NOTE: Set Powerup State
07/20/17-04:26:58 (GMT) (tRAID): NOTE: SOD Sequence is Normal, 0
07/20/17-04:26:58 (GMT) (tRAID): NOTE: Turning on tray summary fault LED
07/20/17-04:26:59 (GMT) (tRAID): NOTE: I2C transaction returned 0x04227d01
07/20/17-04:26:59 (GMT) (tRAID): NOTE: I2C transaction returned 0x04227d01
07/20/17-04:26:59 (GMT) (tRAID): NOTE: I2C transaction returned 0x04227d00
07/20/17-04:26:59 (GMT) (tRAID): NOTE: I2C transaction returned 0x04227d00
07/20/17-04:26:59 (GMT) (tRAID): NOTE: I2C transaction returned 0x04227d00
07/20/17-04:26:59 (GMT) (tRAID): NOTE: I2C transaction returned 0x04227f01
07/20/17-04:26:59 (GMT) (tRAID): NOTE: I2C transaction returned 0x04227f01
07/20/17-04:26:59 (GMT) (tRAID): NOTE: I2C transaction reporting disabled.
07/20/17-04:26:59 (GMT) (tRAID): NOTE: SYMBOL: SYMbolAPI registered.
07/20/17-04:27:00 (GMT) (tRAID): NOTE: lost persistent dq data because buffer was modified or size changed.
esmc0: LinkUp event
07/20/17-04:27:01 (GMT) (tNetCfgInit): NOTE: Acquiring network parameters for interface esmc0 using DHCP
07/20/17-04:27:03 (GMT) (tRAID): NOTE: Initiating Drive channel: ioc:0 bringup
07/20/17-04:27:06 (GMT) (tRAID): NOTE: IOC Firmware Version: 00-24-63-00
07/20/17-04:27:11 (GMT) (tNetCfgInit): NOTE: DHCP obtained a lease for interface esmc0
07/20/17-04:27:11 (GMT) (tNetCfgInit): NOTE: DHCP server name:
07/20/17-04:27:11 (GMT) (tNetCfgInit): NOTE: DNS domain name: welcome-group.ru
07/20/17-04:27:11 (GMT) (tNetCfgInit): NOTE: DHCP server: 192.168.46.1
07/20/17-04:27:11 (GMT) (tNetCfgInit): NOTE: Assigned IP address: 192.168.46.173
07/20/17-04:27:11 (GMT) (tNetCfgInit): NOTE: Assigned subnet mask: 255.255.255.0
07/20/17-04:27:11 (GMT) (tNetCfgInit): NOTE: Client IP routers:
07/20/17-04:27:11 (GMT) (tNetCfgInit): NOTE: 192.168.46.1
07/20/17-04:27:11 (GMT) (tNetCfgInit): NOTE: Client DNS name servers:
07/20/17-04:27:11 (GMT) (tNetCfgInit): NOTE: 192.168.47.100
07/20/17-04:27:11 (GMT) (tNetCfgInit): NOTE: 192.168.47.112
07/20/17-04:27:11 (GMT) (tNetCfgInit): WARN: **WARNING** The DHCP Server did not assign a permanent IP for esmc0.
07/20/17-04:27:11 (GMT) (tNetCfgInit): WARN: Network access to this controller may eventually fail.
07/20/17-04:27:11 (GMT) (tNetCfgInit): NOTE: Network Ready
07/20/17-04:27:16 (GMT) (tSasEvtWkr): NOTE: sasIocPhyUp: chan:0 phy:0 prevNumActivePhys:2 numActivePhys:2
07/20/17-04:27:16 (GMT) (tSasEvtWkr): NOTE: sasIocPhyUp: chan:0 phy:1 prevNumActivePhys:2 numActivePhys:2
07/20/17-04:27:16 (GMT) (tSasEvtWkr): NOTE: sasIocPhyUp: chan:1 phy:2 prevNumActivePhys:2 numActivePhys:2
07/20/17-04:27:16 (GMT) (tSasEvtWkr): NOTE: sasIocPhyUp: chan:1 phy:3 prevNumActivePhys:2 numActivePhys:2
07/20/17-04:27:18 (GMT) (tSasEvtWkr): NOTE: sasLinkUp: expDevHandle:x9 - WidePort: #1 - Alt Controller, chan:0 encl:0, subExp:255, FAILED to DEGRADED
07/20/17-04:27:18 (GMT) (tSasEvtWkr): NOTE: sasLinkUp: expDevHandle:x9 - WidePort: #1 - Alt Controller, chan:0 encl:0, subExp:255, DEGRADED to OPTIMAL
07/20/17-04:27:18 (GMT) (tSasEvtWkr): NOTE: sasLinkUp: expDevHandle:xa - WidePort: #0 - Alt Controller, chan:1 encl:0, subExp:255, FAILED to DEGRADED
07/20/17-04:27:18 (GMT) (tSasEvtWkr): NOTE: sasLinkUp: expDevHandle:xa - WidePort: #0 - Alt Controller, chan:1 encl:0, subExp:255, DEGRADED to OPTIMAL
07/20/17-04:27:18 (GMT) (tSasCfg011): NOTE: Alt Controller path up - chan:0 phy:18 itn:9
07/20/17-04:27:18 (GMT) (tSasCfg012): NOTE: Alt Controller path up - chan:1 phy:16 itn:10
07/20/17-04:27:26 (GMT) (tRAID): NOTE: IonMgr: Drive Interface Enabled
07/20/17-04:27:26 (GMT) (tRAID): NOTE: SOD: Instantiation Phase Complete
07/20/17-04:27:26 (GMT) (tRAID): NOTE: Inter-Controller Communication Channels Opened
07/20/17-04:27:26 (GMT) (tRAID): NOTE: LockMgr Role is Master
07/20/17-04:27:26 (GMT) (tRAID): WARN: FBM:validateSubModel: Exception - Alt controller not ready
07/20/17-04:27:26 (GMT) (tSasDiscCom): NOTE: SAS Discovery complete task spawned
07/20/17-04:27:26 (GMT) (fbmLockdownTask): WARN: Invalid Sub-Model ID, locking down
07/20/17-04:27:26 (GMT) (tRAID): NOTE: spmEarlyData: No data available
07/20/17-04:27:26 (GMT) (sasCheckExpanderSet): NOTE: Expander Firmware Version: 0116-e05c
07/20/17-04:27:27 (GMT) (sasCheckExpanderSet): NOTE: Expander SAS address: Hi = x500a0b84 Low = x5a488a10

-=<###>=-
Attaching interface lo0... done

Adding 9767 symbols for standalone.
Error
07/20/17-04:27:33 (GMT) (tRootTask): NOTE: I2C transaction returned 0x0423fe00

Hello belykh_k,

So based on the output of your second controller you have both controller are thinking that the master. Seeing this error as well as you stating that this came up after a firmware upgrade on your MD3000. If you boot your MD3000 with the controller that you gathered the second serial capture in slot0 & remove the other controller your MD3000 should boot.

You will need to put your MD3000 into simplex mode using SMCLI. To set the controller in slot0 to simplex in SMCli with the command “set storageArray redundancyMode=simplex;”

Once the controlle is in simplex mode then you will want to upgrade the firmware to the latest version. Here is a link to the firmware just in case you need it. http://www.dell.com/support/home/us/en/19/drivers/driversdetails?driverId=R315163&fileId=2731121563&osCode=WS8R2&productCode=powervault-md3000&languageCode=en&categoryId=SF

Once the firmware has been updated then you will need to put your controller back into duplex mode. To set controller back to duplex with the following command “set storageArray redundancyMode=duplex;”

After system is set to duplex then hot add the second controller back to your MD3000. The second controller should begin to upgrade the firmware on the controller to match the first controller. If the controller still fails then you will need to replace the controller.

Here are the steps to do a firmware upgrade if you need them.

Steps to update the firmware:

1.        Ensure you have a tested backup of the data contained on the array

2.         It is highly recommended to gather support bundle from the "Support" tab of the MD Storage Manager before performing any firmware upgrade operations. Also recommend getting another support bundle after the upgrade is complete.

3.         You must stop all I/O going to the array

4.         Load MDSM to verify connectivity, then close MDSM but leave the main management screen open

5.         From the main management screen in MDSM use the “upgrade raid controller firmware” on the “devices” screen to upgrade to the latest firmware

6.         Verify connectivity through MDSM

7.         Update the hard drive firmware if applicable

8.         Reboot the stack

a.         Power down the server(s)

b.         Power down the MD3000(i)

c.         Power down any attached storage (MD1000/12xx)

d.         Leave the power off for 2-3 minutes

e.         Power up in reverse order

9.         Verify connection to the array and virtual disks and verify that data is presented to each of the connected servers

This update process will take 60-90 minutes but will vary depending on how long the reboots take to complete.

Please let us know if you have any other questions.

Hello,

i can't connect to powervault via smcli or MDSM, ports TCP/UDP 2463 and 2044 are closed. The only ports opened that i can see in nmap is TCP 23 (Telnet). Here is full output of nmap to second controller, then it is in slot 0:

Starting Nmap 7.40 ( https://nmap.org ) at 2017-07-21 11:04 RTZ 3 (ceia)
Nmap scan report for 192.168.46.173 
Host is up (0.0010s latency).

PORT STATE SERVICE
23/tcp open  telnet
2044/tcp closed rimsl
2044/udp closed rimsl
2463/tcp closed lsi-raid-mgmt
2463/udp closed lsi-raid-mgmt 

MAC Address: 00:A0:B8:**:**:** (NetApp)

Hello belykh_k,

Ok your controller is going to need to be replaced. The controller on boot you should be able to access those ports.

Please let us know if you have any other questions.