Yesterday, we had a power outage, beyond the life of my UPS. When close to the battery limit, I had to shut down my MD3000i. An hour later, when power was restored, the array started fine and showed a blue light.
However, when starting my two Hyper-V servers, each connected to a iSCSI port, both were unable to connect.
The first thing I tried was a ping, and it turned out I could not ping the MD3000i. Looking closer at the ports, I noticed that they kept flashing in a irregular way. The "connection" light, ususally steady would turn off and back on every minute.
Next, I connect my laptops NIC directly to each of the ports, just to see the same problem. Now, I connect the serial cable and look at the diagnostics output. It seems the iSCSI ports are failing during self test.
Below is a diagnostics output, from a clean start with no network connections in the iSCSI ports. When started (I could see no errors this far) I connect the cable to my laptop and start seeing a sewuence of restart events
-=<###>=- Attaching interface lo0... done
Adding 9768 symbols for standalone. Error 01/11/15-09:52:52 (GMT) (tRootTask): NOTE: I2C transaction returned 0x0423fe00
Reset, Power-Up Diagnostics - Loop 1 of 1 3600 Processor DRAM 01 Data lines Passed 02 Address lines Passed 3300 NVSRAM 01 Data lines Passed 5900 Ethernet 91c111 #1 01 Register read Passed 02 Register test Passed 3A00 NAND Flash 06 Bad Blocks Test Passed 2310 Application Accelerator Unit 01 AAU Register Test Passed 6D00 LSI SAS 1068 IOC--Base Board 01 IOC Register Read Test Passed 02 IOC Register Address Lines Test Passed 03 IOC Register Data Lines Test Passed 6F01 QLOGIC EP4032 CHIP 0 01 Register Read Test Passed 02 Register Address Lines Test Passed 03 Register Data Lines Test Passed 3900 Real-Time Clock 01 RT Clock Tick Passed Diagnostic Manager exited normally.
Current date: 01/11/15 time: 02:26:34
Send for Service Interface or baud rate change 01/11/15-09:53:11 (GMT) (tRAID): NOTE: Set Powerup State 01/11/15-09:53:11 (GMT) (tRAID): NOTE: SOD Sequence is Normal, 0 01/11/15-09:53:11 (GMT) (tRAID): NOTE: SOD: removed SAS host from index 0 01/11/15-09:53:11 (GMT) (tRAID): NOTE: In iscsiIOQLIscsiInitDq. iscsiIoFstrBase = 0x0 01/11/15-09:53:11 (GMT) (tRAID): NOTE: Turning on tray summary fault LED 01/11/15-09:53:13 (GMT) (tRAID): NOTE: SYMBOL: SYMbolAPI registered. esmc0: LinkUp event 01/11/15-09:53:16 (GMT) (tRAID): NOTE: Initiating Drive channel: ioc:0 bringup 01/11/15-09:53:19 (GMT) (tRAID): NOTE: IOC Firmware Version: 00-24-63-00 01/11/15-09:53:19 (GMT) (tNetCfgInit): NOTE: Network Ready 01/11/15-09:53:26 (GMT) (tSasEvtWkr): NOTE: sasIocPhyUp: chan:0 phy:0 prevNumActivePhys:2 numActivePhys:2 01/11/15-09:53:27 (GMT) (tSasEvtWkr): NOTE: sasIocPhyUp: chan:0 phy:1 prevNumActivePhys:2 numActivePhys:2 01/11/15-09:53:36 (GMT) (tRAID): NOTE: IonMgr: Drive Interface Enabled 01/11/15-09:53:37 (GMT) (tRAID): NOTE: SOD: Instantiation Phase Complete 01/11/15-09:53:37 (GMT) (tRAID): WARN: No attempt made to open Inter-Controller Communication Channels 01/11/15-09:53:37 (GMT) (tRAID): NOTE: LockMgr Role is Master 01/11/15-09:53:37 (GMT) (tRAID): WARN: FBM:validateSubModel: Exception - Alt controller not ready 01/11/15-09:53:37 (GMT) (tSasDiscCom): NOTE: SAS Discovery complete task spawned 01/11/15-09:53:37 (GMT) (tRAID): NOTE: spmEarlyData: Using cached data 01/11/15-09:53:37 (GMT) (sasCheckExpanderSet): NOTE: Expander Firmware Version: 0116-e05c 01/11/15-09:53:37 (GMT) (sasCheckExpanderSet): NOTE: Expander SAS address: Hi = x5848f694 Low = xf73ad410 01/11/15-09:53:43 (GMT) (tSasDiscCom): WARN: SAS: Initial Discovery Complete Time: 30 seconds 01/11/15-09:53:43 (GMT) (tRAID): NOTE: WWN baseName 0002848f-69f73ad4 (valid==>SigMatch) 01/11/15-09:53:43 (GMT) (tRAID): NOTE: IonMgr: Host Interface Enabled 01/11/15-09:53:43 (GMT) (tRAID): NOTE: SOD: Pre-Initialization Phase Complete 01/11/15-09:53:54 (GMT) (tRAID): NOTE: ACS: Icon ping to alternate failed: -2, resp: 0 01/11/15-09:53:54 (GMT) (tRAID): NOTE: ACS: autoCodeSync(): Process start. Comm Mode: 0, Status: 0 01/11/15-09:53:54 (GMT) (tRAID): WARN: ACS: autoCodeSync(): Skipped since alt not communicating. 01/11/15-09:53:54 (GMT) (tRAID): NOTE: SOD: Code Synchronization Initialization Phase Complete 01/11/15-09:53:54 (GMT) (tRAID): NOTE: Caught IconSendInfeasibleException Error in iop::requestAltIopDelay 01/11/15-09:53:54 (GMT) (tRAID): NOTE: CheckInMonitor: Check-in failed (IconSendInfeasibleException Error) 01/11/15-09:53:55 (GMT) (NvpsPersistentSyncM): NOTE: NVSRAM Persistent Storage updated successfully 01/11/15-09:53:55 (GMT) (tRAID): NOTE: USM Mgr initialization complete with 0 records. 01/11/15-09:53:56 (GMT) (tRAID): WARN: Received IconSendInfeasibleException Error adding small edr records from alt controller 01/11/15-09:53:56 (GMT) (tRAID): WARN: spm: unable to exchange features, assuming none 01/11/15-09:53:56 (GMT) (tRAID): NOTE: SPM acquireObjects exception: IconSendInfeasibleException Error 01/11/15-09:53:57 (GMT) (tRAID): NOTE: DBRead 0.244 secs 01/11/15-09:53:57 (GMT) (tRAID): NOTE: sas: Peering Disabled (Alt Unavailable) 01/11/15-09:53:58 (GMT) (tRAID): NOTE: QLStartFw: Downloading Driver's FW image 03.00.01.47 from 0058c3a0 4c0c8 bytes , result 0 01/11/15-09:54:01 (GMT) (tRAID): NOTE: ******************************************************************************** 01/11/15-09:54:01 (GMT) (tRAID): NOTE: QLogic Target Application, Version 2.01.08 6-13-2005 (W2K) 01/11/15-09:54:01 (GMT) (tRAID): NOTE: iSCSI Target Application 01/11/15-09:54:01 (GMT) (tRAID): NOTE: ******************************************************************************** 01/11/15-09:54:01 (GMT) (tRAID): NOTE: QLInitializeFW: iSNS Server 0.0.0.0:3205 01/11/15-09:54:01 (GMT) (tRAID): NOTE: QLInitializeFW: ISNSServerIPv6Addr 00:00:00:00:00:00:00:00 :3205 01/11/15-09:54:01 (GMT) (tRAID): NOTE: QLInitializeFW: iSCSI Name iqn.1984-05.com.dell:powervault.md3000i.6842b2b000092450000000004c4f68b9 01/11/15-09:54:01 (GMT) (tRAID): NOTE: QLInitializeFW: port = 0, IPv4 Enable = 1, IPv6 Enable = 0 01/11/15-09:54:01 (GMT) (tRAID): NOTE: QLInitializeFW: IP Address 192.168.131.110:3260 01/11/15-09:54:01 (GMT) (tRAID): NOTE: QLInitializeFW: Firmware waiting for DHCP lease. State 18 01/11/15-09:54:01 (GMT) (tRAID): NOTE: QLInitializeFW: Time 000/010 FwState 18 01/11/15-09:54:02 (GMT) (tRAID): NOTE: QLInitializeFW: Time 001/010 FwState 18 01/11/15-09:54:03 (GMT) (tRAID): NOTE: QLInitializeFW: Time 002/010 FwState 18 01/11/15-09:54:04 (GMT) (tRAID): NOTE: QLInitializeFW: Time 003/010 FwState 18 01/11/15-09:54:05 (GMT) (tRAID): NOTE: QLInitializeFW: Time 004/010 FwState 18 01/11/15-09:54:06 (GMT) (tRAID): NOTE: QLInitializeFW: Time 005/010 FwState 18 01/11/15-09:54:07 (GMT) (tRAID): NOTE: QLInitializeFW: Time 006/010 FwState 18 01/11/15-09:54:08 (GMT) (tRAID): NOTE: QLInitializeFW: Time 007/010 FwState 18 01/11/15-09:54:09 (GMT) (tRAID): NOTE: QLInitializeFW: Time 008/010 FwState 18 01/11/15-09:54:10 (GMT) (tRAID): NOTE: QLInitializeFW: Time 009/010 FwState 18 01/11/15-09:54:11 (GMT) (tRAID): NOTE: QLInitializeFW: port = 1, IPv4 Enable = 1, IPv6 Enable = 0 01/11/15-09:54:11 (GMT) (tRAID): NOTE: QLInitializeFW: IP Address 192.168.131.101:3260 01/11/15-09:54:11 (GMT) (tRAID): NOTE: QLInitializeFW: Firmware waiting for DHCP lease. State 18 01/11/15-09:54:11 (GMT) (tRAID): NOTE: QLInitializeFW: Time 000/010 FwState 18 01/11/15-09:54:12 (GMT) (tRAID): NOTE: QLInitializeFW: Time 001/010 FwState 18 01/11/15-09:54:13 (GMT) (tRAID): NOTE: QLInitializeFW: Time 002/010 FwState 18 01/11/15-09:54:14 (GMT) (tRAID): NOTE: QLInitializeFW: Time 003/010 FwState 18 01/11/15-09:54:15 (GMT) (tRAID): NOTE: QLInitializeFW: Time 004/010 FwState 18 01/11/15-09:54:16 (GMT) (tRAID): NOTE: QLInitializeFW: Time 005/010 FwState 18 01/11/15-09:54:17 (GMT) (tRAID): NOTE: QLInitializeFW: Time 006/010 FwState 18 01/11/15-09:54:18 (GMT) (tRAID): NOTE: QLInitializeFW: Time 007/010 FwState 18 01/11/15-09:54:19 (GMT) (tRAID): NOTE: QLInitializeFW: Time 008/010 FwState 18 01/11/15-09:54:20 (GMT) (tRAID): NOTE: QLInitializeFW: Time 009/010 FwState 18 01/11/15-09:54:22 (GMT) (tRAID): NOTE: b_isn: Peering Disabled (Alt Unavailable) 01/11/15-09:54:22 (GMT) (tRAID): NOTE: ion: Peering Disabled (Alt Unavailable) 01/11/15-09:54:22 (GMT) (tRAID): NOTE: PM - reading DB (records 1..0) 01/11/15-09:54:22 (GMT) (tRAID): NOTE: CheckInMonitor: Check-in failed (IconSendInfeasibleException Error) 01/11/15-09:54:24 (GMT) (tRAID): NOTE: CCM: initialize(): Configuring cache 01/11/15-09:54:24 (GMT) (tRAID): WARN: Failed to send a powerup message packet - status: -1 01/11/15-09:54:24 (GMT) (tRAID): NOTE: Starting UWManager::initialize, entries 510, invalid index -1 01/11/15-09:54:24 (GMT) (tRAID): NOTE: Size of NVSRAM IW Queue is 0 01/11/15-09:54:26 (GMT) (tRAID): NOTE: RTR: IO Released 01/11/15-09:54:26 (GMT) (tRAID): NOTE: DiagVolManager::initialize: Exception - Alt controller not ready 01/11/15-09:54:26 (GMT) (tRAID): NOTE: Caught IconSendInfeasibleException Error in iop::requestAltIopDelay 01/11/15-09:54:26 (GMT) (tRAID): NOTE: SOD: Initialization Phase Complete ============================================== Title: Disk Array Controller Copyright 2005-2011 LSI Logic Corporation, All Rights Reserved.
Talked to a Dell representative today, and got a confirmation that the following line verifies my controller card is broken and needs to be replaced:
01/11/15-10:09:24 (GMT) (qlRetryTask): ERROR: QLGetFwState: MBOX_CMD_GET_FW_STATE failed. Stat f000
Unfortunately, these controllers are discontinued and can only be offered as a replacement with return of the existing controller. I just wish I had a second one...
Yep when you get that error on the controller in most cases it is in need of being replaced. I have seen sometimes that a customer can get it to come back online but it will soon fail again & then they won’t be able to get it to come back online again.
I would also check online as I think you can still purchase replacement controller from a Dell reseller or Ebay.
Please let us know if you have any other questions.
roarfred
2 Posts
772
0
Posted January 13th, 2015 06:00
Talked to a Dell representative today, and got a confirmation that the following line verifies my controller card is broken and needs to be replaced:
01/11/15-10:09:24 (GMT) (qlRetryTask): ERROR: QLGetFwState: MBOX_CMD_GET_FW_STATE failed. Stat f000
Unfortunately, these controllers are discontinued and can only be offered as a replacement with return of the existing controller. I just wish I had a second one...