Moderator

 • 

7.8K Posts

April 21st, 2017 14:00

Hello FuriousRabbit,

Here is the link to the guide for upgrading your MD3200i from simplex mode to Duplex mode. http://downloads.dell.com/manuals/common/powervault-md3200_user%27s%20guide3_en-us.pdf

Please let us know if you have any other questions.

May 13th, 2017 16:00

So I went through the steps outlined in the document, along with updating the firmware of the controllers, drives and updating to the latest version of MDSM. The controllers started working for about 3 minutes and then the array was unavailable. I have already backed up all of the data on the array and migrated, so at this point I'm looking to restore to factory settings.

The only way that I'm able to access the management port from MDSM is if all of the disks are removed. If I try to boot the array with 1 controller and 2 disks are installed in slot 0 and slot 1 I get the following from the console output:

Driver Build Date:Jan 18 2017, Time:16:06:27
ServerEngines iSCSI Target Driver loaded
iSCSI Firmware Version:2.901.326.0
iSCSI Driver Version:2.901.242.0
Found BladeEngine2 in Gen2 x8 PCIe slot
05/13/17-17:20:20 (tRAID): NOTE:  RPA memory region : start 0x471977e00 size 16777216 (ConIO-size 16777216) registered for pre-post data buffers on port 0
05/13/17-17:20:21 (tRAID): NOTE:  sebeReadEEPROMChecksum: EEPROM checksum match: 0x81986278
05/13/17-17:20:21 (tRAID): NOTE:  Target:irq_ptr:0x0xe3aaf0, func:0x1, bcmId:0x1
05/13/17-17:20:21 (tRAID): NOTE:  VendorID 0x19a2 DeviceId 0x210 bcmId 1 BDF 2.0.1
05/13/17-17:20:21 (tRAID): NOTE:  Target:VKI_INT_CONNECT success:status:0x0
05/13/17-17:20:21 (tRAID): ERROR: Target:ISR registered:irq_ptr:0xe3aaf0, phba:0x2f03fe8, func:0x1
05/13/17-17:20:21 (tRAID): NOTE:  BcmId [1] PCI [2.0.1]
05/13/17-17:20:21 (tRAID): NOTE:     DevId 0x0210 VendId 0x19a2
05/13/17-17:20:21 (tRAID): NOTE:     BAR0 IO base 0x00000000 size 0
05/13/17-17:20:21 (tRAID): NOTE:     BAR1 MEM base 0xf0260000 size 1
05/13/17-17:20:21 (tRAID): NOTE:     BAR2 MEM base 0xf0280000 size 1
05/13/17-17:20:21 (tRAID): NOTE:     BAR4 MEM base 0xf02a0000 size 1
05/13/17-17:20:21 (tRAID): NOTE:     MPU_EP_SEMAPHORE 0x0000c000
Driver Build Date:Jan 18 2017, Time:16:06:27
ServerEngines iSCSI Target Driver loaded
iSCSI Firmware Version:2.901.326.0
iSCSI Driver Version:2.901.242.0
Found BladeEngine2 in Gen2 x8 PCIe slot
05/13/17-17:20:22 (tRAID): NOTE:  RPA memory region : start 0x472977e00 size 16777216 (ConIO-size 16777216) registered for pre-post data buffers on port 1
05/13/17-17:20:22 (tRAID): NOTE:  sebeReadEEPROMChecksum: EEPROM checksum match: 0x81986278
05/13/17-17:20:23 (tRAID): NOTE:  Target:irq_ptr:0x0xe3a9a8, func:0x2, bcmId:0x2
05/13/17-17:20:23 (tRAID): NOTE:  VendorID 0x19a2 DeviceId 0x210 bcmId 2 BDF 2.0.2
05/13/17-17:20:23 (tRAID): NOTE:  Target:VKI_INT_CONNECT success:status:0x0
05/13/17-17:20:23 (tRAID): ERROR: Target:ISR registered:irq_ptr:0xe3a9a8, phba:0x2f04708, func:0x2
05/13/17-17:20:23 (tRAID): NOTE:  BcmId [2] PCI [2.0.2]
05/13/17-17:20:23 (tRAID): NOTE:     DevId 0x0210 VendId 0x19a2
05/13/17-17:20:23 (tRAID): NOTE:     BAR0 IO base 0x00000000 size 0
05/13/17-17:20:23 (tRAID): NOTE:     BAR1 MEM base 0xf02c0000 size 1
05/13/17-17:20:23 (tRAID): NOTE:     BAR2 MEM base 0xf02e0000 size 1
05/13/17-17:20:23 (tRAID): NOTE:     BAR4 MEM base 0xf0300000 size 1
05/13/17-17:20:23 (tRAID): NOTE:     MPU_EP_SEMAPHORE 0x0000c000
Driver Build Date:Jan 18 2017, Time:16:06:27
ServerEngines iSCSI Target Driver loaded
iSCSI Firmware Version:2.901.326.0
iSCSI Driver Version:2.901.242.0
Found BladeEngine2 in Gen2 x8 PCIe slot
05/13/17-17:20:24 (tRAID): NOTE:  RPA memory region : start 0x473977e00 size 16777216 (ConIO-size 16777216) registered for pre-post data buffers on port 2
05/13/17-17:20:24 (tRAID): NOTE:  sebeReadEEPROMChecksum: EEPROM checksum match: 0x81986278
05/13/17-17:20:25 (tRAID): NOTE:  Target:irq_ptr:0x0xe3a860, func:0x3, bcmId:0x3
05/13/17-17:20:25 (tRAID): NOTE:  VendorID 0x19a2 DeviceId 0x210 bcmId 3 BDF 2.0.3
05/13/17-17:20:25 (tRAID): NOTE:  Target:VKI_INT_CONNECT success:status:0x0
05/13/17-17:20:25 (tRAID): ERROR: Target:ISR registered:irq_ptr:0xe3a860, phba:0x2f04e28, func:0x3
05/13/17-17:20:25 (tRAID): NOTE:  BcmId [3] PCI [2.0.3]
05/13/17-17:20:25 (tRAID): NOTE:     DevId 0x0210 VendId 0x19a2
05/13/17-17:20:25 (tRAID): NOTE:     BAR0 IO base 0x00000000 size 0
05/13/17-17:20:25 (tRAID): NOTE:     BAR1 MEM base 0xf0320000 size 1
05/13/17-17:20:25 (tRAID): NOTE:     BAR2 MEM base 0xf0340000 size 1
05/13/17-17:20:25 (tRAID): NOTE:     BAR4 MEM base 0xf0360000 size 1
05/13/17-17:20:25 (tRAID): NOTE:     MPU_EP_SEMAPHORE 0x0000c000
Driver Build Date:Jan 18 2017, Time:16:06:27
ServerEngines iSCSI Target Driver loaded
iSCSI Firmware Version:2.901.326.0
iSCSI Driver Version:2.901.242.0
Found BladeEngine2 in Gen2 x8 PCIe slot
05/13/17-17:20:26 (tRAID): NOTE:  RPA memory region : start 0x474977e00 size 16777216 (ConIO-size 16777216) registered for pre-post data buffers on port 3
05/13/17-17:20:26 (tRAID): SOD: Pre-Initialization Phase Complete
05/13/17-17:20:34 (tRAID): WARN:  cmgr::readSystemRecord: DB system record not found.
05/13/17-17:20:34 (tRAID): WARN:  cmgr::isBoardDiff: No controller record
05/13/17-17:20:34 (tRAID): WARN:  cmgr::isHostBoardIdDiff: No controller record
05/13/17-17:20:34 (tRAID): WARN:  cmgr::isSubModelIdDiff: No controller record
05/13/17-17:20:34 (tRAID): WARN:  ACS: testCommunication: Domi Exception caught: IconSendInfeasibleException Error
05/13/17-17:20:34 (tRAID): NOTE:  ACS: autoCodeSync(): Process start. Comm Status: 0
05/13/17-17:20:34 (tRAID): WARN:  ACS: autoCodeSync(): Skipped since alt not communicating.
05/13/17-17:20:34 (tRAID): ERROR: BackupDatabaseManager:initialReadFromNative Exception: IOException
05/13/17-17:20:34 (tRAID): ERROR: ADM Read-Write Initialize failed on Drive 0xa0010000 with Status I/O Error
05/13/17-17:20:34 (tRAID): ERROR: BackupDatabaseManager::updateRecord: bdbm record is not initialized
05/13/17-17:20:34 (tRAID): ERROR: BackupDatabaseManager::recordTypeModifiedCommon caught DbmIOException: recType: 140, status: -9
05/13/17-17:20:34 (tRAID): ERROR: SubRecord::deleteRecord: Memory allocation error! Aborting txn: 0x0a61a270
05/13/17-17:20:34 (tRAID): WARN:  CmgrIoException on write, stat = -11 Line 101 File cmgrSod.cc
05/13/17-17:20:34 (tRAID): NOTE:  iocfi: Peering Disabled (Alt Unavailable)
05/13/17-17:20:34 (tRAID): SOD: Code Synchronization Initialization Phase Complete
05/13/17-17:20:35 (tRAID): NOTE:  SAFE: Process new features
05/13/17-17:20:35 (tRAID): NOTE:  SAFE: Process legacy features
05/13/17-17:20:35 (tRAID): ERROR: BackupDatabaseManager::updateRecord: bdbm record is not initialized
05/13/17-17:20:35 (tRAID): ERROR: BackupDatabaseManager::recordTypeModifiedCommon caught DbmIOException: recType: 140, status: -9
05/13/17-17:20:35 (tRAID): ERROR: SubRecord::writeRecord: listener failed! Aborting txn: 0x0a6ce248
05/13/17-17:20:35 (batteryMgr): ERROR: BackupDatabaseManager::updateRecord: bdbm record is not initialized
05/13/17-17:20:35 (batteryMgr): ERROR: BackupDatabaseManager::recordTypeModifiedCommon caught DbmIOException: recType: 140, status: -9
05/13/17-17:20:35 (batteryMgr): ERROR: SubRecord::writeRecord: listener failed! Aborting txn: 0x09c31fa0
05/13/17-17:20:36 (batteryMgr): ERROR: BackupDatabaseManager::updateRecord: bdbm record is not initialized
05/13/17-17:20:36 (batteryMgr): ERROR: BackupDatabaseManager::recordTypeModifiedCommon caught DbmIOException: recType: 140, status: -9
05/13/17-17:20:36 (batteryMgr): ERROR: SubRecord::writeRecord: listener failed! Aborting txn: 0x09edde18
05/13/17-17:20:36 (tRAID): NOTE:  IOManager::restoreData - dataSize:0x2800000, startAddress:0x4779f7e00
05/13/17-17:20:37 (tRAID): NOTE:  ncb::IOManager::restoreData - Successful
05/13/17-17:20:37 (tRAID): NOTE:  OBB Restore From Flash Completed
05/13/17-17:20:38 (bdbmBGTask): ERROR: sendFreezeStateToAlternate: Exception IconSendInfeasibleException Error
05/13/17-17:20:38 (tRAID): NOTE:  SPM acquireObjects exception: IconSendInfeasibleException Error
05/13/17-17:20:38 (tRAID): NOTE:  sas: Peering Disabled (Alt unavailable)
05/13/17-17:20:38 (tRAID): NOTE:  b_isn: Peering Disabled (Alt Unavailable)
05/13/17-17:20:38 (tRAID): NOTE:  ion: Peering Disabled (Alt Unavailable)

Exception: Program (Illegal Opcode)
   pc:  0x00000000
  esr:  0x08000000

Registers:
   r0     = ffffffff   sp     =  16fb6b0   r2     =        0   r3     =        0
   r4     =        1   r5     =  16fb5e0   r6     =  16fbd60   r7     =        0
   r8     =        0   r9     =        0   r10    =  16fbd60   r11    =  98173e0
   r12    = 28000284   r13    =        0   r14    =        0   r15    =        0
   r16    =        0   r17    =        0   r18    =        0   r19    =        0
   r20    =        0   r21    =        0   r22    =        0   r23    =        0
   r24    =      ef1   r25    =  16fbaf0   r26    =        0   r27    =        0
   r28    =  3ae0000   r29    =  51e8318   r30    =  353b6e0   r31    =  3ac0000
   msr    =    29200   lr     =  3544754   ctr    =        0   pc     =        0
   cr     = 28000284   xer    = 20000000   pgTblPtr =        0   asid   =        1


Stack Trace:
0x0027caac vxTaskEntry  +0x5c : 0x0344faa4 (0, 0)
0x0344fdb4 sodMain      +0x314: sodInitialization() ()
0x03451b8c sodInitialization()+0x2c : sodInitializeApplicationServices() ()
0x03451fcc sodInitializeApplicationServices()+0x10c: sodLogStartup(void (*)()) ()
0x034529fc sodLogStartup(void (*)())+0xbc : 0x0353b6e4 ()
0x0353b714 pm::initialize()+0x34 : pm::PmManager::initialize() ()
0x0353c018 pm::PmManager::initialize()+0x18 : pm::PmManager::readDb() ()
0x0353c638 pm::PmManager::readDb()+0x598: pm::CreateTransactor::CreateTransactor(txn::Transaction *, pm::Parameters  (0x51e27e4, 0xa81ed18)
0x03547388 pm::CreateTransactor::CreateTransactor(txn::Transaction *, pm::Parameters +0xe8 : pm::PmRecord::createMe(pm::Parameters &) ()

   NAME       TID     PRI   STATUS       PC         SP      ERRNO
---------- ---------- --- ---------- ---------- ---------- -------
tRAID      0x16fbd60  125 READY      0x1650e0   0x16fb6b0  550002

task stack: base 0x16fbd60   end 0x16f6d60   size 20480  high 5328   margin 15152
exc. stack: base 0x16fd000   end 0x16fc010   start 0x16fd010
exc. stack: size 4080        high 3272       margin 808

proc id: 0x6361b8 ((null))
r0         = 0xffffffff   sp         = 0x016fb6b0   r2         = 0x00000000
r3         = 0x00000000   r4         = 0x00000001   r5         = 0x016fb5e0
r6         = 0x016fbd60   r7         = 0x00000000   r8         = 0x00000000
r9         = 0x00000000   r10        = 0x016fbd60   r11        = 0x098173e0
r12        = 0x28000284   r13        = 0x00000000   r14        = 0x00000000
r15        = 0x00000000   r16        = 0x00000000   r17        = 0x00000000
r18        = 0x00000000   r19        = 0x00000000   r20        = 0x00000000
r21        = 0x00000000   r22        = 0x00000000   r23        = 0x00000000
r24        = 0x00000ef1   r25        = 0x016fbaf0   r26        = 0x00000000
r27        = 0x00000000   r28        = 0x03ae0000   r29        = 0x051e8318
r30        = 0x0353b6e0   r31        = 0x03ac0000   msr        = 0x00029200
lr         = 0x03544754   ctr        = 0x00000000   pc         = 0x00000000
cr         = 0x28000284   xer        = 0x20000000   pgTblPtr   = 0x00000000
asid       = 0x00000001

-=<###>=-
Instantiating /ram as rawFs,  device = 0x1
Formatting /ram for DOSFS
Instantiating /ram as rawFs, device = 0x1
Formatting...Retrieved old volume params with %38 confidence:
Volume Parameters: FAT type: FAT32, sectors per cluster 0
  0 FAT copies, 0 clusters, 0 sectors per FAT
  Sectors reserved 0, hidden 0, FAT sectors 0
  Root dir entries 0, sysId (null)  , serial number f10000
  Label:"           " ...
Disk with 1024 sectors of 512 bytes will be formatted with:
Volume Parameters: FAT type: FAT12, sectors per cluster 1
  2 FAT copies, 1010 clusters, 3 sectors per FAT
  Sectors reserved 1, hidden 0, FAT sectors 6
  Root dir entries 112, sysId VXDOS12 , serial number f10000
  Label:"           " ...

RTC Error:  Real-time clock device is not working
OK.

Adding 14607 symbols for standalone.
05/13/17-17:20:47 (tSystem): ERROR: FPGA FW is out of date
"Rhone03 rev17" currently in use
"Rhone03 rev20" available for update
05/13/17-17:20:52 (tNetCfgInit): NOTE:  eth0: LinkUp event
05/13/17-17:20:52 (tNetCfgInit): NOTE:  Acquiring network parameters for interface gei0 using DHCP
05/13/17-17:21:01 (ipdhcpc): NOTE:  netCfgDhcpReplyCallback :: received OFFER on interface gei0, unit 0
05/13/17-17:21:02 (ipdhcpc): NOTE:   DHCP server: 192.168.1.1
05/13/17-17:21:02 (ipdhcpc): WARN:   **WARNING** The DHCP Server did not assign a permanent IP for gei0.
05/13/17-17:21:02 (ipdhcpc): WARN:               Network access to this controller may eventually fail.
05/13/17-17:21:02 (ipdhcpc): NOTE:   Client IP routers:   192.168.1.1
05/13/17-17:21:02 (ipdhcpc): NOTE:   Client DNS name servers: 192.168.1.1
05/13/17-17:21:02 (ipdhcpc): NOTE:   DNS domain name: fios-router.home
05/13/17-17:21:02 (ipdhcpc): NOTE:   Assigned IP address: 192.168.1.191
05/13/17-17:21:02 (ipdhcpc): NOTE:   Assigned subnet mask: 255.255.255.0
05/13/17-17:21:02 (tNetReset): NOTE:  Network Ready
Current date: 05/13/17  time: 17:21:12

NetApp, Inc. RAID Controller
Copyright 2008-2017, NetApp, Inc. All Rights Reserved.
Copyright 1984-2006 Wind River Systems, Inc.
VxWorks: VxWorks 6.5   Kernel: WIND version 2.10
Model: 2660    Firmware version: 08.20.21.61
05/13/17-17:21:12 (tRAID): SOD Sequence is Normal, 0 on controller A

->
Serial Port shell started.
05/13/17-17:21:12 (tRAID): NOTE:  loading flash file: ECTDell
05/13/17-17:21:12 (tRAID): NOTE:  unloading flash file: ECTDell
05/13/17-17:21:13 (tRAID): NOTE:  Turning on tray summary fault LED
05/13/17-17:21:13 (tRAID): NOTE:  SODRebootLoop- Limit:5 Cnt:5
05/13/17-17:21:13 (tRAID): NOTE:  Flashing tray summary fault LED
05/13/17-17:21:13 (tRAID): WARN:  Controller entering SODReboot lockdown state...


At this point the controller goes into this loop five times and then locks. I have tried clearing the locked status with lemClearLockdown, but if there are drives installed the controller goes back into the endless loop (showing the same output as above) and locks itself again. Again, I'm not worried about the data on this array as it has all been migrated I just want to get this working properly. 

I've been doing some research and I'm wondering if the drives need to be wiped? Would they still have the original configuration and the modules can't load the configuration for some reason?

No Events found!

Top