UNSOLVED

FuriousRabbit

updated

10 years ago

0

1321

April 21st, 2017 10:00

Adding a RAID Module to a MD3200i

I have a MD3200i running firmware version 07.84.53.60 on the RAID module. This MD3200i has always only had one controller. Everything has been running fine but I want to add a second module to this system.

Are there any gotchas that I should be aware of before attempting this? Do I need to power down the MD3200i and all ESXi hosts attached to it before installing the module? What if the new module has a newer firmware version than the firmware installed on the current module?

  • DELL-Sam L

    Community Manager

    •

    8147 Posts

    •

    34706 Points

    818

    0

    Posted April 21st, 2017 14:00

    Hello FuriousRabbit,

    Here is the link to the guide for upgrading your MD3200i from simplex mode to Duplex mode. http://downloads.dell.com/manuals/common/powervault-md3200_user%27s%20guide3_en-us.pdf

    Please let us know if you have any other questions.

  • 818

    0

    Posted May 13th, 2017 16:00

    So I went through the steps outlined in the document, along with updating the firmware of the controllers, drives and updating to the latest version of MDSM. The controllers started working for about 3 minutes and then the array was unavailable. I have already backed up all of the data on the array and migrated, so at this point I'm looking to restore to factory settings.

    The only way that I'm able to access the management port from MDSM is if all of the disks are removed. If I try to boot the array with 1 controller and 2 disks are installed in slot 0 and slot 1 I get the following from the console output:

    Driver Build Date:Jan 18 2017, Time:16:06:27
    ServerEngines iSCSI Target Driver loaded
    iSCSI Firmware Version:2.901.326.0
    iSCSI Driver Version:2.901.242.0
    Found BladeEngine2 in Gen2 x8 PCIe slot
    05/13/17-17:20:20 (tRAID): NOTE:  RPA memory region : start 0x471977e00 size 16777216 (ConIO-size 16777216) registered for pre-post data buffers on port 0
    05/13/17-17:20:21 (tRAID): NOTE:  sebeReadEEPROMChecksum: EEPROM checksum match: 0x81986278
    05/13/17-17:20:21 (tRAID): NOTE:  Target:irq_ptr:0x0xe3aaf0, func:0x1, bcmId:0x1
    05/13/17-17:20:21 (tRAID): NOTE:  VendorID 0x19a2 DeviceId 0x210 bcmId 1 BDF 2.0.1
    05/13/17-17:20:21 (tRAID): NOTE:  Target:VKI_INT_CONNECT success:status:0x0
    05/13/17-17:20:21 (tRAID): ERROR: Target:ISR registered:irq_ptr:0xe3aaf0, phba:0x2f03fe8, func:0x1
    05/13/17-17:20:21 (tRAID): NOTE:  BcmId [1] PCI [2.0.1]
    05/13/17-17:20:21 (tRAID): NOTE:     DevId 0x0210 VendId 0x19a2
    05/13/17-17:20:21 (tRAID): NOTE:     BAR0 IO base 0x00000000 size 0
    05/13/17-17:20:21 (tRAID): NOTE:     BAR1 MEM base 0xf0260000 size 1
    05/13/17-17:20:21 (tRAID): NOTE:     BAR2 MEM base 0xf0280000 size 1
    05/13/17-17:20:21 (tRAID): NOTE:     BAR4 MEM base 0xf02a0000 size 1
    05/13/17-17:20:21 (tRAID): NOTE:     MPU_EP_SEMAPHORE 0x0000c000
    Driver Build Date:Jan 18 2017, Time:16:06:27
    ServerEngines iSCSI Target Driver loaded
    iSCSI Firmware Version:2.901.326.0
    iSCSI Driver Version:2.901.242.0
    Found BladeEngine2 in Gen2 x8 PCIe slot
    05/13/17-17:20:22 (tRAID): NOTE:  RPA memory region : start 0x472977e00 size 16777216 (ConIO-size 16777216) registered for pre-post data buffers on port 1
    05/13/17-17:20:22 (tRAID): NOTE:  sebeReadEEPROMChecksum: EEPROM checksum match: 0x81986278
    05/13/17-17:20:23 (tRAID): NOTE:  Target:irq_ptr:0x0xe3a9a8, func:0x2, bcmId:0x2
    05/13/17-17:20:23 (tRAID): NOTE:  VendorID 0x19a2 DeviceId 0x210 bcmId 2 BDF 2.0.2
    05/13/17-17:20:23 (tRAID): NOTE:  Target:VKI_INT_CONNECT success:status:0x0
    05/13/17-17:20:23 (tRAID): ERROR: Target:ISR registered:irq_ptr:0xe3a9a8, phba:0x2f04708, func:0x2
    05/13/17-17:20:23 (tRAID): NOTE:  BcmId [2] PCI [2.0.2]
    05/13/17-17:20:23 (tRAID): NOTE:     DevId 0x0210 VendId 0x19a2
    05/13/17-17:20:23 (tRAID): NOTE:     BAR0 IO base 0x00000000 size 0
    05/13/17-17:20:23 (tRAID): NOTE:     BAR1 MEM base 0xf02c0000 size 1
    05/13/17-17:20:23 (tRAID): NOTE:     BAR2 MEM base 0xf02e0000 size 1
    05/13/17-17:20:23 (tRAID): NOTE:     BAR4 MEM base 0xf0300000 size 1
    05/13/17-17:20:23 (tRAID): NOTE:     MPU_EP_SEMAPHORE 0x0000c000
    Driver Build Date:Jan 18 2017, Time:16:06:27
    ServerEngines iSCSI Target Driver loaded
    iSCSI Firmware Version:2.901.326.0
    iSCSI Driver Version:2.901.242.0
    Found BladeEngine2 in Gen2 x8 PCIe slot
    05/13/17-17:20:24 (tRAID): NOTE:  RPA memory region : start 0x473977e00 size 16777216 (ConIO-size 16777216) registered for pre-post data buffers on port 2
    05/13/17-17:20:24 (tRAID): NOTE:  sebeReadEEPROMChecksum: EEPROM checksum match: 0x81986278
    05/13/17-17:20:25 (tRAID): NOTE:  Target:irq_ptr:0x0xe3a860, func:0x3, bcmId:0x3
    05/13/17-17:20:25 (tRAID): NOTE:  VendorID 0x19a2 DeviceId 0x210 bcmId 3 BDF 2.0.3
    05/13/17-17:20:25 (tRAID): NOTE:  Target:VKI_INT_CONNECT success:status:0x0
    05/13/17-17:20:25 (tRAID): ERROR: Target:ISR registered:irq_ptr:0xe3a860, phba:0x2f04e28, func:0x3
    05/13/17-17:20:25 (tRAID): NOTE:  BcmId [3] PCI [2.0.3]
    05/13/17-17:20:25 (tRAID): NOTE:     DevId 0x0210 VendId 0x19a2
    05/13/17-17:20:25 (tRAID): NOTE:     BAR0 IO base 0x00000000 size 0
    05/13/17-17:20:25 (tRAID): NOTE:     BAR1 MEM base 0xf0320000 size 1
    05/13/17-17:20:25 (tRAID): NOTE:     BAR2 MEM base 0xf0340000 size 1
    05/13/17-17:20:25 (tRAID): NOTE:     BAR4 MEM base 0xf0360000 size 1
    05/13/17-17:20:25 (tRAID): NOTE:     MPU_EP_SEMAPHORE 0x0000c000
    Driver Build Date:Jan 18 2017, Time:16:06:27
    ServerEngines iSCSI Target Driver loaded
    iSCSI Firmware Version:2.901.326.0
    iSCSI Driver Version:2.901.242.0
    Found BladeEngine2 in Gen2 x8 PCIe slot
    05/13/17-17:20:26 (tRAID): NOTE:  RPA memory region : start 0x474977e00 size 16777216 (ConIO-size 16777216) registered for pre-post data buffers on port 3
    05/13/17-17:20:26 (tRAID): SOD: Pre-Initialization Phase Complete
    05/13/17-17:20:34 (tRAID): WARN:  cmgr::readSystemRecord: DB system record not found.
    05/13/17-17:20:34 (tRAID): WARN:  cmgr::isBoardDiff: No controller record
    05/13/17-17:20:34 (tRAID): WARN:  cmgr::isHostBoardIdDiff: No controller record
    05/13/17-17:20:34 (tRAID): WARN:  cmgr::isSubModelIdDiff: No controller record
    05/13/17-17:20:34 (tRAID): WARN:  ACS: testCommunication: Domi Exception caught: IconSendInfeasibleException Error
    05/13/17-17:20:34 (tRAID): NOTE:  ACS: autoCodeSync(): Process start. Comm Status: 0
    05/13/17-17:20:34 (tRAID): WARN:  ACS: autoCodeSync(): Skipped since alt not communicating.
    05/13/17-17:20:34 (tRAID): ERROR: BackupDatabaseManager:initialReadFromNative Exception: IOException
    05/13/17-17:20:34 (tRAID): ERROR: ADM Read-Write Initialize failed on Drive 0xa0010000 with Status I/O Error
    05/13/17-17:20:34 (tRAID): ERROR: BackupDatabaseManager::updateRecord: bdbm record is not initialized
    05/13/17-17:20:34 (tRAID): ERROR: BackupDatabaseManager::recordTypeModifiedCommon caught DbmIOException: recType: 140, status: -9
    05/13/17-17:20:34 (tRAID): ERROR: SubRecord::deleteRecord: Memory allocation error! Aborting txn: 0x0a61a270
    05/13/17-17:20:34 (tRAID): WARN:  CmgrIoException on write, stat = -11 Line 101 File cmgrSod.cc
    05/13/17-17:20:34 (tRAID): NOTE:  iocfi: Peering Disabled (Alt Unavailable)
    05/13/17-17:20:34 (tRAID): SOD: Code Synchronization Initialization Phase Complete
    05/13/17-17:20:35 (tRAID): NOTE:  SAFE: Process new features
    05/13/17-17:20:35 (tRAID): NOTE:  SAFE: Process legacy features
    05/13/17-17:20:35 (tRAID): ERROR: BackupDatabaseManager::updateRecord: bdbm record is not initialized
    05/13/17-17:20:35 (tRAID): ERROR: BackupDatabaseManager::recordTypeModifiedCommon caught DbmIOException: recType: 140, status: -9
    05/13/17-17:20:35 (tRAID): ERROR: SubRecord::writeRecord: listener failed! Aborting txn: 0x0a6ce248
    05/13/17-17:20:35 (batteryMgr): ERROR: BackupDatabaseManager::updateRecord: bdbm record is not initialized
    05/13/17-17:20:35 (batteryMgr): ERROR: BackupDatabaseManager::recordTypeModifiedCommon caught DbmIOException: recType: 140, status: -9
    05/13/17-17:20:35 (batteryMgr): ERROR: SubRecord::writeRecord: listener failed! Aborting txn: 0x09c31fa0
    05/13/17-17:20:36 (batteryMgr): ERROR: BackupDatabaseManager::updateRecord: bdbm record is not initialized
    05/13/17-17:20:36 (batteryMgr): ERROR: BackupDatabaseManager::recordTypeModifiedCommon caught DbmIOException: recType: 140, status: -9
    05/13/17-17:20:36 (batteryMgr): ERROR: SubRecord::writeRecord: listener failed! Aborting txn: 0x09edde18
    05/13/17-17:20:36 (tRAID): NOTE:  IOManager::restoreData - dataSize:0x2800000, startAddress:0x4779f7e00
    05/13/17-17:20:37 (tRAID): NOTE:  ncb::IOManager::restoreData - Successful
    05/13/17-17:20:37 (tRAID): NOTE:  OBB Restore From Flash Completed
    05/13/17-17:20:38 (bdbmBGTask): ERROR: sendFreezeStateToAlternate: Exception IconSendInfeasibleException Error
    05/13/17-17:20:38 (tRAID): NOTE:  SPM acquireObjects exception: IconSendInfeasibleException Error
    05/13/17-17:20:38 (tRAID): NOTE:  sas: Peering Disabled (Alt unavailable)
    05/13/17-17:20:38 (tRAID): NOTE:  b_isn: Peering Disabled (Alt Unavailable)
    05/13/17-17:20:38 (tRAID): NOTE:  ion: Peering Disabled (Alt Unavailable)

    Exception: Program (Illegal Opcode)
       pc:  0x00000000
      esr:  0x08000000

    Registers:
       r0     = ffffffff   sp     =  16fb6b0   r2     =        0   r3     =        0
       r4     =        1   r5     =  16fb5e0   r6     =  16fbd60   r7     =        0
       r8     =        0   r9     =        0   r10    =  16fbd60   r11    =  98173e0
       r12    = 28000284   r13    =        0   r14    =        0   r15    =        0
       r16    =        0   r17    =        0   r18    =        0   r19    =        0
       r20    =        0   r21    =        0   r22    =        0   r23    =        0
       r24    =      ef1   r25    =  16fbaf0   r26    =        0   r27    =        0
       r28    =  3ae0000   r29    =  51e8318   r30    =  353b6e0   r31    =  3ac0000
       msr    =    29200   lr     =  3544754   ctr    =        0   pc     =        0
       cr     = 28000284   xer    = 20000000   pgTblPtr =        0   asid   =        1


    Stack Trace:
    0x0027caac vxTaskEntry  +0x5c : 0x0344faa4 (0, 0)
    0x0344fdb4 sodMain      +0x314: sodInitialization() ()
    0x03451b8c sodInitialization()+0x2c : sodInitializeApplicationServices() ()
    0x03451fcc sodInitializeApplicationServices()+0x10c: sodLogStartup(void (*)()) ()
    0x034529fc sodLogStartup(void (*)())+0xbc : 0x0353b6e4 ()
    0x0353b714 pm::initialize()+0x34 : pm::PmManager::initialize() ()
    0x0353c018 pm::PmManager::initialize()+0x18 : pm::PmManager::readDb() ()
    0x0353c638 pm::PmManager::readDb()+0x598: pm::CreateTransactor::CreateTransactor(txn::Transaction *, pm::Parameters  (0x51e27e4, 0xa81ed18)
    0x03547388 pm::CreateTransactor::CreateTransactor(txn::Transaction *, pm::Parameters +0xe8 : pm::PmRecord::createMe(pm::Parameters &) ()

       NAME       TID     PRI   STATUS       PC         SP      ERRNO
    ---------- ---------- --- ---------- ---------- ---------- -------
    tRAID      0x16fbd60  125 READY      0x1650e0   0x16fb6b0  550002

    task stack: base 0x16fbd60   end 0x16f6d60   size 20480  high 5328   margin 15152
    exc. stack: base 0x16fd000   end 0x16fc010   start 0x16fd010
    exc. stack: size 4080        high 3272       margin 808

    proc id: 0x6361b8 ((null))
    r0         = 0xffffffff   sp         = 0x016fb6b0   r2         = 0x00000000
    r3         = 0x00000000   r4         = 0x00000001   r5         = 0x016fb5e0
    r6         = 0x016fbd60   r7         = 0x00000000   r8         = 0x00000000
    r9         = 0x00000000   r10        = 0x016fbd60   r11        = 0x098173e0
    r12        = 0x28000284   r13        = 0x00000000   r14        = 0x00000000
    r15        = 0x00000000   r16        = 0x00000000   r17        = 0x00000000
    r18        = 0x00000000   r19        = 0x00000000   r20        = 0x00000000
    r21        = 0x00000000   r22        = 0x00000000   r23        = 0x00000000
    r24        = 0x00000ef1   r25        = 0x016fbaf0   r26        = 0x00000000
    r27        = 0x00000000   r28        = 0x03ae0000   r29        = 0x051e8318
    r30        = 0x0353b6e0   r31        = 0x03ac0000   msr        = 0x00029200
    lr         = 0x03544754   ctr        = 0x00000000   pc         = 0x00000000
    cr         = 0x28000284   xer        = 0x20000000   pgTblPtr   = 0x00000000
    asid       = 0x00000001

    -=<###>=-
    Instantiating /ram as rawFs,  device = 0x1
    Formatting /ram for DOSFS
    Instantiating /ram as rawFs, device = 0x1
    Formatting...Retrieved old volume params with %38 confidence:
    Volume Parameters: FAT type: FAT32, sectors per cluster 0
      0 FAT copies, 0 clusters, 0 sectors per FAT
      Sectors reserved 0, hidden 0, FAT sectors 0
      Root dir entries 0, sysId (null)  , serial number f10000
      Label:"           " ...
    Disk with 1024 sectors of 512 bytes will be formatted with:
    Volume Parameters: FAT type: FAT12, sectors per cluster 1
      2 FAT copies, 1010 clusters, 3 sectors per FAT
      Sectors reserved 1, hidden 0, FAT sectors 6
      Root dir entries 112, sysId VXDOS12 , serial number f10000
      Label:"           " ...

    RTC Error:  Real-time clock device is not working
    OK.

    Adding 14607 symbols for standalone.
    05/13/17-17:20:47 (tSystem): ERROR: FPGA FW is out of date
    "Rhone03 rev17" currently in use
    "Rhone03 rev20" available for update
    05/13/17-17:20:52 (tNetCfgInit): NOTE:  eth0: LinkUp event
    05/13/17-17:20:52 (tNetCfgInit): NOTE:  Acquiring network parameters for interface gei0 using DHCP
    05/13/17-17:21:01 (ipdhcpc): NOTE:  netCfgDhcpReplyCallback :: received OFFER on interface gei0, unit 0
    05/13/17-17:21:02 (ipdhcpc): NOTE:   DHCP server: 192.168.1.1
    05/13/17-17:21:02 (ipdhcpc): WARN:   **WARNING** The DHCP Server did not assign a permanent IP for gei0.
    05/13/17-17:21:02 (ipdhcpc): WARN:               Network access to this controller may eventually fail.
    05/13/17-17:21:02 (ipdhcpc): NOTE:   Client IP routers:   192.168.1.1
    05/13/17-17:21:02 (ipdhcpc): NOTE:   Client DNS name servers: 192.168.1.1
    05/13/17-17:21:02 (ipdhcpc): NOTE:   DNS domain name: fios-router.home
    05/13/17-17:21:02 (ipdhcpc): NOTE:   Assigned IP address: 192.168.1.191
    05/13/17-17:21:02 (ipdhcpc): NOTE:   Assigned subnet mask: 255.255.255.0
    05/13/17-17:21:02 (tNetReset): NOTE:  Network Ready
    Current date: 05/13/17  time: 17:21:12

    NetApp, Inc. RAID Controller
    Copyright 2008-2017, NetApp, Inc. All Rights Reserved.
    Copyright 1984-2006 Wind River Systems, Inc.
    VxWorks: VxWorks 6.5   Kernel: WIND version 2.10
    Model: 2660    Firmware version: 08.20.21.61
    05/13/17-17:21:12 (tRAID): SOD Sequence is Normal, 0 on controller A

    ->
    Serial Port shell started.
    05/13/17-17:21:12 (tRAID): NOTE:  loading flash file: ECTDell
    05/13/17-17:21:12 (tRAID): NOTE:  unloading flash file: ECTDell
    05/13/17-17:21:13 (tRAID): NOTE:  Turning on tray summary fault LED
    05/13/17-17:21:13 (tRAID): NOTE:  SODRebootLoop- Limit:5 Cnt:5
    05/13/17-17:21:13 (tRAID): NOTE:  Flashing tray summary fault LED
    05/13/17-17:21:13 (tRAID): WARN:  Controller entering SODReboot lockdown state...


    At this point the controller goes into this loop five times and then locks. I have tried clearing the locked status with lemClearLockdown, but if there are drives installed the controller goes back into the endless loop (showing the same output as above) and locks itself again. Again, I'm not worried about the data on this array as it has all been migrated I just want to get this working properly. 

    I've been doing some research and I'm wondering if the drives need to be wiped? Would they still have the original configuration and the modules can't load the configuration for some reason?