I have a MD3200i running firmware version 07.84.53.60 on the RAID module. This MD3200i has always only had one controller. Everything has been running fine but I want to add a second module to this system.
Are there any gotchas that I should be aware of before attempting this? Do I need to power down the MD3200i and all ESXi hosts attached to it before installing the module? What if the new module has a newer firmware version than the firmware installed on the current module?
So I went through the steps outlined in the document, along with updating the firmware of the controllers, drives and updating to the latest version of MDSM. The controllers started working for about 3 minutes and then the array was unavailable. I have already backed up all of the data on the array and migrated, so at this point I'm looking to restore to factory settings.
The only way that I'm able to access the management port from MDSM is if all of the disks are removed. If I try to boot the array with 1 controller and 2 disks are installed in slot 0 and slot 1 I get the following from the console output:
Driver Build Date:Jan 18 2017, Time:16:06:27 ServerEngines iSCSI Target Driver loaded iSCSI Firmware Version:2.901.326.0 iSCSI Driver Version:2.901.242.0 Found BladeEngine2 in Gen2 x8 PCIe slot 05/13/17-17:20:20 (tRAID): NOTE: RPA memory region : start 0x471977e00 size 16777216 (ConIO-size 16777216) registered for pre-post data buffers on port 0 05/13/17-17:20:21 (tRAID): NOTE: sebeReadEEPROMChecksum: EEPROM checksum match: 0x81986278 05/13/17-17:20:21 (tRAID): NOTE: Target:irq_ptr:0x0xe3aaf0, func:0x1, bcmId:0x1 05/13/17-17:20:21 (tRAID): NOTE: VendorID 0x19a2 DeviceId 0x210 bcmId 1 BDF 2.0.1 05/13/17-17:20:21 (tRAID): NOTE: Target:VKI_INT_CONNECT success:status:0x0 05/13/17-17:20:21 (tRAID): ERROR: Target:ISR registered:irq_ptr:0xe3aaf0, phba:0x2f03fe8, func:0x1 05/13/17-17:20:21 (tRAID): NOTE: BcmId [1] PCI [2.0.1] 05/13/17-17:20:21 (tRAID): NOTE: DevId 0x0210 VendId 0x19a2 05/13/17-17:20:21 (tRAID): NOTE: BAR0 IO base 0x00000000 size 0 05/13/17-17:20:21 (tRAID): NOTE: BAR1 MEM base 0xf0260000 size 1 05/13/17-17:20:21 (tRAID): NOTE: BAR2 MEM base 0xf0280000 size 1 05/13/17-17:20:21 (tRAID): NOTE: BAR4 MEM base 0xf02a0000 size 1 05/13/17-17:20:21 (tRAID): NOTE: MPU_EP_SEMAPHORE 0x0000c000 Driver Build Date:Jan 18 2017, Time:16:06:27 ServerEngines iSCSI Target Driver loaded iSCSI Firmware Version:2.901.326.0 iSCSI Driver Version:2.901.242.0 Found BladeEngine2 in Gen2 x8 PCIe slot 05/13/17-17:20:22 (tRAID): NOTE: RPA memory region : start 0x472977e00 size 16777216 (ConIO-size 16777216) registered for pre-post data buffers on port 1 05/13/17-17:20:22 (tRAID): NOTE: sebeReadEEPROMChecksum: EEPROM checksum match: 0x81986278 05/13/17-17:20:23 (tRAID): NOTE: Target:irq_ptr:0x0xe3a9a8, func:0x2, bcmId:0x2 05/13/17-17:20:23 (tRAID): NOTE: VendorID 0x19a2 DeviceId 0x210 bcmId 2 BDF 2.0.2 05/13/17-17:20:23 (tRAID): NOTE: Target:VKI_INT_CONNECT success:status:0x0 05/13/17-17:20:23 (tRAID): ERROR: Target:ISR registered:irq_ptr:0xe3a9a8, phba:0x2f04708, func:0x2 05/13/17-17:20:23 (tRAID): NOTE: BcmId [2] PCI [2.0.2] 05/13/17-17:20:23 (tRAID): NOTE: DevId 0x0210 VendId 0x19a2 05/13/17-17:20:23 (tRAID): NOTE: BAR0 IO base 0x00000000 size 0 05/13/17-17:20:23 (tRAID): NOTE: BAR1 MEM base 0xf02c0000 size 1 05/13/17-17:20:23 (tRAID): NOTE: BAR2 MEM base 0xf02e0000 size 1 05/13/17-17:20:23 (tRAID): NOTE: BAR4 MEM base 0xf0300000 size 1 05/13/17-17:20:23 (tRAID): NOTE: MPU_EP_SEMAPHORE 0x0000c000 Driver Build Date:Jan 18 2017, Time:16:06:27 ServerEngines iSCSI Target Driver loaded iSCSI Firmware Version:2.901.326.0 iSCSI Driver Version:2.901.242.0 Found BladeEngine2 in Gen2 x8 PCIe slot 05/13/17-17:20:24 (tRAID): NOTE: RPA memory region : start 0x473977e00 size 16777216 (ConIO-size 16777216) registered for pre-post data buffers on port 2 05/13/17-17:20:24 (tRAID): NOTE: sebeReadEEPROMChecksum: EEPROM checksum match: 0x81986278 05/13/17-17:20:25 (tRAID): NOTE: Target:irq_ptr:0x0xe3a860, func:0x3, bcmId:0x3 05/13/17-17:20:25 (tRAID): NOTE: VendorID 0x19a2 DeviceId 0x210 bcmId 3 BDF 2.0.3 05/13/17-17:20:25 (tRAID): NOTE: Target:VKI_INT_CONNECT success:status:0x0 05/13/17-17:20:25 (tRAID): ERROR: Target:ISR registered:irq_ptr:0xe3a860, phba:0x2f04e28, func:0x3 05/13/17-17:20:25 (tRAID): NOTE: BcmId [3] PCI [2.0.3] 05/13/17-17:20:25 (tRAID): NOTE: DevId 0x0210 VendId 0x19a2 05/13/17-17:20:25 (tRAID): NOTE: BAR0 IO base 0x00000000 size 0 05/13/17-17:20:25 (tRAID): NOTE: BAR1 MEM base 0xf0320000 size 1 05/13/17-17:20:25 (tRAID): NOTE: BAR2 MEM base 0xf0340000 size 1 05/13/17-17:20:25 (tRAID): NOTE: BAR4 MEM base 0xf0360000 size 1 05/13/17-17:20:25 (tRAID): NOTE: MPU_EP_SEMAPHORE 0x0000c000 Driver Build Date:Jan 18 2017, Time:16:06:27 ServerEngines iSCSI Target Driver loaded iSCSI Firmware Version:2.901.326.0 iSCSI Driver Version:2.901.242.0 Found BladeEngine2 in Gen2 x8 PCIe slot 05/13/17-17:20:26 (tRAID): NOTE: RPA memory region : start 0x474977e00 size 16777216 (ConIO-size 16777216) registered for pre-post data buffers on port 3 05/13/17-17:20:26 (tRAID): SOD: Pre-Initialization Phase Complete 05/13/17-17:20:34 (tRAID): WARN: cmgr::readSystemRecord: DB system record not found. 05/13/17-17:20:34 (tRAID): WARN: cmgr::isBoardDiff: No controller record 05/13/17-17:20:34 (tRAID): WARN: cmgr::isHostBoardIdDiff: No controller record 05/13/17-17:20:34 (tRAID): WARN: cmgr::isSubModelIdDiff: No controller record 05/13/17-17:20:34 (tRAID): WARN: ACS: testCommunication: Domi Exception caught: IconSendInfeasibleException Error 05/13/17-17:20:34 (tRAID): NOTE: ACS: autoCodeSync(): Process start. Comm Status: 0 05/13/17-17:20:34 (tRAID): WARN: ACS: autoCodeSync(): Skipped since alt not communicating. 05/13/17-17:20:34 (tRAID): ERROR: BackupDatabaseManager:initialReadFromNative Exception: IOException 05/13/17-17:20:34 (tRAID): ERROR: ADM Read-Write Initialize failed on Drive 0xa0010000 with Status I/O Error 05/13/17-17:20:34 (tRAID): ERROR: BackupDatabaseManager::updateRecord: bdbm record is not initialized 05/13/17-17:20:34 (tRAID): ERROR: BackupDatabaseManager::recordTypeModifiedCommon caught DbmIOException: recType: 140, status: -9 05/13/17-17:20:34 (tRAID): ERROR: SubRecord::deleteRecord: Memory allocation error! Aborting txn: 0x0a61a270 05/13/17-17:20:34 (tRAID): WARN: CmgrIoException on write, stat = -11 Line 101 File cmgrSod.cc 05/13/17-17:20:34 (tRAID): NOTE: iocfi: Peering Disabled (Alt Unavailable) 05/13/17-17:20:34 (tRAID): SOD: Code Synchronization Initialization Phase Complete 05/13/17-17:20:35 (tRAID): NOTE: SAFE: Process new features 05/13/17-17:20:35 (tRAID): NOTE: SAFE: Process legacy features 05/13/17-17:20:35 (tRAID): ERROR: BackupDatabaseManager::updateRecord: bdbm record is not initialized 05/13/17-17:20:35 (tRAID): ERROR: BackupDatabaseManager::recordTypeModifiedCommon caught DbmIOException: recType: 140, status: -9 05/13/17-17:20:35 (tRAID): ERROR: SubRecord::writeRecord: listener failed! Aborting txn: 0x0a6ce248 05/13/17-17:20:35 (batteryMgr): ERROR: BackupDatabaseManager::updateRecord: bdbm record is not initialized 05/13/17-17:20:35 (batteryMgr): ERROR: BackupDatabaseManager::recordTypeModifiedCommon caught DbmIOException: recType: 140, status: -9 05/13/17-17:20:35 (batteryMgr): ERROR: SubRecord::writeRecord: listener failed! Aborting txn: 0x09c31fa0 05/13/17-17:20:36 (batteryMgr): ERROR: BackupDatabaseManager::updateRecord: bdbm record is not initialized 05/13/17-17:20:36 (batteryMgr): ERROR: BackupDatabaseManager::recordTypeModifiedCommon caught DbmIOException: recType: 140, status: -9 05/13/17-17:20:36 (batteryMgr): ERROR: SubRecord::writeRecord: listener failed! Aborting txn: 0x09edde18 05/13/17-17:20:36 (tRAID): NOTE: IOManager::restoreData - dataSize:0x2800000, startAddress:0x4779f7e00 05/13/17-17:20:37 (tRAID): NOTE: ncb::IOManager::restoreData - Successful 05/13/17-17:20:37 (tRAID): NOTE: OBB Restore From Flash Completed 05/13/17-17:20:38 (bdbmBGTask): ERROR: sendFreezeStateToAlternate: Exception IconSendInfeasibleException Error 05/13/17-17:20:38 (tRAID): NOTE: SPM acquireObjects exception: IconSendInfeasibleException Error 05/13/17-17:20:38 (tRAID): NOTE: sas: Peering Disabled (Alt unavailable) 05/13/17-17:20:38 (tRAID): NOTE: b_isn: Peering Disabled (Alt Unavailable) 05/13/17-17:20:38 (tRAID): NOTE: ion: Peering Disabled (Alt Unavailable)
Exception: Program (Illegal Opcode) pc: 0x00000000 esr: 0x08000000
NAME TID PRI STATUS PC SP ERRNO ---------- ---------- --- ---------- ---------- ---------- ------- tRAID 0x16fbd60 125 READY 0x1650e0 0x16fb6b0 550002
task stack: base 0x16fbd60 end 0x16f6d60 size 20480 high 5328 margin 15152 exc. stack: base 0x16fd000 end 0x16fc010 start 0x16fd010 exc. stack: size 4080 high 3272 margin 808
-=<###>=- Instantiating /ram as rawFs, device = 0x1 Formatting /ram for DOSFS Instantiating /ram as rawFs, device = 0x1 Formatting...Retrieved old volume params with %38 confidence: Volume Parameters: FAT type: FAT32, sectors per cluster 0 0 FAT copies, 0 clusters, 0 sectors per FAT Sectors reserved 0, hidden 0, FAT sectors 0 Root dir entries 0, sysId (null) , serial number f10000 Label:" " ... Disk with 1024 sectors of 512 bytes will be formatted with: Volume Parameters: FAT type: FAT12, sectors per cluster 1 2 FAT copies, 1010 clusters, 3 sectors per FAT Sectors reserved 1, hidden 0, FAT sectors 6 Root dir entries 112, sysId VXDOS12 , serial number f10000 Label:" " ...
RTC Error: Real-time clock device is not working OK.
Adding 14607 symbols for standalone. 05/13/17-17:20:47 (tSystem): ERROR: FPGA FW is out of date "Rhone03 rev17" currently in use "Rhone03 rev20" available for update 05/13/17-17:20:52 (tNetCfgInit): NOTE: eth0: LinkUp event 05/13/17-17:20:52 (tNetCfgInit): NOTE: Acquiring network parameters for interface gei0 using DHCP 05/13/17-17:21:01 (ipdhcpc): NOTE: netCfgDhcpReplyCallback :: received OFFER on interface gei0, unit 0 05/13/17-17:21:02 (ipdhcpc): NOTE: DHCP server: 192.168.1.1 05/13/17-17:21:02 (ipdhcpc): WARN: **WARNING** The DHCP Server did not assign a permanent IP for gei0. 05/13/17-17:21:02 (ipdhcpc): WARN: Network access to this controller may eventually fail. 05/13/17-17:21:02 (ipdhcpc): NOTE: Client IP routers: 192.168.1.1 05/13/17-17:21:02 (ipdhcpc): NOTE: Client DNS name servers: 192.168.1.1 05/13/17-17:21:02 (ipdhcpc): NOTE: DNS domain name: fios-router.home 05/13/17-17:21:02 (ipdhcpc): NOTE: Assigned IP address: 192.168.1.191 05/13/17-17:21:02 (ipdhcpc): NOTE: Assigned subnet mask: 255.255.255.0 05/13/17-17:21:02 (tNetReset): NOTE: Network Ready Current date: 05/13/17 time: 17:21:12
NetApp, Inc. RAID Controller Copyright 2008-2017, NetApp, Inc. All Rights Reserved. Copyright 1984-2006 Wind River Systems, Inc. VxWorks: VxWorks 6.5 Kernel: WIND version 2.10 Model: 2660 Firmware version: 08.20.21.61 05/13/17-17:21:12 (tRAID): SOD Sequence is Normal, 0 on controller A
-> Serial Port shell started. 05/13/17-17:21:12 (tRAID): NOTE: loading flash file: ECTDell 05/13/17-17:21:12 (tRAID): NOTE: unloading flash file: ECTDell 05/13/17-17:21:13 (tRAID): NOTE: Turning on tray summary fault LED 05/13/17-17:21:13 (tRAID): NOTE: SODRebootLoop- Limit:5 Cnt:5 05/13/17-17:21:13 (tRAID): NOTE: Flashing tray summary fault LED 05/13/17-17:21:13 (tRAID): WARN: Controller entering SODReboot lockdown state...
At this point the controller goes into this loop five times and then locks. I have tried clearing the locked status with lemClearLockdown, but if there are drives installed the controller goes back into the endless loop (showing the same output as above) and locks itself again. Again, I'm not worried about the data on this array as it has all been migrated I just want to get this working properly.
I've been doing some research and I'm wondering if the drives need to be wiped? Would they still have the original configuration and the modules can't load the configuration for some reason?
DELL-Sam L
Community Manager
•
8147 Posts
•
34706 Points
818
0
Posted April 21st, 2017 14:00
Hello FuriousRabbit,
Here is the link to the guide for upgrading your MD3200i from simplex mode to Duplex mode. http://downloads.dell.com/manuals/common/powervault-md3200_user%27s%20guide3_en-us.pdf
Please let us know if you have any other questions.