jeremyw2
1 Copper

Help! production CX500 down!

I'm not a real SAN guy so I'll explain as much as I can. We had a thermal event in our server room. 94 degrees! Some things began thermal shutdowns. Our CX500 became unresponsive via either of the management cards (processors). It's up and apprears to be running but no storage is available on our Hyper-V servers. We do know our C9120 Fiber switch shutdown but now seems to be fine. SAN is completely unavailable via webpage. So I have used the serial/console port to connect to it and try to see what is happening. On both cards I was getting the following:

.... Storage System Failure - Contact your Service Representative ...

ErrorCode: 0x00000133 ErrorDesc: Device: LCC 0 UART FRU: STORAGE PROCESSOR Description: LCC Error Reply Error! Verify of Invalid Enclosure Address failed Type: 17, Error: 0, Detect Addr: 0, Cmd: 90, Orig Encl Addr: 0 EndError: ErrorTime: 09/25/2015 17:47:34

I happen to have a spare CX500 sitting in the office. I took a Processor card from it and put it in the B slot of our production CX500. To which I received the following on the console connection:

AG: init DMD to FC_SPEED_2_GBPS FCDMTL 0 [1.4.0] Cached memory - 0xF77B9 bytes @ 0x1A721280 FCDMTL 0 [1.4.0] Noncached memory - 0xC037F bytes @ 0x1A660EA0 (0x1A660EA0 phys) FCDMTL 0 [1.4.0] DVM Initialized FCDMTL 0 [1.4.0] IMQ base ptr = 1A6D0000; IMQ length = 8000 Dualmode fibre init completed FCDMTL 0 [1.4.0] TPM Notify: st=0xA000000, flg=0x4, cmd=0x1 FCDMTL 0 [1.4.0] TPM Hndle API Event: cntx=0x1A818F44, evnt=0x4002, info=0x0 FCDMTL 0 [1.4.0] TPM Lnk Up: state=0xA000000, flg=0x84 Link Event: 0x00030005 Device Event (0xEF): 0x00030012, tach_ptr: 0x1C0640BC Device Event (0xE8): 0x00030012, tach_ptr: 0x1C0640BC Device Event (0xE4): 0x00030012, tach_ptr: 0x1C0640BC Device Event (0xE2): 0x00030012, tach_ptr: 0x1C0640BC Device Event (0xE1): 0x00030012, tach_ptr: 0x1C0640BC DL waited 1s for discovery Target 0 is online Target 1 is online Target 2 is online Target 3 is online Target 4 is online Relocating Data Directory Boot Service (DDBS: Rev. 02.08)... Autoflash POST? DDBS: MDDE (Rev 2) on disk 0 POST/DIAG image located at sector LBA 0x00012048 Autoflash BIOS? DDBS: MDDE (Rev 2) on disk 0 BIOS image located at sector LBA 0x00011048 ************************************************************ *                        WARNINGS ************************************************************ WARNING: LCC0 DISK STATUS: ~ins=0 flt=0 byp=0 PBC=2 WARNING: LCC 1 UART: Duplicate Enclosure Address Found [2] ************************************************************ EndTime: 09/25/2015 19:16:39 DDBS: K10_REBOOT_DATA: Count = 1 DDBS: K10_REBOOT_DATA: State = 0 DDBS: K10_REBOOT_DATA: ForceDegradedMode = 0 DDBS: SP B Normal Boot Partition DDBS: MDDE (Rev 2) on disk 1 DDBS: MDDE (Rev 2) on disk 3 DDBS: MDB read from both disks. DDBS: Chassis and disk WWN seeds match. DDBS: First disk is valid for boot. DDBS: Second disk is valid for boot. FLARE image (0x00400009) located at sector LBA 0x0002284C Disk Set: 1 3 Total Sectors: 0x00583F29 Relative Sectors: 0x0000003F Calculated mirror drive geometry: Sectors: 63 Heads: 255 Cylinders: 360 Capacity: 5783400 sectors Total Sectors: 0x00583F29 Relative Sectors: 0x0000003F Calculated mirror drive geometry: Sectors: 63 Heads: 255 Cylinders: 360 Capacity: 5783400 sectors int13 - RESET (1) int13 - READ PARAMETERS (3) int13 - RESET (5) int13 - READ PARAMETERS (7) int13 - READ PARAMETERS (24) int13 - READ PARAMETERS (515) int13 - CHECK EXTENSIONS PRESENT (516) int13 - GET DRIVE PARAMETERS (Extended) (517) int13 - READ PARAMETERS (520) int13 - CHECK EXTENSIONS PRESENT (521) int13 - GET DRIVE PARAMETERS (Extended) (522) int13 - READ PARAMETERS (524) int13 - CHECK EXTENSIONS PRESENT (525) int13 - GET DRIVE PARAMETERS (Extended) (526) int13 - READ PARAMETERS (535) int13 - CHECK EXTENSIONS PRESENT (536) int13 - GET DRIVE PARAMETERS (Extended) (537) int13 - DRIVE TYPE (557) int13 - READ PARAMETERS (558) int13 - DRIVE TYPE (559) int13 - CHECK EXTENSIONS PRESENT (561) int13 - GET DRIVE PARAMETERS (Extended) (562) int13 - READ PARAMETERS (563) int13 - CHECK EXTENSIONS PRESENT (564) int13 - GET DRIVE PARAMETERS (Extended) (565) int13 - READ PARAMETERS (575) int13 - CHECK EXTENSIONS PRESENT (576) int13 - GET DRIVE PARAMETERS (Extended) (577) int13 - READ PARAMETERS (580) int13 - CHECK EXTENSIONS PRESENT (581) int13 - GET DRIVE PARAMETERS (Extended) (582) int13 - READ PARAMETERS (1521) int13 - CHECK EXTENSIONS PRESENT (1522) int13 - GET DRIVE PARAMETERS (Extended) (1523) int13 - READ PARAMETERS (1538) int13 - CHECK EXTENSIONS PRESENT (1539) int13 - GET DRIVE PARAMETERS (Extended) (1540) int13 - READ PARAMETERS (1554) int13 - CHECK EXTENSIONS PRESENT (1555) int13 - GET DRIVE PARAMETERS (Extended) (1556) int13 - READ PARAMETERS (1572) int13 - CHECK EXTENSIONS PRESENT (1573) int13 - GET DRIVE PARAMETERS (Extended) (1574) int13 - READ PARAMETERS (1590) int13 - CHECK EXTENSIONS PRESENT (1591) int13 - GET DRIVE PARAMETERS (Extended) (1592) int13 - READ PARAMETERS (1664) int13 - CHECK EXTENSIONS PRESENT (1665) int13 - GET DRIVE PARAMETERS (Extended) (1666) int13 - READ PARAMETERS (1681) int13 - CHECK EXTENSIONS PRESENT (1682) int13 - GET DRIVE PARAMETERS (Extended) (1683) int13 - READ PARAMETERS (1698) int13 - CHECK EXTENSIONS PRESENT (1699) int13 - GET DRIVE PARAMETERS (Extended) (1700) int13 - READ PARAMETERS (1726) int13 - CHECK EXTENSIONS PRESENT (1727) int13 - GET DRIVE PARAMETERS (Extended) (1728) int13 - READ PARAMETERS (1765) int13 - CHECK EXTENSIONS PRESENT (1766) int13 - GET DRIVE PARAMETERS (Extended) (1767)

So I'm not sure how to proceed from here. We don't have support on this unit and a local company wants $2200 just to come look to see if they can fix it. Any advice is appreciated. Thanks!

Labels (1)
Reply
1 Reply
Highlighted
kelleg
5 Tungsten

Re: Help! production CX500 down!

Your best bet is to contact EMC support. This is a customer forum and as such is not a true support forum.

There is one message that seems important:

Duplicate Enclosure Address Found


This is the Disk Enclosures - the DAE's - check that each DAE has a unique address - this is on the back of each DAE in the middle. If the Address of the DAE was changed for some reason (on the older DAE's this was a switch) then the disks will not be addressed correctly.


glen

Reply