This post is more than 5 years old
10 Posts
0
3625
Clariion CX3-10c both SPE stuck in POST
Hello All.
I recently acquired a CX3-10c from a client; comprising of a SPE (SPA/B), SPS and 2 Disk enclosures.
I have hooked everything up and the SPE begins the normal boot cycle.
There are no fault lights on SPA or B nor the DA's.
The problem is that SPA and B do not come out of POST.
Both SPs are showing the same issue.
On each SP:
1. Power LED is on solid and Green
2. SP fault LED is amber and blinking at 1Hz.
3. I/O Module LED is Off
4. BE Port Link is solid Blue
5. FE Port is link is Off (not connected)
Inside the SPs between the BE port and the FE ports the following LEDs are cycling with the following pattern.
CR5 - Green
CR10, Cr13 - Green, CR11, CR14 - Amber
CR5 - Green, CR14 - Amber
Each SP cycles through this diagnostic light arrangement while the SP fault light is blinking at 1 Hz.
There is no activity on the DAE Vault drives and the SP fault light never goes to 4 Hz, so I assume that Flare is not loading. I cannot discover or ping the SPs on the standard or known IPs.
All comments are welcome. I am an Electrical Engineer, so if I need to get inside or run diagnostics, that is not a problem.
I do not have the serial/console cable, but can get one if need be.
I am using this personally, so technical advice is much appreciated.
Thanks in advance.
Michael.
Message was edited by: mchristsen I updated the correct internal light sequence.
mchristsen1
10 Posts
0
April 8th, 2013 18:00
Glen
Thanks for the tip. However out of a fit of genius, I checked the cable hook ups again and found that SPA was plugged into the PRI port of LCCB and vice versa for SPB. Once I swapped the cabling so that SPA was connected to LCCA and so on, the system booted
I came to this conclusion when I hooked up the second enclosure (without the Vault discs) directly to SPA/B and the error changed to "Not enclosure 0" or something like that. That told me that the cabling was communicating correctly between the enclosures and the SPs. Obviously after going back through the setup manual, it clearly shows that the "A" side is all connected and so for the "B" side.
In any case, thank you for your help on this issue and with my other post concerning the cable models.
Michael.
Vipin VK
812 Posts
0
March 28th, 2013 22:00
Just check the backend cable connectivity. The BE0 port from each SP should be connected to the LCC's PRI port ( of first DAE's - DAE-OS)
mchristsen1
10 Posts
0
March 29th, 2013 07:00
Vipin
Thanks, I checked each cable and both are seated correctly into their respective places. Each HSSDC2 end is plugged into the PRI port of the DAE that (Bus 0, Unit 0). The speed light on both the DAE and the SP show the link is 4GB (solid blue).
During startup on the SP, BE0 lights up blue when the link is established, the other ports (iSCSI are initialized), then BE0 flashed Blue/Green about 9 times then goes Blue. I assume this is link testing and negotiation. It does not stay Blue/Green.
The model number of the SP-DAE cables are 038-003-503 Rev A03 2.1m
Michael.
Vipin VK
812 Posts
0
March 29th, 2013 07:00
What about the vault drives (first 5 disks in DAE0 with a warning label) are they in place properly and all are glowing in green ?
Also ensure the Storage processors are good without any physical damage etc... and also check the DIMMs are properly seated.
Is it possible for you to collect the boot logs from the both SPs ?
mchristsen1
10 Posts
0
March 29th, 2013 08:00
Vipin
I pulled out SPA/B and reseated the memory. The Vault drives were seated and they spin up and are green. No damage to the SP. I noticed there is a Clear CMOS jumper in the SP. Would clearing CMOS help or cause more problems?
As for collecting the boot logs, I need to get my hands on the Serial cable to connect to the console (I assume that is how you get the boot logs). If there is another way, please let me know. I did try to ping the LAN service ports on the designated IPs, but no luck there.
Thanks again for you assistance.
Michael.
Vipin VK
812 Posts
1
March 29th, 2013 08:00
I will not recommend any such troubleshooting as these are not done in field.
Yes, serial cables are required for collecting boot logs and these logs will give a clear picture why it fails to boot . As both the SPs are in similar state, the issue should be in the boot path (i.e, backend cables, LCCs, vault drives, or wrong connectivity)
kelleg
4.5K Posts
1
March 29th, 2013 10:00
You will need to establish a serial connection, During boot, the serial port is set to 9600 baud and you use a terminal window to see the POST. You would want to turn on capture in the terminal window to capture the boot messages- that can tell support where in the boot process it''s failing. After POST, the serial port is set to 115k baud.
glen
mchristsen1
10 Posts
0
March 29th, 2013 10:00
Vipin
Not a problem. I will get the cable to get the logs.
Is there a link on the directions for collection the logs? Is it a direct serial or do I need to create the PPP connection and connect to http://192.168.1.1/setup (or 1.2 in case of SPB).
Michael.
mchristsen1
10 Posts
0
March 29th, 2013 11:00
Glen
Appreciate the tips and thanks for the serial information.
Michael.
kelleg
4.5K Posts
1
March 29th, 2013 15:00
Some addition detail:
Connect a serial cable from a Windows server/desktop/laptop. The cable connects from a Comm port on the computer to the serial port on SPA.
The Comm Port on the computer should be set to:
• 9600 baud
• 8-bit
• No parity
• 1 stop-bit
• No flow control
Start HyperTerminal Logging (Mandatory)
Turn on HyperTerminal logging of the entire conversion process. This is done with the Transfer and Capture Text… menus of HyperTerminal.
• This log will be needed for Triage if an issue occurs that requires escalation.
Reboot the Storage Processor.
You should see the POST messages. At some point, it will stop outputing these messages and the two lights on the SP will stop blinking and stay steady.
The POST lights should blink fast (4 per sec), medium (1 per sec), slow (4 sec per blink), the stop - that's when the SP is fully booted and you should see no more messages. At this point, the baud rate will change to 115k and you'll need to establish a PPP connection.
The blink rate of 1 per sec indicates that the SP is stuck in POST testing.
glen
mchristsen1
10 Posts
0
April 3rd, 2013 17:00
OK, so I finally got the Console cable and captured the boot log (attached). It seems to be related to cabling or the LCC.
I did not receive the cables when I got the SPE/DAEs. I had to buy the cables separate as I did not receive them. The cables I have are 038-003-503 for the SP to DAE and 038-003-509 fro the DAE to DAE.
Are these correct? If not, which part numbers are. I tried searching Powerlink, but I could not find a Bill of Materials for a CX3-10c.
Thanks for you input.
1 Attachment
EMC.TXT.txt
mchristsen1
10 Posts
0
April 5th, 2013 20:00
Does anyone have any comments on the boot log file I captured? It seems that I have the right cables. Any ideas on next steps for trouble shooting?
Michael.
kelleg
4.5K Posts
0
April 8th, 2013 13:00
Seems to be an error when attempting to initialize the backend ports - the ones that go from the SP to the DAE
FCDMTL 8 [9.0.0] Unknown I/O hw_type : 0x00020011
FCDMTL 8 [9.0.0] SFP *** Waiting for atd_ptr->api_init_state = FC_CTLR_INIT_STATE_ONLINE; ***
As this is a DPE (the SP and disks are in the same enclosure) you should try disconnecting all cables, pull all disks except for the first and third (disk 0 and disk 3) disks starting on the left, pull out SPB, then try booting SPA. If that does not work, then try with SPB - pull SPA, pull all disks but the 2nd and 4th (disks 1 and 3) and power on SPB.
If it still does not work then check that the first 5 disks (0-4) are labled as OS disks in the correct order for the slot. It sounds like the flare disks may not have an OS installed. If that is the case, there are two options: purchase new OS disks or get EMC support on-site to verify that it is the disks and not something else.
glen