FC630 Blade Power Light Blinking - will not power on

Hi, thanks for choosing Dell, at this point I would take the server to a minimum to POST configuration.

That's what we are thinking as well, maybe more of a chassis problem than a server problem since all four acted up at the same time.  We did clear the NVRAM and drained the board by pressing on the power button (undocked) for 15 seconds (read that somewhere).

What steps could we take to troubleshoot the chassis?  It is an FX2s.

Done and done.  Unfortunately the green power button still blinks on/off with the minimal boot hardware.

More information, we removed the rSPI card in order to install the IDSDM and that's where our trouble started.

This happened to all four FC630's at the same time.  We reinstalled the rSPI cards but the same thing, green flashing power button.

In the CMC it knows there are servers there but can't identify them, nor can it control them.  Since the issue we have updated the CMC firmware as well.

Hello,

You might also try performing a power drain on each of the server nodes, if you haven't already. The NVRAM jumper on the board would clear NVRAM (and also reset BIOS values to defaults), which may also help. If all 4 servers continue to exhibit the behavior, I'd be inclined to troubleshoot the chassis.

The CMC logs are really the best option. The log collection that gets generated following the page below is very encompassing, but can also be pretty busy. I'd recommend searching it for terms like "Server 1" (might also be Server-1, I forget the specific notation). There is also an active errors section that searching for "Errors" should help pull out. These logs can be a bit much, if you're not used to text log files. Pretty much everything the chassis can tell you is contained in this output, so the odds of it being able to give some idea of the issue is pretty good.

https://dell.to/35uOPY9

I was able to get the log file dump and looked it over. It seems the controller cannot inventory the servers nor can it initiate the iDRAC. 

I'm seeing the CMC "FAILED to query Blade-1 power state. rc=20744 CompCode=255".

This error is on each blade.

Blade Inventory is showing this for each blade:

**** FW inventory - detailed for Blade 1 ****
Inventory collection failed 0 components available, rc 20995

Also seeing this:

core-services.service: Main process exited, code=exited, status=1/FAILURE
 systemd[1]: core-services.service: Failed with result 'exit-code'.
 systemd[1]: Failed to start CMC core services checkpoint.

More power stuff:

mms_pwrmgmt_chassis_power_state_get (Line 314): Failed to acquire shm lock(status=4108)

HandleBladeInsert:

HandleBladeInsert(): FAILED to find Blade-(0,0) Geometry=1

Anyway, like you said, the log file is huge. Nothing really jumps out except the power stuff.  As a precaution I changed out the power supplies as I saw a heat sink failure (then ran this again and it cleared).

Would it be easier to share the log file?

Thanks for your assistance; this is bugging the you-know-what out of me.

Based on that log snippet, it seems like the CMC itself may be the issue, what with it reporting that core services failed to start. The FX2s only has a singular CMC, so a failover isn't an option. If there's another chassis you can test one of these nodes in, that may be a good plan, just to confirm the chassis as the issue.

As to the log file, you should be able to upload it at the URL below and we can see if we can find anything actionable in there.

https://dell.to/3cMtTjI

Thank you, log file uploaded.

Unfortunately I don't have another chassis but I may be able to get my hands on another CMC to test with.

Thank you for being willing to check out the log; If I knew it was the CMC for sure I'd be a bit relieved. The last thing I want is to replace four blades or the chassis!

Testing with another CMC would accomplish mostly the same desired effect. If you can do this, it would probably be helpful.

Only other thing we'll need is the service tag used to make the upload. If you share it here, we can remove it from the forum. You're also welcome to PM it to YoungAh, or myself.

Service Tag: {removed by mod to protect privacy}

Have you by chance noticed any odd activity with the fans? The only thing that has jumped out at me thus far is some interesting fan readings, like a -1 tachometer reading, which I'd bet is to represent an error condition. Some more time needs to be spent with this log, but we do have it and this fan behavior could explain why the system refuses to power the nodes up, but I'd expect to see iDRAC data, still.

What happens if you try to access the iDRAC of a node directly? Does it give you anything, or does it just act like an unresolvable webpage?

SSH does work with the iDRAC in much the same way as it does the CMC. Connecting and then issuing something like racadm racreset or racadm getsel might be good commands, one just resets the iDRAC, while the other should output the SEL contents to the screen. This should at least show if the iDRAC is working, and in case of getsel, potentially show any additional info.

The only unexpected fan behavior (which might be normal) is when the chassis power is off and the CMC is removed, the fans suddenly go to full speed and stay there until power is disconnected or the CMC is put back in.  With the chassis power off the fans still operate (I assume to keep the CMC cool) at a very low level (or at least the one fan in front of the CMC).

Another oddity was that with all four blades installed today I noticed the CMC web interface said that blade slot 3 & 4 were one and it wouldn't recognize a server being in that slot.  A warm boot of the CMC got it to come up correctly.

I've tried to browse to the iDRAC (or at least the IP address that shows up on the switch) and don't get any web page.

Is the iDRAC capable of accepting ssh traffic?

Thank you.