Dell VRTX CMC failed but did not failover properly

Hi,

We had an issue this week where we were not able to login to the web gui for our CMC so we rebooted the entire chassis (a complete shutdown). When we fired it up again we were able to login to the CMC but it had two issues:

1.) The CMC had lost redundancy and the link light on one of the redundant links was not working

2.) We could not power on any of the blade servers either using the webgui or by manually powering them on. The webgui reported that idrac was not working and indeed it was not showing any information on the servers.

I tried reseating the blades and also using the racadm command to virtually reseat them to no avail. I could still not power them on. There was no information in any of the logs.

After speaking to dell support (who were really great it has to be said), they advised me to swap over the cmc cards in the chassis which I did and then we were able to power up the blades but the cmc was still reporting that the cmc was not redundant. Dell sent a new cmc out and it is now fitted and the issue has gone away.

What is concerning is that this server is in test at the moment and about to be deployed in to a colocation facility. How is it that this server (being a flagship of the dell server product line) with the CMC redundancy built in, stopped the powering up of the blade servers if the CMC card failed? To me it sounds like there is something in the way the software or the hardware has been built that is not redundant. If CMC one fails then we should be able to power on the blades without having to swap the cmc cards over. Is IDRAC hardwired into CMC1 or something?

I would like to get to the bottom of this issue as it is concerning. At least, I know now if it does it again that we need to swap the cards over but a really pain to have to go to a colo facility and pull out all the kit to do it.

I would be very grateful for any help, advice or suggestions that could help make sure this issue never happens again and I would expect that all dell vrtx customers would benefit from this issue being resolved if indeed it is not a one off.

thanks,

Paul