VRTX's CMC redundancy is lost ?

Hello

You need to resolve the firmware issue on the CMC listed as N/A. You need to troubleshoot to find out if there is an issue with the CMC, slot, or the firmware running on the CMC. I would try powering on the chassis with just the CMC that is showing N/A firmware. If you are able to access the CMC then update the firmware on it to the same version of the other CMC. If it will not function then test it in the other slot. If there is a firmware mismatch between the CMCs then it will cause redundancy issues.

The CMC controls power throughout the system. The nodes should continue operating even if the CMCs are not functional. You will be unable to power on a node without a functional CMC, but nodes already powered should continue functioning. The node shutdown issue is not likely related to the CMC issue.

http://www.dell.com/support/

Thanks

Thanks Daniel !

And if I update the CMC's firmware, the VRTX Server need restart or not ???

A CMC firmware update does not require the nodes or chassis to be restarted. While the CMC is offline, restarting, or updating the fans should run at maximum RPM and the nodes will likely run in a reduced performance state.

I have a similar issue currently in a VRTX Blade, CMC-1 isn´t active and redundancy is lost, and basically when you select it on the CMC website, said firmare version N/A, i got this issue during a firmware upgrade process for both CMC, i would like to know if the CMC card can be removed with the VRTX on Power ON, I would like to remove the CMC card and re-plug to see if that solve the issue, but i dont want to shutdown the all VRTX,  there are two Blades with VMWare esxi installed there 

Hello RKG69,

A shutdown is not required for a reseat as the CMCs are hot pluggable. Everything in the chassis will continue to function.

As you know it involves opening the chassis, so use cautions when opening the system.

Thanks for the quick answer...

Hi Charles:

on the comment by RKG69, how can i force the update on the CMC-1 that shows N/A, we went from CMC version 3.0 to 3.4 following the Dell Article that says to update both CMCs at the same time, the issue here is that now CMC-2 is the active one, CMC-1 either has no FW or has a mismatch on the FW level, we reseeded the board (no hardware issue, both are up but one shows N/A, in this case CMC-1, we tried  racadm fwupdate -g -u -a xxx.xxx.xxx.xxx -d vrtx_cmc_3.40.bin -m cmc-1 and we get ERROR: the syntax of the command specified is not correct.

Hello tstuardo,

Have you tried update the standy through gui?

 1.        Go to any of the following pages:

Chassis Overview>Update

Chassis Overview>Chassis Controller>Update

2.        On the Firmware Update page, in the CMC Firmware section, select the required components under the Update Targets column for the CMC or CMCs (if standby CMC is present) you want to update, and then click Apply CMC Update.

3.        In the Firmware Image field, enter the path to the firmware image file on the management station or shared network, or click Browse to navigate to the file location. The default name of the CMC firmware image file is firmimg.cmc.

Try:

racadm fwupdate -g -u -a https://dell.to/30FySPK -d vrtx_cmc_3.40.bin -m cmc-standby

Additional reference : Page 34 fwupdate

https://dell.to/3cssP3N

Have you tried a re-seat of the CMC?

i did try your command and i got error:'cmc-standby' not listed in inventory

see below pre upgrade

Hi charles:

the standby CMC is present but does not show up on the gui, only the active one appears, pre upgrade both appeared (cmc-1 being the active and cmc-2 being the standby) after the upgrade from 3.0.200 to 3.4 only cmc-2 appears, i connected via serial to the CMC and  i see the below

getmodinfo

Hello tstuardo

I think you indicated you tried a reseat.

Does a fail over allow you to switch over? I think if it's not showing up that may not work,.

Have you tried swapping slots or with just the CMC showing Not OK?

You could consider a maintenance window

*Try swapping slots

*Try with just the CMC reporing Not OK

If in either event you can get access update the firmware

Hi Charles R, following the case above, these tasks were done on Saturday, but we still are facing issues with one of the CMC card

What we did was:
Action 1: Reseat cmc-1.
Result: Cmc-1 tries to upgrade from 3.0 to 3.4, fails, then cmc-2 boots to 3.1 without issues.
Cmc-1 present standby not ok.
Cmc-2 present active ok.

Action 2 : Remove cmc-2 to force booting into cmc-1.
Result: Cmc-1 tries to upgrade from 3.0 to 3.4 automatically, fails.
Keeps booting to recovery. Additional commands like "recover getniccfg" are not accepted.
Error observed over the serial connection:Updating image "/root_fs"
Verify image crc32: bytes=0x2ac0000, crc32=0x9f6448df
Writing data at 0x2811000 --  87% complete.writing NAND page at offset 0x2832000 failed
Data did not fit into device, due to remapped blocks
FAIL: 1193:02:01:628 Verified redundant update image "3.40 200" at 0x6000000

Action 3: Remove the cmc-1 SD card and format to Fat32. Reseat cmc-1 only.
Result: Cmc-1 tries to upgrade from 3.0 to 3.4 automatically , fails.
Same error as above observed on the serial connection.Conclusion: The cmc-1 itself must have some local storage besides the SD card (which is recognized as extended storage) so every time it boots, finds itself running version 3.0 but with the version 3.4 available locally and tries to auto upgrade. Fails writing data with the error "data did not fit into device".

Not sure what else we can do about it, any other option beyond replace the CMC for a new one?

Regards

Hi Charles:

we tried with rkg69 a couple of times this saturday, so what we see with cmc1 is that once you type "recover" it goes on a loop and it gets stuck on the same error "Writing data at 0x2811000 --  87% complete.writing NAND page at offset 0x2832000 failed
Data did not fit into device, due to remapped blocks
FAIL: 1193:02:01:628 Verified redundant update image "3.40 200" at 0x6000000", so what are our options?

Hello RKG69 and  tstuardo

Given the extensive trouble shooting that has been completed, including swapping slots where issue follows the CMC, it looks like you will need to replace the CMC. 

If I can find anything else that can help I will post it.  Maybe the community has some members that has experienced and could have additional input.