Isilon: Node-Firmwareupgrade schlägt auf CMC fehl

Während eines Node-Firmwareupgrades schlägt das CMC-Firmwareupdate fehl/tritt ein Fehler auf.

In der /var/log/isi_firmware.log auf dem Node, der fehlgeschlagen ist, wird eine Ausgabe ähnlich der folgenden angezeigt:

2017-05-11 15:18:58,991 ERROR    [17093] : Node 6: CMCFirmware._update: update failed: error=65280 output:
TARGET UPGRADE CAPABILITIES
-------------------------------
HPM.1 version............0
Component 0 presence....[y]
Component 1 presence....[y]
Component 2 presence....[y]
Component 3 presence....[y]
Component 4 presence....[n]
Component 5 presence....[n]
Component 6 presence....[n]
Component 7 presence....[y]
Upgrade undesirable.....[n]
Aut rollback override...[n]
IPMC degraded...........[y]
Defered activation......[n]
Service affected........[n]
Manual rollback.........[n]
Automatic rollback......[n]
Self test...............[n]
Upgrade timeout.........[300 sec]
Self test timeout.......[0 sec]
Rollback timeout........[300 sec]
Inaccessibility timeout.[300 sec]

GENERAL PROPERTIES
-------------------------------
Payload cold reset req....[n]
Def. activation supported.[n]
Comparison supported......[n]
Preparation supported.....[y]
Rollback supported........[y]

Description string: CMC FW

Current Version:
 Major: 00
 Minor: 0b
 Aux  : 00 00 00 00

Validating firmware image integrity...OK
Performing preparation stage...TARGET UPGRADE CAPABILITIES
-------------------------------
HPM.1 version............0
Component 0 presence....[y]
Component 1 presence....[y]
Component 2 presence....[y]
Component 3 presence....[y]
Component 4 presence....[n]
Component 5 presence....[n]
Component 6 presence....[n]
Component 7 presence....[y]
Upgrade undesirable.....[n]
Aut rollback override...[n]
IPMC degraded...........[y]
Defered activation......[n]
Service affected........[n]
Manual rollback.........[n]
Automatic rollback......[n]
Self test...............[n]
Upgrade timeout.........[300 sec]
Self test timeout.......[0 sec]
Rollback timeout........[300 sec]
Inaccessibility timeout.[300 sec]

GENERAL PROPERTIES
-------------------------------
Payload cold reset req....[n]
Def. activation supported.[n]
Comparison supported......[n]
Preparation supported.....[y]
Rollback supported........[y]

OK
Error initiating upgrade action, compcode = ff

Firmware upgrade procedure failed

Error aborting upgrade, compcode = 80

...

-----------------------------------------------------------------
|ID | Name      | Active  |   Upload Progress   | Upload| Image |
|   |           | Version |0%      50%      100%| Time  | Size  |
|---|-----------|---------||----+----+----+----||-------|-------|
...Failed...
-----------------------------------------------------------------

In OneFS 8.0, this would also cause output similar to the following for upgrade commands:

Isilon-1# isi upgrade nodes firmware progress list
Lnns  Device               Old Version             New Version             Status
-----------------------------------------------------------------------------------
...
6     CMCSDR_Yeti          00.06                   00.06                   -
6     CMC_Yeti             00.0b                   02.05                   error
6     IsilonFPV1           UI.01.36                UI.01.36                -
...
-----------------------------------------------------------------------------------
Total: 75
Isilon-1# isi upgrade nodes view 6
             Node LNN: 6
   Node Upgrade State: error
        Error Details: Failed Step: InstallFirmware
                       Error Log: /ifs/.ifsvar/upgrade/Agent-6-000e1e591a20232c8a58011809d7e56ff3fb.log
  Last Upgrade Action: -
   Last Action Result: -
Node Upgrade Progress: None
         Node OS Version: 8.0.0.4_build(53)style(5)

Isilon-1# isi_upgrade_logs
Copied log to /ifs/upgrade_error_logs/6_isi_Upgrade_Agent.py-InstallFirmware.log

Isilon-1# cat /ifs/upgrade_error_logs/6_isi_Upgrade_Agent.py-InstallFirmware.log
Starting /usr/sbin/isi_Upgrade_Agent.py InstallFirmware

Verifying boot device mirrors are synced...
2017-05-11 15:18:42,876 DEBUG Execute Command: InstallFirmware
2017-05-11 15:18:59,005 ERROR Failed to update CMC_Yeti firmware on Node 6

Dieses Problem wird dadurch verursacht, dass die CMC-Komponente während des Node-Firmwareupdates nicht rechtzeitig reagiert.

Das Problem kann entweder durch einen Kaltstart der BMC- und CMC-Komponenten auf dem Node behoben werden, oder, falls der Kaltstart das Problem nicht behebt, muss ein Herunterfahren und eine Stromableitung des Node gemäß https://support.emc.com/kb/466373 durchgeführt werden.  Wenn das Problem durch keinen der Schritte behoben werden kann, eskalieren Sie zur weiteren Analyse an die Teams des technischen Isilon-Supports.

Um den CMC zurückzusetzen, führen Sie den folgenden Befehl auf dem betroffenen Node aus:
/usr/bin/isi_hwtools/isi_ipmicmc -c -a cmc

Beispielausgabe:

Isilon-6# /usr/bin/isi_hwtools/isi_ipmicmc -c -a cmc
Cold reset CMC
Note - 15 sec wait for CMC...
Checking if Firmware Version MSB is set
MSB set in Firmware Version
Waiting 5 seconds before trying again
Success: MSB cleared in Firmware Version
Checking self test results
Success: self test passed

Once reset, retry the firmware update. 
In OneFS versions prior to 8.0:
isi firmware update

In OneFS versions 8.0 and later:
Review the 'Node Upgrade State' for each node, identify the node that is in an error state, this is the node that the upgrade will be retried on:
isi upgrade nodes list

To retry the upgrade on the node in an error state, run the following command, replace LNN with the node that is showing in an error state:
isi upgrade retry-last-action LNN