Isilon: Knooppuntfirmware-upgrade mislukt op CMC

Tijdens een firmware-upgrade voor een knooppunt mislukt de CMC-firmware-update.

In de /var/log/isi_firmware.log op het knooppunt dat is mislukt, is uitvoer te zien die vergelijkbaar is met het volgende:

2017-05-11 15:18:58,991 ERROR    [17093] : Node 6: CMCFirmware._update: update failed: error=65280 output:
TARGET UPGRADE CAPABILITIES
-------------------------------
HPM.1 version............0
Component 0 presence....[y]
Component 1 presence....[y]
Component 2 presence....[y]
Component 3 presence....[y]
Component 4 presence....[n]
Component 5 presence....[n]
Component 6 presence....[n]
Component 7 presence....[y]
Upgrade undesirable.....[n]
Aut rollback override...[n]
IPMC degraded...........[y]
Defered activation......[n]
Service affected........[n]
Manual rollback.........[n]
Automatic rollback......[n]
Self test...............[n]
Upgrade timeout.........[300 sec]
Self test timeout.......[0 sec]
Rollback timeout........[300 sec]
Inaccessibility timeout.[300 sec]

GENERAL PROPERTIES
-------------------------------
Payload cold reset req....[n]
Def. activation supported.[n]
Comparison supported......[n]
Preparation supported.....[y]
Rollback supported........[y]

Description string: CMC FW

Current Version:
 Major: 00
 Minor: 0b
 Aux  : 00 00 00 00

Validating firmware image integrity...OK
Performing preparation stage...TARGET UPGRADE CAPABILITIES
-------------------------------
HPM.1 version............0
Component 0 presence....[y]
Component 1 presence....[y]
Component 2 presence....[y]
Component 3 presence....[y]
Component 4 presence....[n]
Component 5 presence....[n]
Component 6 presence....[n]
Component 7 presence....[y]
Upgrade undesirable.....[n]
Aut rollback override...[n]
IPMC degraded...........[y]
Defered activation......[n]
Service affected........[n]
Manual rollback.........[n]
Automatic rollback......[n]
Self test...............[n]
Upgrade timeout.........[300 sec]
Self test timeout.......[0 sec]
Rollback timeout........[300 sec]
Inaccessibility timeout.[300 sec]

GENERAL PROPERTIES
-------------------------------
Payload cold reset req....[n]
Def. activation supported.[n]
Comparison supported......[n]
Preparation supported.....[y]
Rollback supported........[y]

OK
Error initiating upgrade action, compcode = ff

Firmware upgrade procedure failed

Error aborting upgrade, compcode = 80

...

-----------------------------------------------------------------
|ID | Name      | Active  |   Upload Progress   | Upload| Image |
|   |           | Version |0%      50%      100%| Time  | Size  |
|---|-----------|---------||----+----+----+----||-------|-------|
...Failed...
-----------------------------------------------------------------

In OneFS 8.0, this would also cause output similar to the following for upgrade commands:

Isilon-1# isi upgrade nodes firmware progress list
Lnns  Device               Old Version             New Version             Status
-----------------------------------------------------------------------------------
...
6     CMCSDR_Yeti          00.06                   00.06                   -
6     CMC_Yeti             00.0b                   02.05                   error
6     IsilonFPV1           UI.01.36                UI.01.36                -
...
-----------------------------------------------------------------------------------
Total: 75
Isilon-1# isi upgrade nodes view 6
             Node LNN: 6
   Node Upgrade State: error
        Error Details: Failed Step: InstallFirmware
                       Error Log: /ifs/.ifsvar/upgrade/Agent-6-000e1e591a20232c8a58011809d7e56ff3fb.log
  Last Upgrade Action: -
   Last Action Result: -
Node Upgrade Progress: None
         Node OS Version: 8.0.0.4_build(53)style(5)

Isilon-1# isi_upgrade_logs
Copied log to /ifs/upgrade_error_logs/6_isi_Upgrade_Agent.py-InstallFirmware.log

Isilon-1# cat /ifs/upgrade_error_logs/6_isi_Upgrade_Agent.py-InstallFirmware.log
Starting /usr/sbin/isi_Upgrade_Agent.py InstallFirmware

Verifying boot device mirrors are synced...
2017-05-11 15:18:42,876 DEBUG Execute Command: InstallFirmware
2017-05-11 15:18:59,005 ERROR Failed to update CMC_Yeti firmware on Node 6

Dit probleem wordt veroorzaakt doordat de CMC-component niet tijdig reageert tijdens de update van de knooppuntfirmware.

Het probleem kan worden opgelost door ofwel een koude reset van de BMC- en CMC-componenten op het knooppunt uit te voeren, of, als het koude resetten het probleem niet oplost, moet een stroomonderbreking en stroomafvoer van het knooppunt worden uitgevoerd in overeenstemming met https://support.emc.com/kb/466373 moet worden uitgevoerd.  Als in geen van beide stappen het probleem is opgelost, escaleer dan naar Isilon Technical Support-teams voor verdere analyse.

Om de CMC opnieuw in te stellen, voert u de volgende opdracht uit op het betreffende knooppunt:
/usr/bin/isi_hwtools/isi_ipmicmc -c -a cmc

Voorbeelduitvoer:

Isilon-6# /usr/bin/isi_hwtools/isi_ipmicmc -c -a cmc
Cold reset CMC
Note - 15 sec wait for CMC...
Checking if Firmware Version MSB is set
MSB set in Firmware Version
Waiting 5 seconds before trying again
Success: MSB cleared in Firmware Version
Checking self test results
Success: self test passed

Once reset, retry the firmware update. 
In OneFS versions prior to 8.0:
isi firmware update

In OneFS versions 8.0 and later:
Review the 'Node Upgrade State' for each node, identify the node that is in an error state, this is the node that the upgrade will be retried on:
isi upgrade nodes list

To retry the upgrade on the node in an error state, run the following command, replace LNN with the node that is showing in an error state:
isi upgrade retry-last-action LNN