Isilon: L'aggiornamento del firmware del nodo non riesce su CMC

Durante l'aggiornamento del firmware di un nodo, l'aggiornamento del firmware CMC non riesce/si verifica un errore.

In /var/log/isi_firmware.log sul nodo in cui si è verificato l'errore, verrà visualizzato un output simile al seguente:

2017-05-11 15:18:58,991 ERROR    [17093] : Node 6: CMCFirmware._update: update failed: error=65280 output:
TARGET UPGRADE CAPABILITIES
-------------------------------
HPM.1 version............0
Component 0 presence....[y]
Component 1 presence....[y]
Component 2 presence....[y]
Component 3 presence....[y]
Component 4 presence....[n]
Component 5 presence....[n]
Component 6 presence....[n]
Component 7 presence....[y]
Upgrade undesirable.....[n]
Aut rollback override...[n]
IPMC degraded...........[y]
Defered activation......[n]
Service affected........[n]
Manual rollback.........[n]
Automatic rollback......[n]
Self test...............[n]
Upgrade timeout.........[300 sec]
Self test timeout.......[0 sec]
Rollback timeout........[300 sec]
Inaccessibility timeout.[300 sec]

GENERAL PROPERTIES
-------------------------------
Payload cold reset req....[n]
Def. activation supported.[n]
Comparison supported......[n]
Preparation supported.....[y]
Rollback supported........[y]

Description string: CMC FW

Current Version:
 Major: 00
 Minor: 0b
 Aux  : 00 00 00 00

Validating firmware image integrity...OK
Performing preparation stage...TARGET UPGRADE CAPABILITIES
-------------------------------
HPM.1 version............0
Component 0 presence....[y]
Component 1 presence....[y]
Component 2 presence....[y]
Component 3 presence....[y]
Component 4 presence....[n]
Component 5 presence....[n]
Component 6 presence....[n]
Component 7 presence....[y]
Upgrade undesirable.....[n]
Aut rollback override...[n]
IPMC degraded...........[y]
Defered activation......[n]
Service affected........[n]
Manual rollback.........[n]
Automatic rollback......[n]
Self test...............[n]
Upgrade timeout.........[300 sec]
Self test timeout.......[0 sec]
Rollback timeout........[300 sec]
Inaccessibility timeout.[300 sec]

GENERAL PROPERTIES
-------------------------------
Payload cold reset req....[n]
Def. activation supported.[n]
Comparison supported......[n]
Preparation supported.....[y]
Rollback supported........[y]

OK
Error initiating upgrade action, compcode = ff

Firmware upgrade procedure failed

Error aborting upgrade, compcode = 80

...

-----------------------------------------------------------------
|ID | Name      | Active  |   Upload Progress   | Upload| Image |
|   |           | Version |0%      50%      100%| Time  | Size  |
|---|-----------|---------||----+----+----+----||-------|-------|
...Failed...
-----------------------------------------------------------------

In OneFS 8.0, this would also cause output similar to the following for upgrade commands:

Isilon-1# isi upgrade nodes firmware progress list
Lnns  Device               Old Version             New Version             Status
-----------------------------------------------------------------------------------
...
6     CMCSDR_Yeti          00.06                   00.06                   -
6     CMC_Yeti             00.0b                   02.05                   error
6     IsilonFPV1           UI.01.36                UI.01.36                -
...
-----------------------------------------------------------------------------------
Total: 75
Isilon-1# isi upgrade nodes view 6
             Node LNN: 6
   Node Upgrade State: error
        Error Details: Failed Step: InstallFirmware
                       Error Log: /ifs/.ifsvar/upgrade/Agent-6-000e1e591a20232c8a58011809d7e56ff3fb.log
  Last Upgrade Action: -
   Last Action Result: -
Node Upgrade Progress: None
         Node OS Version: 8.0.0.4_build(53)style(5)

Isilon-1# isi_upgrade_logs
Copied log to /ifs/upgrade_error_logs/6_isi_Upgrade_Agent.py-InstallFirmware.log

Isilon-1# cat /ifs/upgrade_error_logs/6_isi_Upgrade_Agent.py-InstallFirmware.log
Starting /usr/sbin/isi_Upgrade_Agent.py InstallFirmware

Verifying boot device mirrors are synced...
2017-05-11 15:18:42,876 DEBUG Execute Command: InstallFirmware
2017-05-11 15:18:59,005 ERROR Failed to update CMC_Yeti firmware on Node 6

Questo problema è causato dal componente CMC che non risponde in modo tempestivo durante l'aggiornamento del firmware del nodo.

Il problema può essere risolto eseguendo un ripristino a freddo dei componenti BMC e CMC sul nodo oppure, se il ripristino a freddo non risolve il problema, è necessario spegnere e scaricare l'elettricità statica del nodo in conformità a https://support.emc.com/kb/466373 .  Se nessuna delle due procedure risolve il problema, eseguire l'escalation ai team del supporto tecnico Isilon per un'ulteriore analisi.

Per reimpostare il CMC, eseguire il seguente comando sul nodo interessato:
/usr/bin/isi_hwtools/isi_ipmicmc -c -a cmc

Output di esempio:

Isilon-6# /usr/bin/isi_hwtools/isi_ipmicmc -c -a cmc
Cold reset CMC
Note - 15 sec wait for CMC...
Checking if Firmware Version MSB is set
MSB set in Firmware Version
Waiting 5 seconds before trying again
Success: MSB cleared in Firmware Version
Checking self test results
Success: self test passed

Once reset, retry the firmware update. 
In OneFS versions prior to 8.0:
isi firmware update

In OneFS versions 8.0 and later:
Review the 'Node Upgrade State' for each node, identify the node that is in an error state, this is the node that the upgrade will be retried on:
isi upgrade nodes list

To retry the upgrade on the node in an error state, run the following command, replace LNN with the node that is showing in an error state:
isi upgrade retry-last-action LNN