Isilon: Node firmware upgrade fails on CMC
Summary: Node firmware upgrades to CMC components may encounter a failure/error and need to be cold reset, or the node power drained, to resolve this issue.
This article applies to
This article does not apply to
This article is not tied to any specific product.
Not all product versions are identified in this article.
Symptoms
During a node firmware upgrade, the CMC firmware update fails/errors.
In the /var/log/isi_firmware.log on the node that failed, output similar to the following will be seen:
2017-05-11 15:18:58,991 ERROR [17093] : Node 6: CMCFirmware._update: update failed: error=65280 output: TARGET UPGRADE CAPABILITIES ------------------------------- HPM.1 version............0 Component 0 presence....[y] Component 1 presence....[y] Component 2 presence....[y] Component 3 presence....[y] Component 4 presence....[n] Component 5 presence....[n] Component 6 presence....[n] Component 7 presence....[y] Upgrade undesirable.....[n] Aut rollback override...[n] IPMC degraded...........[y] Defered activation......[n] Service affected........[n] Manual rollback.........[n] Automatic rollback......[n] Self test...............[n] Upgrade timeout.........[300 sec] Self test timeout.......[0 sec] Rollback timeout........[300 sec] Inaccessibility timeout.[300 sec] GENERAL PROPERTIES ------------------------------- Payload cold reset req....[n] Def. activation supported.[n] Comparison supported......[n] Preparation supported.....[y] Rollback supported........[y] Description string: CMC FW Current Version: Major: 00 Minor: 0b Aux : 00 00 00 00 Validating firmware image integrity...OK Performing preparation stage...TARGET UPGRADE CAPABILITIES ------------------------------- HPM.1 version............0 Component 0 presence....[y] Component 1 presence....[y] Component 2 presence....[y] Component 3 presence....[y] Component 4 presence....[n] Component 5 presence....[n] Component 6 presence....[n] Component 7 presence....[y] Upgrade undesirable.....[n] Aut rollback override...[n] IPMC degraded...........[y] Defered activation......[n] Service affected........[n] Manual rollback.........[n] Automatic rollback......[n] Self test...............[n] Upgrade timeout.........[300 sec] Self test timeout.......[0 sec] Rollback timeout........[300 sec] Inaccessibility timeout.[300 sec] GENERAL PROPERTIES ------------------------------- Payload cold reset req....[n] Def. activation supported.[n] Comparison supported......[n] Preparation supported.....[y] Rollback supported........[y] OK Error initiating upgrade action, compcode = ff Firmware upgrade procedure failed Error aborting upgrade, compcode = 80 ... ----------------------------------------------------------------- |ID | Name | Active | Upload Progress | Upload| Image | | | | Version |0% 50% 100%| Time | Size | |---|-----------|---------||----+----+----+----||-------|-------| ...Failed... -----------------------------------------------------------------
In OneFS 8.0, this would also cause output similar to the following for upgrade commands:
Isilon-1# isi upgrade nodes firmware progress list Lnns Device Old Version New Version Status ----------------------------------------------------------------------------------- ... 6 CMCSDR_Yeti 00.06 00.06 - 6 CMC_Yeti 00.0b 02.05 error 6 IsilonFPV1 UI.01.36 UI.01.36 - ... ----------------------------------------------------------------------------------- Total: 75 Isilon-1# isi upgrade nodes view 6 Node LNN: 6 Node Upgrade State: error Error Details: Failed Step: InstallFirmware Error Log: /ifs/.ifsvar/upgrade/Agent-6-000e1e591a20232c8a58011809d7e56ff3fb.log Last Upgrade Action: - Last Action Result: - Node Upgrade Progress: None Node OS Version: 8.0.0.4_build(53)style(5) Isilon-1# isi_upgrade_logs Copied log to /ifs/upgrade_error_logs/6_isi_Upgrade_Agent.py-InstallFirmware.log Isilon-1# cat /ifs/upgrade_error_logs/6_isi_Upgrade_Agent.py-InstallFirmware.log Starting /usr/sbin/isi_Upgrade_Agent.py InstallFirmware Verifying boot device mirrors are synced... 2017-05-11 15:18:42,876 DEBUG Execute Command: InstallFirmware 2017-05-11 15:18:59,005 ERROR Failed to update CMC_Yeti firmware on Node 6
Cause
This issue is caused by the CMC component not responding in a timely manner during the node firmware update.
Resolution
The issue may be resolved by either performing a cold reset of the BMC and CMC components on the node, or, if the cold reset does not resolve the issue, a power down and power drain of the node in accordance with https://support.emc.com/kb/466373 must be performed. If neither steps resolve the issue, escalate to Isilon Technical Support teams for further analysis.
To reset the CMC, run the following command on the affected node:
/usr/bin/isi_hwtools/isi_ipmicmc -c -a cmc
Example output:
To reset the CMC, run the following command on the affected node:
/usr/bin/isi_hwtools/isi_ipmicmc -c -a cmc
Example output:
Isilon-6# /usr/bin/isi_hwtools/isi_ipmicmc -c -a cmc Cold reset CMC Note - 15 sec wait for CMC... Checking if Firmware Version MSB is set MSB set in Firmware Version Waiting 5 seconds before trying again Success: MSB cleared in Firmware Version Checking self test results Success: self test passed
Once reset, retry the firmware update.
In OneFS versions prior to 8.0:
isi firmware update
In OneFS versions 8.0 and later:
Review the 'Node Upgrade State' for each node, identify the node that is in an error state, this is the node that the upgrade will be retried on:
isi upgrade nodes list
To retry the upgrade on the node in an error state, run the following command, replace LNN with the node that is showing in an error state:
isi upgrade retry-last-action LNN
Affected Products
IsilonProducts
Isilon, Isilon HD400, Isilon NL410, Isilon S210, Isilon X210, Isilon X410Article Properties
Article Number: 000050382
Article Type: Solution
Last Modified: 11 Oct 2024
Version: 3
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.