Isilon: Gen6: Node Fails to boot with error message 'Unable to perform receipt update for Hardware Upgrade'
Summary: Gen6 node fails to boot during the PSI configuration phase, showing the 'isi_upgrade_hardware' script running followed by an error message stating 'Unable to perform receipt update for Hardware Upgrade'. ...
This article applies to
This article does not apply to
This article is not tied to any specific product.
Not all product versions are identified in this article.
Symptoms
Gen6 node fails to boot during the PSI configuration phase, showing the 'isi_upgrade_hardware' script running followed by an error message stating 'Unable to perform receipt update for Hardware Upgrade'. This can occur after a recent hardware upgrade on the cluster such as a DIMM, NIC or SSD upgrade. The end of the boot sequence will look similar to the following (important lines are bolded):
2021-05-05T11:55:15-07:00 isi_upgrade_hardware[470]: Update receipt for hardware upgrade
2021-05-05T11:55:15-07:00 isi_upgrade_hardware[470]: Updating receipt - cmd: /usr/bin/isi_hwtools/isi_json_update /mfg/psi/receipt.json FE 002-iFEIO-10GBE-B-psi
2021-05-05T11:55:15-07:00 isi_upgrade_hardware[470]: Updating receipt - cmd: /usr/bin/isi_hwtools/isi_json_update /mfg/psi/receipt.json MODEL_BOM,compute 001-infinity-compute-low-64GB-psi
2021-05-05T11:55:15-07:00 isi_upgrade_hardware[470]: Updating receipt - cmd: /usr/bin/isi_hwtools/isi_json_update /mfg/psi/receipt.json MODEL_BOM,ssd 001-infinity-ssd-1x400gb-psi
2021-05-05T11:55:15-07:00 isi_upgrade_hardware[470]: Next Step - Run isi_upgrade_hardware --node --validate
2021-05-05T11:55:15-07:00 isi_upgrade_hardware[470]: Automatic Replacement Recognition already enabled
2021-05-05T11:56:16-07:00 isi_upgrade_hardware[470]: Unable to parse HW Upgrade gconfig: (upgrade in hwupgrade.upgrades): No such file or directory
2021-05-05T11:56:16-07:00 python: [520] WARNING: IPMI error: rv=0, CC=0xc1. IPMB Address: 0x20 cmd: 0xbf, netfn: 0x30, cmd data: [01]
2021-05-05T11:56:16-07:00 python: [520] WARNING: CC[c1]: Invalid Command.
2021-05-05T11:56:16-07:00 python: bootstrap_get_minbay: minbay 1, major 0, minor 0, waited 0 sec 6458 usec
Unable to perform receipt update for Hardware Upgrade
To prevent a DL situation, contact Dell EMC Customer Support immediately:
2021-05-05T11:55:15-07:00 isi_upgrade_hardware[470]: Update receipt for hardware upgrade
2021-05-05T11:55:15-07:00 isi_upgrade_hardware[470]: Updating receipt - cmd: /usr/bin/isi_hwtools/isi_json_update /mfg/psi/receipt.json FE 002-iFEIO-10GBE-B-psi
2021-05-05T11:55:15-07:00 isi_upgrade_hardware[470]: Updating receipt - cmd: /usr/bin/isi_hwtools/isi_json_update /mfg/psi/receipt.json MODEL_BOM,compute 001-infinity-compute-low-64GB-psi
2021-05-05T11:55:15-07:00 isi_upgrade_hardware[470]: Updating receipt - cmd: /usr/bin/isi_hwtools/isi_json_update /mfg/psi/receipt.json MODEL_BOM,ssd 001-infinity-ssd-1x400gb-psi
2021-05-05T11:55:15-07:00 isi_upgrade_hardware[470]: Next Step - Run isi_upgrade_hardware --node --validate
2021-05-05T11:55:15-07:00 isi_upgrade_hardware[470]: Automatic Replacement Recognition already enabled
2021-05-05T11:56:16-07:00 isi_upgrade_hardware[470]: Unable to parse HW Upgrade gconfig: (upgrade in hwupgrade.upgrades): No such file or directory
2021-05-05T11:56:16-07:00 python: [520] WARNING: IPMI error: rv=0, CC=0xc1. IPMB Address: 0x20 cmd: 0xbf, netfn: 0x30, cmd data: [01]
2021-05-05T11:56:16-07:00 python: [520] WARNING: CC[c1]: Invalid Command.
2021-05-05T11:56:16-07:00 python: bootstrap_get_minbay: minbay 1, major 0, minor 0, waited 0 sec 6458 usec
Unable to perform receipt update for Hardware Upgrade
To prevent a DL situation, contact Dell EMC Customer Support immediately:
Cause
The recent hardware upgrade was run using an older version of the isi_hardware_upgrade script which contained a bug that caused the cleanup phase of the process to not be successfully completed. The /root/node_hardware_upgrade.json file was left behind on the node. This leads the node to believe that it is still in the middle of a hardware upgrade, and it is attempting to execute the next step in the process, which fails because there is not actually an upgrade still in progress. This will prevent the node from completing boot.
Resolution
1. First off, verify with the customer and any field personnel present that there is not in fact a hardware upgrade currently in progress. Then, from the recovery shell menu, press 1 to enter recovery shell, then press Enter to load the default shell.
2. Check to see if the following file is present on the node: /root/node_hardware_upgrade.json
3. If found, rename this file, then reboot the node:
# mv /root/node_hardware_upgrade.json /root/node_hardware_upgrade.json.bak
# reboot
4. If the node boots successfully, it is recommended to preemptively check the other nodes in the cluster for the presence of this file, and remove it if found:
# isi_for_array -s 'mv /root/node_hardware_upgrade.json /root/node_hardware_upgrade.json.bak'
2. Check to see if the following file is present on the node: /root/node_hardware_upgrade.json
3. If found, rename this file, then reboot the node:
# mv /root/node_hardware_upgrade.json /root/node_hardware_upgrade.json.bak
# reboot
4. If the node boots successfully, it is recommended to preemptively check the other nodes in the cluster for the presence of this file, and remove it if found:
# isi_for_array -s 'mv /root/node_hardware_upgrade.json /root/node_hardware_upgrade.json.bak'
Affected Products
Isilon, Isilon A200Article Properties
Article Number: 000191007
Article Type: Solution
Last Modified: 26 Aug 2022
Version: 2
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.