Isilon: Gen6: Node Fails to boot with error message 'Unable to perform receipt update for Hardware Upgrade'

Gen6 node fails to boot during the PSI configuration phase. After showing the 'isi_upgrade_hardware' script running, you receive an error message 'Unable to perform receipt update for Hardware Upgrade'. This can occur after a recent hardware upgrade on the cluster such as a DIMM, NIC, or SSD upgrade. The end of the boot sequence looks similar to the following (important lines are bolded):

2021-05-05T11:55:15-07:00 isi_upgrade_hardware[470]: Update receipt for hardware upgrade
2021-05-05T11:55:15-07:00 isi_upgrade_hardware[470]: Updating receipt - cmd: /usr/bin/isi_hwtools/isi_json_update /mfg/psi/receipt.json FE 002-iFEIO-10GBE-B-psi
2021-05-05T11:55:15-07:00 isi_upgrade_hardware[470]: Updating receipt - cmd: /usr/bin/isi_hwtools/isi_json_update /mfg/psi/receipt.json MODEL_BOM,compute 001-infinity-compute-low-64GB-psi
2021-05-05T11:55:15-07:00 isi_upgrade_hardware[470]: Updating receipt - cmd: /usr/bin/isi_hwtools/isi_json_update /mfg/psi/receipt.json MODEL_BOM,ssd 001-infinity-ssd-1x400gb-psi
2021-05-05T11:55:15-07:00 isi_upgrade_hardware[470]: Next Step - Run isi_upgrade_hardware --node --validate
2021-05-05T11:55:15-07:00 isi_upgrade_hardware[470]: Automatic Replacement Recognition already enabled
2021-05-05T11:56:16-07:00 isi_upgrade_hardware[470]: Unable to parse HW Upgrade gconfig: (upgrade in hwupgrade.upgrades): No such file or directory
2021-05-05T11:56:16-07:00 python: [520] WARNING: IPMI error: rv=0, CC=0xc1. IPMB Address: 0x20 cmd: 0xbf, netfn: 0x30, cmd data: [01]
2021-05-05T11:56:16-07:00 python: [520] WARNING: CC[c1]: Invalid Command.
2021-05-05T11:56:16-07:00 python: bootstrap_get_minbay: minbay 1, major 0, minor 0, waited 0 sec 6458 usec
Unable to perform receipt update for Hardware Upgrade
To prevent a DL situation, contact Dell EMC Customer Support immediately:

The recent hardware upgrade was run using an older version of the isi_hardware_upgrade script which contained a bug. The /root/node_hardware_upgrade.json file was left behind on the node. This leads the node to believe that it is still in the middle of a hardware upgrade. It is attempting to perform the next step in the process, which fails because there is no upgrade still in progress. This prevents the node from completing boot.

1. First off, verify with the customer and any field personnel present that there is not a hardware upgrade in progress. Then, from the recovery shell menu, press 1 to enter recovery shell, then press Enter to load the default shell.

2. Check to see if the following file is present on the node: /root/node_hardware_upgrade.json

3. If found, rename this file, then reboot the node:

# mv /root/node_hardware_upgrade.json /root/node_hardware_upgrade.json.bak
# reboot

4. If the node boots successfully, check the other nodes in the cluster for the presence of this file, and remove it if found:
# isi_for_array -s 'mv /root/node_hardware_upgrade.json /root/node_hardware_upgrade.json.bak'