Isilon OneFS 8.0: Cannot install firmware package: /var/patch/backup: is not a directory

During firmware package installation, installation would not complete, and output such as:

# isi upgrade patches list

Patch Name                                          Description                                           Status
----------------------------------------------------------------------------------------------------------------------------------
IsiFw_Package_v9.3.5 Package Name  : IsiFw Package v9.3.5 2017-04-04 To... Installing

Would show Status as Installing indefinitely.

Look at the last few lines from /var/log/isi_pkg of each node, and you should see the following error:

# isi_for_array -s 'tail -50 /var/log/isi_pkg'
.........
2017-05-28T13:31:56-05:00 <3.6> node-1 isi_pkg[16306]: Starting task for request 'INSTALL', task 'INSTALL_INIT', hash '2693c810637d37a02ee9cb57a9d01d3d'
2017-05-28T13:31:59-05:00 <3.6> node-1 isi_pkg[16306]: Running requirements file first for IsiFw_Package_v9.3.5..
2017-05-28T13:31:59-05:00 <3.3> node-1 isi_pkg[16306]: /var/patch/backup: is not a directory.
2017-05-28T13:31:59-05:00 <3.3> node-1 isi_pkg[16306]: install_init: Task encountered unknown failures

This issue could happen for any of the firmware packages such as IsiFw_Package_v9.3.5.tar or IsiFw_Package_v10.0.1.tar.

Partition mounted under /var requires at least 40% of free disk space to allow installation of a large patch such as firmware package. If you do not have enough free disk space under /var, the node would fail the installation with error:

/var/patch/backup: is not a directory

Reduce disk usage under /var. It is usually /var/log directory that would contain large files that might need to be truncated. To determine that:

1. Determine which node does not have at least 40% (Capacity should be 60% or less) free space under /var

   # isi_for_array -s 'df -h /var'

2. ssh to the node as root user that does not have enough free space under /var

   # cd /var/log
   # du ./* | sort -n -r | head -n 10

3. Make a backup directory within /ifs/data/Isilon_Support/ to store a backup of the files that will be truncated:

   # mkdir /ifs/data/Isilon_Support/Node<node number>_VarLog_Backup

4. Copy the files that you are truncating to the new backup directory:

   # cp  /ifs/data/Isilon_Support/Node_VarLog_Backup

5. The output of step 2 would give you top 10 largest files under /var/log. You can truncate the files using command:

   # truncate -s 0 <filename>

6. Check the partition again to ensure you have freed up at least 40% (Capacity should be 60% or less) of disk space under /var:

   # isi_for_array -s 'df -h /var'

7. Once you have freed up enough capacity, retry the failed upgrade using command:

   # isi upgrade retry-last-action --nodes=<lnn of failed node>

8. Allow 30 min to complete the upgrade process, and check to ensure that the package is showing Installed for Status:

   # isi upgrade patches list
   Patch Name                                               Description                                           Status
   ------------------------------------------------------------------------------------------------------------------
   IsiFw_Package_v10.0.1 Package Name  : IsiFw Package v10.0.1 2017-05-02 T... Installed

If you do not know if the files can be safely truncated, or if there are no large files under /var/log but /var partition is still too full, or the retry-last-action does not complete patch installation, engage Technical Support for assistance and see this KB as reference.