PowerScale Isilon: Failed to Install Patch, x bytes in /var Partition are Required.

Summary: This article includes a proactive workaround to avoid filling up the /var partition during Roll-Up Patch (RUP) installs.

This article applies to This article does not apply to This article is not tied to any specific product. Not all product versions are identified in this article.
Check out other resources

Symptoms

Patch install fails with output similar to the following: 
# isi upgrade patches install 9.2.1.14_GA-RUP_2022-07_PSP-2429.pkg --rolling
The below patches are deprecated by this patch and will be removed automatically:
- 9.2.1.7_GA-RUP_2021-12_PSP-1698
Would you like to proceed? (yes/[no]): yes
This patch request will require nodes in the cluster to be rebooted either in succession or simultaneously, depending on the options you supplied. Both scenarios can result in lost connectivity. Would you like to proceed? (yes/[no]): yes
Failed to install patch, 724933132 bytes in /var partition are required.

Cause

Security fixes introduced in patches OneFS 9.1.0, 9.2.1, and 9.3.0 have increased the size of patches from around 50MB to more than 150MB in size. Due to the increased size of the patches, a substantial amount of available space in the /var partition is required when installing them. The available space requirements may not be achievable in some instances. This article provides steps to create symbolic links to directories in the /var/crash partition as a workaround.

Resolution

Note: The steps in this workaround cannot be performed on Compliance mode-enabled clusters.
 
Note: It is only necessary to perform the following proactive workaround for the install of OneFS 9.1.0, 9.2.1, or 9.3.0 Roll-Up Patches due to their increased size. The workaround should not be deployed for other patch installs or on other OneFS releases.


Pre-requisites

Prior to performing the proactive workaround of creating symbolic links to directories in the /var/crash partition there are pre-requisites that must be met:
  • The upgrade and patch frameworks must be in an idle clean state
  1. Upgrade framework check; confirm that there are no active upgrade processes by reviewing the output of the following command:
isi upgrade view
  1. Patch framework checks:
    1. Confirm that all patches are in an Installed state in the output of the following command:
    isi upgrade patches list
    1. Confirm that the patch system is not active on any nodes by running the following command:
    isi_patch isactive
    1. Confirm that there are no files in the /var/patch/tmp/ and /var/patch/catalog/ directories:
    isi_for_array 'find /var/patch/catalog/ /var/patch/tmp/ -depth 1 -ls'
    Note: If the two previous patch framework checks [a and b] did not identify any issues and files are found in the /var/patch/tmp/ or /var/patch/catalog/ directories, it is safe to move those files to /ifs/data/Isilon_Support.
  • There must be sufficient available space in the /var (500MB) and /var/crash (1GB) partitions. Clean up of the partitions should be performed on any nodes reporting less than the required amount of available space.
  1. The following command has output on nodes with less than 500MB of available space in the /var partition:
isi_for_array 'df -m /var' | awk '{if ($5 < 500) { print }}'
  1. The following command has output on nodes with less than 1GB of available space in the /var/crash partition:
isi_for_array 'df -m /var/crash' | awk '{if ($5 < 1000) { print }}'


Proactive Workaround

Important Note: Do not forget to perform the remaining steps after patch install is completed!
 
  1. Prior to the patch install and after satisfying all the pre-requisites, run the following sequence of commands to make the directories that are being used under the /var/crash partition and create symbolic links to them:
isi_for_array 'mkdir -p /var/crash/patch/tmp /var/crash/patch/catalog'

isi_for_array 'ln -sF /var/crash/patch/tmp /var/patch/tmp'

isi_for_array 'ln -sF /var/crash/patch/catalog /var/patch/catalog'
  1. Perform the patch install as documented in the patch README.
  2. Upon completion of the patch install, run the following commands to remove the symbolic links:
isi_for_array 'test -L /var/patch/catalog && unlink /var/patch/catalog'

isi_for_array 'test -L /var/patch/tmp && unlink /var/patch/tmp'
  1. Re-create the directory and confirm permissions:
isi_for_array 'umask 037 && mkdir /var/patch/tmp /var/patch/catalog'

isi_for_array 'ls -ld  /var/patch/tmp /var/patch/catalog' 

cluster-1: drwxr-----  2 root  wheel  512 Feb 20 23:37 /var/patch/catalog 
cluster-1: drwxr-----  2 root  wheel  512 Feb 20 23:37 /var/patch/tmp 
cluster-2: drwxr-----  2 root  wheel  512 Feb 20 23:37 /var/patch/catalog 
cluster-2: drwxr-----  2 root  wheel  512 Feb 20 23:37 /var/patch/tmp

Note: If IOCA reports FAIL as seen below and these folders already exist, we can manually modify the permissions with the command: 
# isi_for_array -s 'chmod 740 /var/patch/tmp /var/patch/catalog'
FAIL: Checks have identified permission issues on /var/patch directories that are required for patch installations: 2 
INFO: Please follow step 4 on 'Proactive Workaround section' within KB 000201666 (https://www.dell.com/support/kbdoc/en-ie/000201666) to correct the folder permissions.

Additional Information

Dell article for cleaning up the /var/crash partition: Isilon The /var/crash partition of a node in the cluster has reached 90% capacity alert - Event ID: 100010002

Products

PowerScale OneFS
Article Properties
Article Number: 000201666
Article Type: Solution
Last Modified: 10 Jul 2024
Version:  11
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.