PowerScale: On-Cluster Analysis - Upgrade Status

Isilon On-Cluster Analysis tool is reporting output similar to the following:

Upgrade Status                                    FAIL
  FAIL: Cluster Upgrade State is currently upgrade
  FAIL: Current Upgrade Activity is currently Firmware
  FAIL: The following nodes are reporting a fs_fmt_version of 0: 1,2,3
  FAIL: An odd fs_fmt_version was found, this is indicative that the Upgrade job has not completed and needs to complete prior to an upgrade to OneFS.
  WARN: Service isi_upgrade_d is currently Enabled.
  INFO: Refer to KB210510 (https://support.emc.com/kb/210510) for details.

PowerScale Isilon On-Cluster Analysis health check script identified issues which require resolving.

WARN: Service isi_upgrade_d is currently Enabled.

• If the cluster is running OneFS 8 or later, the isi_upgrade_d service should be disabled as it is no longer used for performing upgrades.
• If the cluster is running OneFS 7 or earlier, the isi_upgrade_d service should only be enabled while performing an OneFS upgrade, and it is enabled automatically.
• If the isi_upgrade_d service is enabled but there is no upgrade in progress, or an upgrade has encountered an issue and is no longer in progress, the isi_upgrade_d service should be disabled.

Verify the current status of the isi_upgrade_d service by running the following command:

isi services isi_upgrade_d

Disable the isi_upgrade_d service by running the following command:

isi services isi_upgrade_d disable

FAIL: Cluster Upgrade State is currently upgrade

FAIL: Current Upgrade Activity is currently Firmware

View the current upgrade activity by running the following command:

isi upgrade view

View the upgrade status across all nodes by running the following command:

isi upgrade nodes list

Any in progress upgrade activities prevents any further OneFS updates, patch install, or firmware updates until completed or cleared. If it is expected that an upgrade is in progress, then this failure may be ignored. If it is not expected, Dell Support should be engaged to investigate the in-progress upgrade activity and resolve any errors that arose during the previous updates.

FAIL: The following nodes are reporting a fs_fmt_version of 0: 1,2,3

A bug in some OneFS codes causes the fs_fmt_version to report 0. This may be ignored if no OneFS upgrade is being performed as it only impacts OneFS upgrades as they verify the fs_fmt_version in the upgrade compatibility checks. Verify the issue using commands from the Notes section and engage Dell Support to further investigate and assist in resolving.

FAIL: An odd fs_fmt_version was found, this is indicative that the Upgrade job has not completed and needs to complete prior to an upgrade to OneFS.

Verify the current fs_fmt_version by running the following command:

isi_for_array sysctl efs.gmp.fs_fmt_version

Verify if there is any running Upgrade jobs by running the following command:

isi job view upgrade

If the fs_fmt_version across all nodes is even, no further action is required.

If there is an odd fs_fmt_version and an Upgrade job is running, this is to be expected and no action is required, continue to monitor the Upgrade job to completion. If the Upgrade job continues to run for over 30 days, engage Dell Support.

If there is an odd fs_fmt_version and no Upgrade job is running, and there have not been any recent OneFS upgrades, engage Dell Support.

FAIL:The following nodes are reporting a fs_fmt_version of 0: 1,2,3

isi_for_array sysctl efs.gmp.fs_fmt_version

If the fs_fmt_version reports zero on any nodes it may be due to a bug in some versions of OneFS that causes the fs_fmt_version to report 0. This may be ignored if no OneFS upgrade is being performed as it only impacts OneFS upgrades as they verify the fs_fmt_version in the upgrade compatibility checks.

The following commands may be run on a cluster:

• To check the current status of the isi_upgrade_d service: isi services isi_upgrade_d
• To check for running isi_upgrade_d processes: isi_for_array -s ps auxxw | grep isi_upgrade_d
• To check the Cluster Upgrade State or Current Upgrade Activity: isi upgrade cluster view
• To check the fs_fmt_version: isi_for_array sysctl efs.gmp.fs_fmt_version