PowerStore: How to Check the Health of the Cluster Before Software Upgrade Using Pre-Upgrade Health Check and System Check

Before you begin

Before performing PowerStoreOS upgrade, it is essential to validate the health of the cluster. These validations are more thorough than the continuous background checks performed by the alert mechanism of PowerStore. Two mechanisms, Pre-Upgrade Health Check (PUHC) and the System Checks, are available to manually validate the health of your cluster.

Each PowerStoreOS upgrade package includes an embedded PUHC for validations that are essential for a successful software upgrade. The PowerStoreOS package also includes a System Check for various validations that are important to be performed before maintenance operations. Starting in PowerStoreOS version 2.1.x.x, stand-alone Health Check packages are supported. The Health Check package is updated separately from the PowerStoreOS upgrade package and used to add additional checks after the PowerStoreOS is released.

• In PowerStoreOS version 3.x and 4.x, the Health Check package supplements the System Check validations and the PUHC that are in the PowerStoreOS.
• In PowerStoreOS version 2.1.x.x, the Health Check package supplements the System Check validations that are in the PowerStoreOS. Software upgrades are blocked until the validation for the System Checks' Upgrade Extensions profile in the Health Check thin package is performed.

This article describes the usage of PUHC, System Check, and Upgrade Extension of the System Check that are used to prepare for an NDU.

Available Health Check tools

Current PowerStore Version	Tools to use before NDU (source of the tool checks)
PowerStoreOS 4.x	Pre-Upgrade Health Check (operating system integrated and Health Check package)
PowerStoreOS 3.x	Pre-Upgrade Health Check (operating system integrated and Health Check package)
PowerStoreOS 2.1.x	Pre-Upgrade Health Check (operating system integrated) System Check Upgrade Extensions (operating system integrated and Health Check package)
PowerStoreOS 2.0.x	Pre-Upgrade Health Check (operating system integrated) System Checks (operating system integrated)
PowerStoreOS 1.x	Pre-Upgrade Health Check (operating system integrated)

 

Overview

1. Determine which Health Check Package to download.
2. Download and Install the Health Check package. (Skip this step if the current running version is 2.0 or less.)  If the cluster is enabled for automatic software updates there are notifications when new Health Check packages are ready and they are automatically uploaded, if the functionality is enabled.
3. Download the relevant PowerStoreOS. 
4. Perform preliminary heath checks at least a week in advance to evaluate the system health and correct issues found. Use all the tools available to evaluate system health - see table 1 (above). 
5. Immediately before the upgrade, perform the relevant health checks. 
6. Continue the upgrade. For NDU, see article PowerStore: How to prepare for a PowerStore Non-Disruptive Upgrade (NDU)

Determine which Health Check Package To Use

Current PowerStore Version	Health Check Package	Health Check Package Contents
PowerStoreOS 4.x	PowerStore-health_check-4.x<build> (when available)	System Check and PUHC
PowerStoreOS 3.x	PowerStore-health_check-3.x<build>	System Check and PUHC
PowerStoreOS 2.1.x	PowerStore-health_check-2.1.x.x-<build>	System Check and Upgrade Extension.
PowerStoreOS 1.x and 2.0.x	Not applicable

The Health Check Package is available to download from the Dell support Drivers & Downloads site. The description on the Drivers & Downloads site contains the details of compatible PowerStoreOS versions.

Check if the latest available Health Check package is already uploaded on your appliance. See instructions in the Additional Notes section below.

Download the System Health Check

Download the stand-alone Health Check package from the Dell support Drivers & Downloads site and then upload it to the PowerStore Manager. See previous section on how to select the package to use.
 

1. Download the latest Health Check package. If the cluster is enabled for automatic software updates to be uploaded, go to step 2.
   1. Sign in to your support account on Dell Support. If you are not logged in, the results are not accurate.
   2. Go to the Product Page for your system, then go to Drivers & Downloads. Using the Keyword search box, enter "Health Checks"
   3. Download the relevant package, if a package exists, to your local computer.
   4. From the PowerStore Manager Upload window, select the Settings icon, and then select Upgrades in the Cluster section. Upload the Health Check package using the UPLOAD PACKAGE button.
2. Install the Health Check package using the UPGRADE button in the PowerStore Manager Upload window.

Download the PowerStoreOS

Download the preferred PowerStoreOS version from the Dell support Drivers & Downloads site.

1. Go to the Product Page for your system, then go to Drivers & Downloads. Using the Download Type selector, select Operating Systems.
2. Download the relevant package to your local computer.
3. From the PowerStore Manager Upload window, select the Settings icon, and then select Upgrades in the Cluster section. Upload the PowerStoreOS package.

Pre-Upgrade Health Check (PUHC)

The Pre-Upgrade Health Check provides health checks that are specific to ensuring a successful software NDU.

• PUHC is taken from the target NDU PowerStoreOS. In PowerStoreOS 3.x and 4.x, the uploaded Health Check package contains additional supplementary PUHC checks which are used for upgrades from the installed (source) 3.x version.
• In the Upgrade window, select HEALTH CHECK. (See figure 1 below.)
• Review any Error or Warning messages that are raised during the Health Check and address each accordingly.
  • If the PUHC fails, select the failed PUHC entry under Recent activities to open the Job Details.
  • Select each row in the Job Details slide-out panel to review the failure or warning messages that were generated during the PUHC.
  • If you require assistance addressing any Error or Warning messages that are displayed, contact Dell Technologies Support Services for assistance.
• A green check mark indicates that a health check step was completed successfully. A green check mark does not indicate that the PUHC completed without any error or warning messages. For more information, see Dell article PowerStore: Health Check shows all green when there are warnings.

System Check before NDU (PowerStoreOS 2.x)

• System Check is available on appliances running PowerStoreOS versions 2.0.x and later. The System Check validations should be performed before maintenance operations and software upgrades when running PowerStoreOS 2.0. The System Check validation should be performed before maintenance operations when running PowerStoreOS 2.1.x.
• Use the RUN SYSTEM CHECK button in the SYSTEM CHECKS tab in the PowerStore Manager user interface (UI). For additional procedural details of how to perform a System Check, consult the Dell PowerStore Monitoring Your System document.
• Review any items reporting a Failed status message that is raised during the System Check and address each accordingly.
• If you require assistance addressing any Failed status messages that are displayed, contact Dell Technologies Support Services for assistance.

Upgrade Extensions (PowerStore 2.1.x)

• Upgrade Extension is available only on appliances running PowerStoreOS version 2.1.x.
• After uploading a software package, the UPGRADE button is not available until the user runs Upgrade Extensions. The tooltip message on the UPGRADE button reads: Upgrade is disabled if Upgrade Extensions have not been run within the last 60 minutes. To run Upgrade Extensions, go to Monitoring System Checks and click the Run Upgrade Extensions button.
  1. Use the Run Upgrade Extension button in the System Checks tab in the PowerStore Manager user interface (UI).
  2. You can see the last time the System Checks was run, the profile used, and the results.
  3. Review any items reporting a Failed status message that is raised during the health check and address each accordingly. Each status message includes the ID of an associated Dell article that provides information about resolving the issue.
  4. If you require assistance addressing any displayed Failed status messages, contact Dell Technologies Support Services for assistance.
   
  Note: As with any health check, any issues that are raised must be resolved before performing the software upgrade.
  
  
• Upgrade Extensions must be run within one hour of starting an upgrade. On the SYSTEM CHECKS tab you can see the last time the System Checks was run, the profile used, and the results:
  

How to determine the version of the installed System Health Check package

Viewing heath check package installation date

From the UI, it is not possible to determine the version of the Health Check package that is installed. The user can compare the release date in the Dell Support Drivers & Downloads site of the Health Check package with the date of the Health Check download activity in the Recent Activity pane.
   

Using PSTCLI to get health check package version information.

To show the installed versions of software and drive firmware on your system, use the following PSTCLI commands. This can be used to check if the latest Health Check package is already installed.

$ pstcli -d xx.xx.xx.xx -u <user> -p <password> software_package show -output nvp |egrep "name|software_package_type|release_version"
      name                  = Health_Check-2.1.1.1-1736451-retail
      software_package_type = Health_Check
      release_version       = 2.1.1.1

$ pstcli -d xx.xx.xx.xx -u <user> -p <password> software_package show -select software_package_state -output nvp
1:    software_package_state = Installed

 

Using REST API to get health check package version information.

An alternative to using the PSTCLI command to show the installed versions of software and drive firmware on your system is the REST API. Use the following REST API to check which Health Check package is installed.

Run the following REST API call to get the list of installed packages on your cluster:

curl -kv --user admin:xxxxx -X GET https://<appliance IP>:443/api/rest/software_package?select=* | python -mjson.tool

An example API call returns output:

----------------------------
.
.
.
Mar 11 2021 11:02:57 INFO: -----------
Mar 11 2021 11:02:57 INFO: ID : XXXXX-8aac-466d-9473-7d239bda31f5
Mar 11 2021 11:02:57 INFO: Release Version : 2.1.0.0-00001-retail (Build: 00001-
retail)
Mar 11 2021 11:02:57 INFO: Release Timestamp : 
2021-03-11T01:56:23+00:00(1615427783)
Mar 11 2021 11:02:57 INFO: Filename : Health_Check-2.1.0.0-00001-retail       <========
Mar 11 2021 11:02:57 INFO: State : Downloaded
Mar 11 2021 11:02:57 INFO: Size : 126 MB
Mar 11 2021 11:02:57 INFO: Type : Health_Check
Mar 11 2021 11:02:57 INFO: -----------
Mar 11 2021 11:02:57 INFO: -----------
.
.
.
Mar 11 2021 11:02:57 INFO: -----------
Mar 11 2021 11:02:57 INFO: ID : XXXXX-2b9a-4a77-ad8f-79ade4c32a03
Mar 11 2021 11:02:57 INFO: Release Version : 3.0.0.0-1323753-retail (Build: 
1323753)
Mar 11 2021 11:02:57 INFO: Release Timestamp : 
2021-03-04T21:45:24+00:00(1614894324)
Mar 11 2021 11:02:57 INFO: Filename : Software_Release-3.0.0.0-1323753-
retail
Mar 11 2021 11:02:57 INFO: State : Downloaded
Mar 11 2021 11:02:57 INFO: Size : 5894 MB
Mar 11 2021 11:02:57 INFO: Type : Software_Release
Mar 11 2021 11:02:57 INFO: -----------
Mar 11 2021 11:02:57 INFO: Exiting...
Mar 11 2021 11:02:57 INFO: Operation - Success
Mar 11 2021 11:02:57 INFO: Elapsed Time: 3.0 secs

View the installed Health Check package in the API output, and compare this to the latest Health Check package available on the Dell support site.

For PowerStoreX running 1.x upgrading to 2.x

The PUHC always displays the following failures when it is run before the pre-upgrade package is installed and the internal ESXi hosts are updated:

Pre-Upgrade Health Check Failed:
Pre-upgrade health check failed on node A - ["XXXXXXXXXXXXXX"] node B -
["XXXXXXXXXXXXXX"]. Please review the health check steps for more details.
(0xXXXXXXXXXXXX)
Health Check Node A
The component version is below the required version. Update FIRMWARE to version
XX.X.X. (incompatible_version)
This upgrade requires a minimum version of VMware ESXi be installed prior to the
upgrade. (incompatible_upgrade_esxi_version)
This upgrade requires the PowerStore X ESXi VIBs be upgraded prior to the upgrade.
(incompatible_upgrade_esxi_vib_version)
The PowerStore X ESXi VMFS partition is not the correct size to proceed with the
upgrade. Please perform the VMFS resize procedure. (check_vmfs_size_failed_A)

To resolve these errors, see the Dell PowerStore Software Upgrade Guide for the following Upgrading PowerStore X model clusters (PowerStoreOS 2.0.x and later)

• Install the Pre-Upgrade package on the cluster (PowerStoreOS 2.0.x and later)
• Update the PowerStore internal ESXi hosts (PowerStoreOS 2.0.x and later)
• Upgrade the PowerStoreOS on the cluster (PowerStoreOS 2.0.x or later)

---