PowerStore: How to Prepare for a PowerStore Non-Disruptive Upgrade

How to Use This Article.

Prior to performing a Non-Disruptive Upgrade (NDU) with PowerStore, carefully review the following subsections in this article for NDU planning considerations and precautions. For further reference, consult the Software Upgrade Guide.

Known Issues and Limitations

PowerStoreOS Specific Issues

• Number of NVMe/TCP ports (when upgrading to PowerStoreOS 3.2 or later)

  • When storage networks are scaled to a substantial number of NVMe/TCP ports (that is greater than 50), with 'Auto Discovery of CDC' option enabled, the NDU process may lead to deficiency of resources in the PowerStore cluster. This scenario can result in CDC connection state on all NVMe/TCP ports that appear 'Uninitialized'. In addition, subsequent upgrade requests may fail.

  • For further details on this issue and how to avoid it before upgrading to PowerStoreOS version 3.2, see article PowerStore: When storage networks scaled to 50 (or more), PowerStoreOS upgrades may fail with "Uninitialized" ports

• Pause all Metro sessions if upgrading from 3.6.1.0 only.
  • For additional information, see KB-000223948  PowerStore: Overlapping I/O on a Metro volume may lead to unexpected reboot or increased latency

PowerStore T Models

Models: 500T, 1000T, 1200T, 3000T, 3200T, 5000T, 5200T, 7000T, 9000T, 9200T
No limitations

PowerStore Q Models

Models: 3200Q
No limitations

PowerStore X Models

Models: 1000X, 3000X, 5000X, 7000X, 9000X

• PowerStoreOS versions 3.5 (or later) are not supported on PowerStore X.

• Verify that SSH is started on both ESXi nodes, and SSH is enabled at boot prior to starting any NDU procedure.
  See article PowerStore: PowerStoreX-SSH disabled may lead to node failure for additional information.

• If VMware NSX-T Data Center is deployed on the PowerStore X model cluster, do not proceed with NDU before reviewing article PowerStore: PowerStoreX: If NSX-T-managed Virtual Distributed Switches are migrated to the vSphere Distributed Switches during vSphere upgrade, the connection will be lost after PowerStore management network reconfiguration for additional information.

• After upgrading the PowerStore X array to PowerStoreOS version 2.1.1.0, go to vCenter Server and check the PowerStore X esxi_node's licensing. Ensure that the status of the VMware vSphere 7 license is OK.

• All external ESXi hosts on the vSphere cluster should be running the same ESXi version as the internal hosts.  

Recommended PowerStoreOS Versions

• See article PowerStore: PowerStoreOS Matrix

• For PowerStore appliances using NVMe Expansion Enclosure/s (ENS24), PowerStoreOS 3.6.0.0 (or later) is recommended.

• For PowerStore appliances using Veeam as a backup solution in their environment, PowerStoreOS 3.6.0.0 (or later) is recommended.

• NDU from PowerStoreOS versions 3.X to 4.X is restricted if iSCSI transport is used as Remote System Transport Protocol (RSTP). Affected customers must change their RSTP to TCP transport before the NDU to 4.X.

Preliminary Checks

About this step

PowerStore includes various tools to help diagnose and proactively identify various issues which should be remediated before NDU.

1. Pre-Upgrade Health Check (PUHC) is an integrated utility that is shipped with every upgrade package, focused on tests aimed at NDU readiness.

2. System Health Check (SHC) is an integrated utility that is shipped with upgrade packages starting from PowerStoreOS 2.0, it includes additional health tests for the appliance.

3. Upgrade Extensions are an enhancement of System Health Check (SHC) which adds NDU-specific tests. This was introduced in PowerStoreOS 2.1 and is only available for upgrades from this version.

Consult the following table to determine, based on your current PowerStoreOS version, the tools to use before the NDU:

See the following three sub-sections for details on the preliminary checks tools.

Pre-Upgrade Health Check

• Consult article PowerStore: How to Check the Health of the Cluster Before Software Upgrade Using Pre-Upgrade Health Check and System Check for instructions on using the Pre-Upgrade Health Check before NDU.

• Review any items reporting an ERROR or WARNING status message that is raised during the Pre-Upgrade Health Check and address each accordingly.

• If you require assistance addressing any ERROR or WARNING messages that are displayed, contact Dell Technologies Support Services for assistance.

CAUTION: A green check mark does not indicate that the Pre-Upgrade Health Check was completed without any error or warning messages. A green check mark only indicates that a health check step was completed successfully. For more information, see article PowerStore: Health Check shows all green when there are warnings.

System Health Check

NOTE: Consider running the System Check at least a week before the NDU to allow enough time for corrective procedures should they be necessary.
 

• System Check, as a tool to be used prior to NDU, is available on appliances running PowerStoreOS versions 2.0.x.

• Consult PowerStore: How to Check the Health of the Cluster Before Software Upgrade Using Pre-Upgrade Health Check and System Check instructions on using the System Health Check before NDU.

• Review any items reporting a FAILED status message that is raised during the System Health Check and address each accordingly.

• If you require assistance addressing any FAILED status messages that are displayed, contact Dell Technologies Support Services for assistance.

Upgrade Extensions

• Upgrade Extensions are available on appliances running PowerStoreOS version 2.1.x.

• Consult article PowerStore: How to Check the Health of the Cluster Before Software Upgrade Using Pre-Upgrade Health Check and System Check for instructions on using Upgrade Extensions before NDU.

• Review any items reporting a FAILED status message that is raised during the Upgrade Extension and address each accordingly.

• If you require assistance addressing any displayed FAILED status messages, contact Dell Technologies Support Services for assistance.

Outstanding Alerts

• Log in to PowerStore Manager of the appliance being upgraded and review any outstanding alerts present by clicking the Monitoring tab.

• Determine if any alerts are outstanding that require attention. Review alerts with a Critical, Major, or Minor severity, and if possible, attempt to rectify any raised alerts reported.

• If needed, contact Dell Technologies Support Services for assistance with evaluating and addressing any outstanding alerts before proceeding with the upgrade.  

Host Readiness Best Practices, Known Issues, and Limitations

Operating system specific

• vSphere ESXi

  • When upgrading to PowerStoreOS version 2.0 (or later), ESXi host running 8.0 U1 using NVMe/FC and NVMe/TCP datastore may experience I/O errors during PowerStore node reboots/failures. For further details on this issue and how to resolve it, see article PowerStore: I/O error observed on NVMe datastore during NDU

• Solaris

• For customer's on PowerStore code older than 3.5, the NDU is supported with Solaris native MPxIO starting at Solaris 11.4 SRU 35. Contact your service provider for PowerStore NDU support on earlier Solaris versions or Solaris updates.

• For customer's already on PowerStore code 3.5 and newer, the NDU is supported with Solaris native MPxIO 10/11.x

General

• See the Dell Technologies PowerPath Simple Support Matrix for supported multipathing options and their associated versions.

• For host connectivity best practices, see the corresponding E-Lab Host Connectivity Guide document per the OS of the connected host.

  • Ensure all hosts accessing the PowerStore appliance are properly configured with redundant paths and properly configured multipathing software.

• See E-Lab Interoperability Navigator for supported configurations (HBA firmware and driver, topologies, known limitations, and general guidelines):

  • For SCSI environments:

    • See E-Lab Simple Support Matrix.

  • For NVMe-oF environments:

    • See E-Lab NVMe/FC Host/Storage Interoperability Simple Support Matrix for supported configurations.

    • See E-Lab NVMe/TCP Host/Storage Interoperability Simple Support Matrix for supported configurations.

• Run the PowerStore Host Validation Script (HVS) to scan and corroborate the host configuration. For instructions on how to download and install HVS for a connected host operating system go to:

  • VMware ESXi - see article PowerStore: Host Validation Script for ESXi.  

General Upgrade Planning Considerations

• General Guidelines

  • Download the latest version of the PowerStore Release Notes to determine which software update packages are required to perform the PowerStore upgrade.

• While this change to the package file name does not affect NDU to the PowerStore appliance, it may affect subsequent procedures done on the appliance using this package such as node repair or Internal M.2 boot module replacement (primary M.2). For further details on this issue and how to avoid it, see article PowerStore: PowerStoreOS re-installation (node repair) failed when the file name of the package used was changed after download from the Dell Support Site.

  • You can upload up to ten (10) upgrade packages on your PowerStore cluster at one time. However, after an upgrade starts, it must be successfully completed across all appliances in the cluster before another upgrade can be performed.

  • When downloading a PowerStoreOS package from the Dell Support site, ensure that its file name remained intact after the download to the local machine, and that no suffix was automatically appended to it. In this case, an NDU using such a package may fail.

  • During the software update process, you may be disconnected from PowerStore Manager temporarily, retry logging into PowerStore Manager after several minutes (up to 5 minutes).

  • Some management operations may be blocked during the upgrade.

  • After the software update process completes successfully, close the browser tab of the user interface, and re-login from a new browser tab for new feature content to be exposed in the user interface.

• Maintenance Window

  • While PowerStore clusters are engineered and tested for fully nondisruptive upgrades, it is recommended to follow IT management best practices when upgrading a PowerStore cluster.

  • To the extent possible, leverage maintenance windows rather than production hours, and perform upgrades when the load on the cluster is lightest.

  • If CPU utilization exceeds 50% on the Nodes of a PowerStore appliance, reduce the I/O load within a maintenance window.

    • CPU utilization can be viewed on the Performance tab under the Hardware menu within PowerStore Manager (UI).

  • When the number of file systems is greater than 50, there is a possibility of a longer I/O pause. Consider a maintenance window.

  • This helps ensure that the upgrades are completed in the shortest time with the lowest impact.

• Obtaining Assistance

  • Contact Dell Technologies Support Services for assistance with coordinating and performing a nondisruptive upgrade (NDU) of your PowerStore appliance if needed.

  • If an upgrade is being planned for a PowerStore X environment, it is recommended to engage Dell Technologies Support for assistance guiding your upgrade to completion.

  • Dell Technologies works closely with you to ensure that the upgrade is scheduled and performed without disruption to your business workload.

• Support Connectivity

  • It is highly recommended to enable Support Connectivity and Remote Service Credentials (RSC) to simplify and reduce the length of an upgrade failure analysis, in case the software update process fails.

• SSH Access

  • Enable external SSH access on the PowerStore Appliance level to simplify and reduce the length of an upgrade failure analysis, in case the software update process fails.

• Service Account

  • Add a password to the service account. If a password is already added to the service account, verify that you can SSH to the PowerStore and log in with the user "service" and the password that is set for that account.

• CHAP settings

  • If you change CHAP settings on a PowerStore T model cluster with NAS enabled, such as enabling or disabling CHAP, changing mutual CHAP to single CHAP, or changing single CHAP to mutual CHAP, you must perform the following actions:

    1. Reboot the cluster nodes one at a time as soon as possible after changing the CHAP settings.

    2. Wait until all the cluster nodes have been rebooted before performing a software update.

• Suppressing dial-home alerts

  • Usually the NDU process will automatically disable dial-home alerts so that a Service Request is not created for alerts that occur during the NDU. However, this may not always activate as intended

  • For clusters with one or more expansion enclosures, the dial-home suppression may clear prematurely before the expansion enclosure upgrade is completed
    • See KB 000125772 for details on how to manually suppress dial-home alerts:  PowerStore: How to disable support notifications for testing and planned maintenance

Caution: The software update fails if the nodes on a PowerStore T model cluster with NAS enabled are not rebooted after changes are made to CHAP settings. For instructions on rebooting PowerStore T model nodes, see the PowerStore power off and Reboot Procedures Guide.

General Upgrade Considerations

• Some internal system operations, such as snapshot and replication schedules, may be paused during an NDU and resume when the NDU is complete.

Concluding Checks

PowerStore includes various tools to help diagnose and proactively identify various issues. These tools should be performed also after successfully completing the NDU to confirm that no issues are present of the cluster.

In this step the PowerStore health-check tools that were covered in the Preliminary Checks section should be performed after the NDU: Pre-Upgrade Health Check (PUHC) and System Health Check (SHC).

Consult the following table to determine, based on your PowerStoreOS version, the tools to use after the NDU:

In addition, after the NDU, ensure to upload and install the Health Check thin package and RxDefinitions package. As the NDU process removes these packages, if they previously existed.

For further details on installing the RxDefinitions package, see article PowerStore: Landing Page for RxDefinitions Issues

For instructions on using these PowerStore tools, consult the articles PowerStore: How to use the System Check feature and PowerStore: How to Check the Health of the Cluster Before Software Upgrade Using Pre-Upgrade Health Check and System Check.