PowerProtect Data Domain: DDHA Upgrade Pre-Check

The High Availability (HA) feature lets you configure two protection systems as an Active-Standby pair, providing redundancy if there is a system failure. HA keeps the active and standby systems synchronized. This is so that if the active node failed due to hardware or software issues, the standby node could take over for that failing node.

HA is supported on the following systems:

• Data Domain DD6800
• PowerProtect DD6900
• Data Domain DD9300
• PowerProtect DD9400
• PowerProtect DD9410
• Data Domain DD9800
• PowerProtect DD9900
• PowerProtect DD9910

Preparing DDHA Systems for Upgrade

Before starting an upgrade, ensure that both nodes are functioning optimally. Upgrading while either node is compromised can lead to failure and service interruptions. Before moving forward with the upgrade, run the following checks:

System Information

Verify the system hostname, serial number, model, current DDOS version, and uptime.

iDRAC Configuration

Confirm whether iDRAC is configured, as it is a valuable tool for monitoring and troubleshooting issues during reboots.

Not that iDRAC is not available on Dell Legacy systems DD6800, DD9300, and DD9800.

File System Status

Ensure that the filesystem is enabled and running.

Alerts and Hardware Failures

Review any current system alerts and resolve any hardware failures.

Alerts are not always shared between the active and standby nodes. Be sure to check both nodes individually for any active alerts that may indicate underlying issues. Most alerts are visible on only one node, so dual-node verification is essential.

HA System Status

Ensure that the system is in a high availability state before initiating the upgrade.

System Capacity

Assess the overall system capacity.

To ensure a smooth DDOS upgrade process, it is recommended to maintain a minimum of 10% free space on the Data Domain system when upgrading to DDOS versions earlier than 7.x, and at least 5% free space for upgrades to DDOS 7.x and later. This available space is essential for handling temporary files and any changes to the index or metadata. This helps to minimize the risk of upgrade failures or performance degradation during the upgrade. 

Firmware Versions

Confirm that all system components are running the latest firmware.

If the target code is earlier than 7.7.5.1, 7.10.1.0, or 7.11.x, check the disk firmware version and refer to KB article 000204252 for guidance.

Network Configuration

Verify the system’s network configuration and ensure that all interfaces are operational with appropriate IP addresses.

On DDHA systems, network interfaces are configured as either floating or fixed. Each type has specific behavior and verification requirements:

• Floating Interfaces: For interfaces configured as floating, verify that each Network Interface Card (NIC) port, alias, or virtual Ethernet (veth) interface is in an enabled and running state on both the active and standby nodes. It is expected that floating interfaces display an IP address on the active node, while showing N/A on the standby node.
• Fixed Interfaces: For interfaces marked as fixed, ensure that each configured NIC port, alias, or veth is in an enabled and running state. Unlike floating interfaces, fixed interfaces are not required to have identical configurations across both nodes.
• HA Interconnect (veth99): Confirm that the HA interconnect (veth99) is present and that all required ports are enabled and running.
  • The number of required connections and their slot locations for veth99 vary by Data Domain model.

Disk Status

Assess the disk states and ensure that there are no Failed (F), Reconstructing (R), Copy (C), or Absent (A) Disks. 

In a DDHA configuration, the standby node does not process or serve data. As a result, the enclosure disks attached to the standby node do not display an "In Use Disks" state. Instead, all disks appear in a "Known Disks" state, indicating that the system recognizes them, but they are not active. This is expected behavior, as the active node handles all data operations.

Enclosure Status

Ensure that all enclosure devices are functioning properly.

Enclosure Topology

Validate the enclosure topology from both nodes. 

Notes:

• The outputs should be mirroring images.
• Look for an asterisk (*) next to enc.ctrl.port and review the Error Message field for issues like controller or cable faults.
• Symbols such as ?, !, or !! indicate errors.
• Confirm that all shelf numbers are correct and consistent.

Replication Status

Ensure that replication is not initializing. If it is, refer to KB article 000014079.

Starting with DDOS version 7.7.1, directory replication is no longer supported. If upgrading from a version earlier than 7.7.1 and directory replication is configured, consult the "Directory-to-MTree Replication Migration" section in the DDOS Administration Guide before upgrading.

Hardware Health Assessment

Initiate a comprehensive hardware health check on Data Domain systems. 

This command is not compatible with all DD platforms. For more information, refer to KB article 000326481.

Filesystem Cleaning

Verify whether the Filesystem cleaning is running. If active, either allow it to complete or manually stop it by running command # filesys clean stop on the active node.

Cloud Cleaning

Verify whether the cloud cleaning is running. If active, either allow it to complete or manually stop it by running command # cloud clean stop on the active node.

Data Movement

For systems with Cloud Tier, verify whether data movement is running. If active, manually stop it by running command # data-movement stop all on the active node.

Backup and Restore

If any backup or restore activity is in progress, pause or stop it before proceeding.

HA Failover Test

The purpose of the failover test is to validate high availability by initiating a failover from the active node to the standby node. Once the transition is complete, perform a second failover to return control to the original node, ensuring full two-way failover functionality.

Active node:

• Run the ha status command to confirm the system is in a highly available state with both nodes online.
• Run ha failover precheck (available in DDOS version 7.8 and later) to confirm readiness for failover.
• Run ha failover on the active node to switch from the active node to the standby node.

New Active node (previous Standby node):

• Run the ha status command to confirm the system is in a highly available state with both nodes online.
• Run ha failover precheck (available in DDOS version 7.8 and later) to ensure that the system is ready for failover.
• Run ha failover on the new active node to switch back to standby.

Upgrade RPM Package

Upload the upgrade RPM package to the active node. Once the system upgrade precheck command is run, the package is automatically copied to the standby node.

The upgrade RPM packages are available on the Data Domain Deduplication Storage Systems page.

• Transfer the RPM file from the client server to the active node using the following command:

client-server# scp <rpm file> sysadmin@<active-node-hostname-or-IP-address>:/ddr/var/releases/

• • The system prompts for a password, which the customer has defined. The target directory on the active node is /ddr/var/releases.
• Once the file transfer is complete, verify the system package information about the active node by running: 

active-node# system package list

System Upgrade Precheck

From the active node, run the system upgrade precheck <file> command. If the precheck completes successfully without any errors, the system is ready to proceed with the DDOS upgrade. For detailed guidance on upgrading a DDHA system, refer to KB article 000009653.