IDPA: How to increase /space partition size in AVE VM before upgrading to 2.7 and above

When AVE’s /space partition size is less than 96 GB, IDPA 2.7 upgrade validation fails. This is reported in the RUCK, an upgrade pre-check tool to ensure that Appliance is ready for upgrade.

See Dell article 196100 PowerProtect DP Series Appliance and IDPA: Instructions on PowerProtect DP Rapid Upgrade Checker utility.

av_space_partition |  Use this KB to increase the size of the /space partition.

IDPA requires the Protection Storage /space partition to be 96 GB in capacity before 2.7 upgrade can be performed. AVE_Scale is a command-line tool that automates expansion of this partition.

The tool is provided as a .zip file on the Dell PowerProtect Data Protection Software page, or download directly with this link (Dell Support sign-in required).
 
The .zip contains the following files:

• ave_scale-19.4.0-7.exe, for Windows
• ave_scale-19.4.0-7, for Linux
• Checksums and README

This utility is supported for IDPA in Avamar Virtual Edition versions 18.2 and above. Ensure that requirements are met in Additional Info field below.

1. Before running this utility, create a checkpoint in AVE and validate it.
   1. Open a PuTTY or SSH to AVE using admin credentials.
   2. Suspend the maintenance scheduler with the command: 

      dpnctl stop maint

   3. Stop the backup scheduler service with the command:

      dpnctl stop sched

   4. Verify that the services are suspended or stopped with the command:

      dpnctl status

   5. Take a checkpoint with the command: 

      avmaint checkpoint --ava

   6. Monitor checkpoint status and make note of the checkpoint name with the command: 

      watch avmaint cpstatus

   7. Once status="completed" and result="OK", run checkpoint validation (HFS check) on manual checkpoint with command: 

      avmaint hfscheck --ava --rolling=true --full=false

       (verify it is validating the newly created checkpoint).
   8. Monitor the HFS check status with the command: 

      watch avmaint hfscheckstatus

   9. Output should be status="completed" and result="OK". This may take some time, depending on the utilization of the Appliance.
   10. Verify the newly created CP and HFScheck on AVE with the command: 

       cplist --full

2. Verify if MCS Flush was completed in the last 12 hours.

/usr/local/avamar/bin/mcserver.sh --status | grep -i "last flush completed"

3. If the Last Flush Completed time is older than 12 hours, or close to 12 hours, then perform MCS Flush operation.

/usr/local/avamar/bin/mcserver.sh --flush

4. Now back on ACM, check if the /home/admin/aveScale directory exists. If the directory exists, rename it to /home/admin/aveScale_old using the following command:

 mv /home/admin/aveScale /home/admin/aveScale_old

Usage:

ave_scale createspace <options>
Options:
Supported options for the createspace command:

Note: If there are any problems, contact Dell Support and share exit code that is provided.
 

Steps to increase /space partition size in AVE VM.

1. Download the ave_scale utility file.

Link to the latest version of the ave_scale utility. The .zip archive contains the Linux and Windows executables, checksums, and README.

2. Log in to ACM as root using an SSH connection.
   
   Ensure that file has executable permission. To get the executable permission, run the following command (19.4.x-x is used as example here, replace 'x.x' with version:

   chmod 755 ave_scale-19.4.x-x

3.  Expand the partition size using the following command:

./ave_scale-19.4.x-x createspace --vc-name <VCSA IP/FQDN> --vc-username administrator@vsphere.local --vc-password <VCSA password> --ave-hostname <AVE IP/FQDN> --ave-vm-name AVE --ave-password <AVE root password> --ave-admin-password <AVE admin user password>

Changing fields as appropriate.

4. Once the utility completes successfully, login to AVE using ssh with admin credentials.
5. Run the df -h command to check /space partition size.
6. Ensure that the total size of the /space partition is 96 GB.

   Filesystem      Size  Used Avail Use% Mounted on
   devtmpfs         18G  4.0K   18G   1% /dev
   tmpfs            18G     0   18G   0% /dev/shm
   tmpfs            18G  1.1G   17G   7% /run
   tmpfs            18G     0   18G   0% /sys/fs/cgroup
   /dev/sda2        16G  6.9G  8.0G  47% /
   /dev/sda1       978M   59M  868M   7% /boot
   /dev/sdc1       1.5T  6.2G  1.5T   1% /data02
   /dev/sdd1       1.5T  6.4G  1.5T   1% /data03
   /dev/sda8        96G   19G   78G  20% /space
   /dev/sdb1       1.5T   68G  1.5T   5% /data01
   /dev/sda6       7.8G  229M  7.2G   4% /var
   

7. Start the maintenance window scheduler and backup scheduler services:

• Start the maintenance scheduler with the command: 

  dpnctl start maint

• Start the backup scheduler service with the command:

  dpnctl start sched

• Verify that the services are started with the command:

  dpnctl status

For details on ave_scale utility, see the Expand AVE in a VMware Environment chapter in the Dell EMC Avamar Virtual Edition Installation and Upgrade Guide.

If the Appliance is not to be upgraded immediately, re-enable the policies that were disabled while performing the steps mentioned in the Requirements section.

• Privileges:
  • On a Linux host, must have root to run the ave_scale tool.
  • The vCenter Server username (--vc-username) option that is specified in the command line must have the privileges to perform the following tasks:
  • Reconfigure a virtual machine guest's hardware.
  • Power on and power off a virtual machine
  • Run commands by using the virtual machine guest agent.
  • Transfer files to and from an AVE instance.
• Network:
  • On Windows and Linux hosts, the ave_scale tool must be able to receive ICMP ping responses from the target AVE instance.
  • The ave_scale tool must be able to connect to the vCenter Server instance by using HTTPS.
• AVE node status:
  • Note the current state of all policies on the Avamar server. It is recommended to disable all policies before running the ave_scale tool.
  • Wait for any current jobs to complete.
  • If File-Level Restore (FLR) functionality to restore files, then some abnormal sessions are triggered. These abnormal sessions keep the Avamar server busy causing the ave_scale tool to fail. Find and terminate these abnormal sessions using the following steps:

1. Log in to the Protection Software as an Admin, using PuTTY
2. Run the following command to see the active sessions in the system:

   avmaint sessions | grep "path\|sessionid\|starttime"

Where:

• path: Displays the path for the client
• sessionid: Displays the unique identifier of the session
• starttime: Displays the UNIX timestamp of when the session began

3. Translate the value from the starttime parameter to a readable format by running the following command:

   t.pl <starttime>

4. Compare the value with the backup scheduler to confirm if the session is running. If the session started several days ago and is not configured as overtime, then it may be a session that stopped responding.
5. Run the following command to remove the unresponsive sessions:

   avmaint kill <sessionid>

6. After all the unresponsive sessions have been removed, run the following command to see the list of sessions running on the Protection Software server:

   avmaint sessions --full

• Check if any backup or replication jobs are running using the following steps:

1. Connect to the utility node using SSH and log in as admin.
2. Run the following command to verify if the server status is idle: opstatus.dpn
3. Run the following commands:

• avmaint sessions | grep path

  (To check if any backup jobs are in progress).

• ​​​​​mccli activity show --active | grep Replication

  (To check if any replication jobs are in progress).

• If any backup or replication jobs are running, can either wait for these jobs to complete or terminate these jobs.

It is recommended to wait for the running jobs to complete.

1. Run the following command to terminate the backup or replication jobs:

   mccli activity cancel --id=<job_id>

2. Run the following commands to confirm that the jobs are no longer in progress.

• avmaint sessions | grep path

  (To check if any backup jobs are in progress).

• mccli activity show --active | grep Replication

  (To check if any replication jobs are in progress).