Avamar: Data partition is full due to disk test files in QA directories

An Avamar grid is experiencing issues with Operating System (OS) file system capacity.

• Garbage collection (GC) or checkpoints (cp) fail with MSG_ERR_DISKFULL.
• Checkpoint validation (aka HFScheck) fails with ERROR: "avmaint: hfscheck: server_exception(MSG_ERR_STRIPECREATE)"
• Backups cannot run, or run and fail due to a full capacity
• Backups fail with either MSG_ERR_STRIPECREATE or report that the target server is full.
• The /data01/cur/gsan.log or /data01/cur/err.log contains the entry "createstripetask::allocstripe not enough space"
• The Avamar server access state is in admin mode.

The backup scheduler is disabled and cannot be resumed.

The Avamar data partitions contain QA directories which are full of dt.N files. These are left over files from an earlier run of the disk-throughput performance-testing script. These files may be present on all nodes.  

Examples:

admin@storagenode1:/data04/QA/>: ls

storagenode1_data04_dt.log  dt.14  dt.20  dt.27  dt.33  dt.4   dt.46  dt.52  dt.59  dt.65  dt.71  dt.78  dt.84  dt.90   
                     dt.0   dt.15  dt.21  dt.28  dt.34  dt.40  dt.47  dt.53  dt.6   dt.66  dt.72  dt.79  dt.85  dt.91
                     dt.1   dt.16  dt.22  dt.29  dt.35  dt.41  dt.48  dt.54  dt.60  dt.67  dt.73  dt.8   dt.86
                     dt.10  dt.17  dt.23  dt.3   dt.36  dt.42  dt.49  dt.55  dt.61  dt.68  dt.74  dt.80  dt.87
                     dt.11  dt.18  dt.24  dt.30  dt.37  dt.43  dt.5   dt.56  dt.62  dt.69  dt.75  dt.81  dt.88
                     dt.12  dt.19  dt.25  dt.31  dt.38  dt.44  dt.50  dt.57  dt.63  dt.7   dt.76  dt.82  dt.89
                     dt.13  dt.2   dt.26  dt.32  dt.39  dt.45  dt.51  dt.58  dt.64  dt.70  dt.77  dt.83  dt.9

admin@storagenode1:/data04/QA/>: du -h

94G

admin@storagenode1:/data04/QA/>: df -h

Filesystem         Size  Used Avail Use% Mounted on
/dev/sda2           16G  4.7G   11G  31% /
udev               2.9G  116K  2.9G   1% /dev
tmpfs              2.9G     0  2.9G   0% /dev/shm
/dev/sda1         1011M   62M  899M   7% /boot
/dev/sda6          7.6G  250M  6.9G   4% /var
/dev/sda8           62G   13G   46G  22% /space
/dev/sdb1          100G   95G  5.9G  95% /data01

admin@storagenode1:/data04/QA/>: du -cksh *

8.0K    storagenode1__data04_dt.log
1.1G    dt.0
1.1G    dt.1
1.1G    dt.10
1.1G    dt.11
1.1G    dt.12
...
1.1G    dt.9
1.1G    dt.90
1.1G    dt.91
1.1G    dt.92
95G     total

Remove the leftover QA test directories.

1. Log in to the Avamar Utility Node as admin.

2. Elevate to root privilege.

3. Load the root ssh keys. For more information about loading keys, see Avamar: How to Log in to an Avamar Server and Load Various Keys.

4. Run the following "mapall" command: 

mapall --parallel --bg --user=root --noerror --all+ '/usr/local/avamar/bin/dtsh --cleanup'

The dt command writes data to the hard drives on a node for benchmark performance-testing purposes.

The command is typically used for disk performance testing of new Avamar systems. It is useful for Avamar Virtual Edition (AVE) where the underlying disk storage performance may be variable.

When the dtsh script launches the dt program to test disk throughput.

The dt program creates a working directory called QA under each data partition. Files named dt.1, dt.2, and so on are created in the QA directories.