Data Domain: Filesystem Space Utilization Alerts

All Data Domain Restorers (DDRs) contain a pool or area of storage known as the 'active tier':

• This is the area of the disk where newly ingested files or data resides. On most DDRs, files remain here until expired or deleted by a client backup application.
• On DDRs configured with Extended Retention (ER) or Long Term Retention (LTR) the data movement process may periodically run to migrate old files from the active tier to archive or cloud tiers.
• The only way in which to reclaim space in the active tier which is used by deleted or migrated files is by running the garbage collection/clean process (GC).

Current utilization of the active tier can be displayed using the filesys show space or df commands:

# df
Active Tier:
Resource           Size GiB   Used GiB   Avail GiB   Use%   Cleanable GiB*
----------------   --------   --------   ---------   ----   --------------
/data: pre-comp           -    33098.9           -      -                -
/data: post-comp    65460.3      518.7     64941.6     1%              0.0
/ddvar                 29.5       19.7         8.3    70%                -
/ddvar/core            31.5        0.2        29.7     1%                -
----------------   --------   --------   ---------   ----   --------------

DD Space Usage alerts information:

• Data Domain system has an internal design where it automatically generates an alert when the active tier space utilization reaches 90%. This alert configuration can be changed. By default, the warning-space-usage value is set to 90% and the critical-space-usage value is set to 95%.
• Check the current values of the warning-space-usage and critical-space-usage parameters:

  • # filesys option show warning-space-usage

  • # filesys option show critical-space-usage 

  • Roles required: admin, limited-admin, user, backup-operator, security, none.
• Set the values of the warning-space-usage and critical-space-usage parameters:

  • # filesys option set warning-space-usage <space-usage-value>

    • The range of <space-usage-value> is 50 - 90.

  • # filesys option set critical-space-usage <space-usage-value>

    • The range of <space-usage-value> is 75 - 98.
  • Roles required: admin, limited-admin.

• The active tier may start to run out of available space causing alerts or messages such as the following to be displayed:
  • EVT-SPACE-00004: Space usage in Data Collection has exceeded 90% threshold: This is the initial alert that occurs when the DD space is 90% consumed. No processes are impacted.
  • EVT-SPACE-00004: Space usage in Data Collection has exceeded 95% threshold: This alert occurs when the DD space-utilization has reached 95%. The performance may get impacted like cleaning and other processes.
  • EVT-SPACE-00004: Space usage in Data Collection has exceeded 100% threshold: If the active tier becomes 100% full, no new data can be written to the DDR. This may cause backups and replication processes to fail. In this scenario, log messages such as the following may be displayed:

    • CRITICAL: MSG-CM-00002: /../vpart:/vol1/col1/cp1/cset: Container set [container set ID] out of space

• In some circumstances, the active tier becoming full may cause the Data Domain File System (DDFS) to become read-only. Existing files cannot be deleted.

Utilization of the active tier must be carefully managed, otherwise the above may occur.

Checklist for DD Capacity issues:

• Details of these steps can be found in KB Article: Data Domain: How to solve high space consumption or low available capacity on Data Domain Restorers (DDRs)
  • Step 1 - Determine whether an active tier clean must be run.
  • Step 2 - Check for large amounts of replication lag against source replication contexts
  • Step 3 - Check for mtrees which are no longer required
  • Step 4 - Check for old mtree snapshots
  • Step 5 - Check for old files on the system
  • Step 6 - Check for backups which include a large number of small files
  • Step 7 - Check for lower than expected de-duplication ratio
  • Step 8 - Check whether the system is a source for collection replication
  • Step 9 - Check whether data movement is being regularly run
  • Step 10 - Add additional storage to the active tier