Data Domain - Space Running Low in /ddvar

The /ddvar directory (/ddvar/ and /ddr/var/ may be used interchangeably) is an administrative directory where the log files, generated support upload bundles, compressed core files and .rpm upgrade packages are placed. Occasionally this directory may become too full triggering the Data Domain system to send an alert:

# alerts show current
Id      Post Time                  Severity   Class               Object   Message
-----   ------------------------   --------   -----------------   ------   --------------------------------------------------------------------------
p0-36   Tue Sep 17 17:39:12 2024   ERROR      SystemMaintenance            Space running low in /ddvar for system maintenance activities.
-----   ------------------------   --------   -----------------   ------   --------------------------------------------------------------------------

The corresponding registry entry looks like this:

dynamic.ems.708.1.401.1243325160.1.5 = Space running low in /ddvar for system maintenance activities.

Some processes, including the FS (which handles all data access) rely on the /ddvar partition for writing logs, and if the partition becomes completely full and unable to be written to, the FS process may eventually fail to work properly and could die and not start up anymore as a result. Hence, it is very important customers pay attention to these alerts and act accordingly.

Too many log files stored in the operative system partition is causing ddvar to run out of space, at times, when core files are stored in /ddr/var/ (and not in separate partitions such as /ddr/var/core/ or /ddr/var/ext/) , this could also be the cause. Sometimes SUBs are very large (several GiB) and customer keeps creating more for troubleshooting , so the disk space is being consumed quickly. At times, there is a problem with large log files failing to be rotated, or leftovers from previous troubleshooting efforts, taking up unnecessary disk space under /ddr/var/.

PURPOSE

# alerts show current
Id      Post Time                  Severity   Class               Object   Message
-----   ------------------------   --------   -----------------   ------   --------------------------------------------------------------------------
p0-36   Tue Sep 17 17:39:12 2024   ERROR      SystemMaintenance            Space running low in /ddvar for system maintenance activities.
-----   ------------------------   --------   -----------------   ------   --------------------------------------------------------------------------

dynamic.ems.708.1.401.1243325160.1.5 = Space running low in /ddvar for system maintenance activities.

SYMPTOMS

• Alert "Space running low in /ddvar for system maintenance activities."
• Alert: "Space is running low in /ddvar. Please clear space or core dump capability will be disabled."
• Alert: "Core dump capability is now disabled due to lack of space in /ddvar"

APPLIES TO

• All Data Domain Systems
• All Software Releases

SOLUTION

==========  SERVER USAGE   ==========
Resource                   Size GiB        Used GiB           Avail GiB      Use%
------------------         --------        --------           ---------      ----
/backup: pre-comp                 -         17814.5                   -         -
/backup: post-comp           1892.9          1099.1               793.8       58%
/ddvar                         19.7            14.1                 4.6       75%

1. Confirm /ddvar is running low on space.
   At the Data Domain system prompt 180649 type:
   df
2. Note that space used in /ddvar may increase temporarily and cause transient space usage alerts as described in the Transient /ddvar space usage increases 181878 KB. If you still need to clean up files in /ddvar to free up additional space, continue with the next steps.
3. Determine the hostname and IP address of the Data Domain system.

   hostname
   The Hostname is: support1.emc.local
   
   	
   net lookup
   support1.emc.local has address 192.168.1.52
   	

# support coredump list

to delete specific core file use:

# support coredump delete [corefile.filename]

to delete all of them use:

# support coredump delete all

# support bundle list

to delete specific support bundle 

# support bundle delete <bundle-name-list> 

to delete all support bundles

# support bundle delete all

1. Ensure the Data Domain system's access list (ACL) is configured 181315 to allow access from the current system.
2. Delete old core files from the Data Domain system. Note: If the hostname does not work, try using the IP address instead.

   ==========  SERVER USAGE   ==========
   Resource                   Size GiB        Used GiB           Avail GiB      Use%
   ------------------         --------        --------           ---------      ----
   /backup: pre-comp                 -         17814.5                   -         -
   /backup: post-comp           1892.9          1099.1               793.8       58%
   /ddvar                         19.7             3.2                16.5       12%
   
   	

   1. On the Windows system, click on the start button and choose Run... and type the IP Address (or hostname) and path of the \ddvar share.
      
      Click OK
   2. Authenticate to the DataDomain Restorer with one of following accounts to have full permissions for /ddvar share:
      • Restorer default  sysadmin    account
      • Other Restorer local account which is part of Class  admin 
        Confirm with command # user show list
      • Active-Directory administrator account that shows Group ID as 50
        Confirm with command # cifs troubleshooting user 10500
   3. Delete core files if present. Open the \core folder.
      
      • If there are core files present, list them by date (detailed view). If there are core files dated within the last 24 hours this may indicate a more serious problem that will need to be analyzed further. If this is the case, please contact your contracted support provider, Upload a Support Bundle (SUB) Link Error 180563 and the most recent core file after creating a Create a service request
      • If the core files are not current, highlight and delete all of the files.
   4. Delete old Support Upload Bundle (SUB) files. Navigate to the \ddvar\support folder.
      
      • Highlight and delete all of the support-bundle files (support-bundle.tar.gz*)
   5. Delete old upgrade files (.rpm files) from the Data Domain system.
      Navigate to the \ddvar\releases folder.Highlight and delete all of the files.
      Note: .rpm files are used temporarily (only during upgrades). After the system has been upgraded, these files are no longer necessary.
   6. Verify space has been recovered.
      At the Data Domain system prompt type:
      df
   Freeing disk space in the /ddvar directory using a Unix based system.

   ==========  SERVER USAGE   ==========
   Resource                   Size GiB        Used GiB           Avail GiB      Use%
   ------------------         --------        --------           ---------      ----
   /backup: pre-comp                 -         17814.5                   -         -
   /backup: post-comp           1892.9          1099.1               793.8       58%
   /ddvar                         19.7             3.2                16.5       12%
   
   

   1. Ensure the Data Domain system's NFS access list (ACL) is configured 181315 to allow access from the current system.
   2. Mount the /ddvar export on the client system. 181310
   3. Delete old core files (if present) from the Data Domain system.
      At the the NFS client system's command prompt type:
      ls -l /ddvar/core
      • If there are core files dated within the last 24 hours this may indicate a more serious problem that will need to be analyzed further. If this is the case, please contact your contracted support provider, Upload a Support Bundle (SUB) Link Error 180563 and the most recent core file after creating a Create a service request
      • If the core files are not current, delete all of the files.
        rm /ddvar/core/*
   4. Delete old Support Upload Bundle (SUB) files. At the the NFS client system's command prompt type:
      rm /ddvar/support/support-bundle.*
   5. Delete old upgrade files (.rpm files)
      At the the NFS client system's command prompt type:
      rm /ddvar/releases/*.rpm
   6. Verify space has been recovered.
      At the Data Domain system prompt type:
      df
   Note: If /ddvar is still reporting full after executing the steps in this article, please contact your contracted support provider, Gather an Autosupport 180561 , and Create a service request

REFERENCE

• Transient /ddvar space usage increases 181878
• Alert Calculation for Space in /ddvar 180997

Note: The links in this article are for registered Dell customers only.