Data Domain: Too Frequent Cleaning Causing Poor Data Locality

• Customer reports cleaning is taking too long or appears to be stuck.
• Poor cache hit rate
• High level of disk utilization
• Poor restore and/or backup performance

This article helps diagnose cases of slow cleaning/garbage collection (GC) caused by poor file locality. Check other possible causes using the slow cleaning troubleshooting document (linked below). This is one step in that troubleshooting procedure.

1. Although too-frequent cleaning can lead to file fragmentation, cleaning every day may be appropriate if the files have a short retention period such as 24 hours.
   Therefore, consider:
   • How often is cleaning scheduled? Check Cleaning Status and Schedule: Data Domain: Check Cleaning Status and Schedule
   • What is the retention period of the files on the system? Check the file age distribution: Data Domain - Storage Management: Viewing File Distribution.
   If there are many old files on the system and cleaning is scheduled more often than once per week, the system may be experiencing poor file locality.
2. Next, check the cleaning copy phase efficiency, which shows the percentage of live data that is copied. If this is low, the retention period is likely low so the copy phase is more efficient at reclaiming more space. This information can be found by searching for "GC copy phase details" in a recent autosupport (ASUP).

   GC copy phase details:                                   Recent    Cumulative
   Number of containers selected:                         13091499     13091499
   Number of containers copied:                           13091499     13091499
   Number of GC containers copied:                         6131917      6131917
   Number of containers generated:                         4173221      4173221
   Number of Data containers copied:                       6959582      6959582
   Percentage of selected copied:                            100.0%       100.0%
   Percentage of live data in copied:                         31.9%        31.9%	<==
   Copy selected speed in containers/second:                 724.2        724.2
   Space freed in MiB:                                  40132251.0   40132251.0

3. GC phase: Pre-enumeration times increasing with time (implies locality becoming worse) trending. This information can be found by searching for "GC stats for Physical Cleaning" in a recent ASUP.

   GC stats for Physical Cleaning on Active Success 1 Aborted 0 Skipped 0
   Most recent successful GC container range: 575 to 442476459
   
   GC phase:        pre-merge    time:     566 average:     566    seg/s:        0    cont/s:   23129
   GC phase:     pre-analysis    time:    6576 average:    6576    seg/s:        0    cont/s:    1990
   GC phase:  pre-enumeration    time:    7722 average:    7722    seg/s: 27104949    cont/s:    1695	<==
   GC phase:       pre-select    time:    6292 average:    6292    seg/s: 18046557    cont/s:   16754
   GC phase:             copy    time:   18078 average:   18078    seg/s:        0    cont/s:     724
   GC phase:          summary    time:    8059 average:    8059    seg/s:        0    cont/s:    1624

4. Review physical space usage trends.

• Create a service request
• Gather an autosupport: Data Domain - Gathering Autosupports
• Upload a support bundle (SUB): Data Domain: How to collect/upload a support bundle (SUB) from a Data Domain Restorer (DDR)

REFERENCE

• Data Domain Cleaning taking too long
• Data Domain: An overview of the Data Domain File System clean/garbage collection phases
• Data Domain: Scheduling Cleaning on a DDR
• Data Domain - How to check the Data Domain File System utilization statistics
• Data Domain - How to review the DDFS Cache statistics with "system show performance" output