Data Domain: Cleanable Size is an Estimate

There is often confusion about the "Cleanable GiB" value presented on a Data Domain, and improper expectations about the amount of space that cleaning can recover.

 

The "Cleanable GiB" number given is purely as an estimate. It is not possible to get an accurate value for how much space cleaning will recover. This is due to the technological choices made when developing the Data Domain file system (DDFS).

 

An explanation follows of why estimates of cleanable space can vary substantially from the space that is recovered. There are other factors not accounted for here that can increase the difference between the estimate and the amount of disk space freed upon running clean.

 

When data is written to the Data Domain, the post-compression value is calculated and stored as static data for every file. The "Cleanable" value is the sum of the post-compression value for all deleted files since the last time cleaning was run to completion.

 

The "Cleanable" value becomes inaccurate if the file segments for deleted files have been used in deduplicating data in other files that have not been deleted. While there is a single file referring to an existing unique segment, the cleaning process does not consider those segments for being reclaimed. A file's post-compression size is added to the "Cleanable GiB" counter as if all its unique segments were about to be disposed of. In reality, some (or many) may not be removed as other files are using them for deduplication.

 

A more detailed example showing this effect follows:

 

Assume that there are five files added one by one to an mtree on a previously-empty Data Domain.

 

Since the first 100 GB files contain all unique data, the compression ratio is 1x (assuming the first file had no redundancy within the file itself). The 2nd-5th files can deduplicate against the first file's data and each of the older files as they get added. Each subsequent file gains increasing deduplication due to the increasing number of files to deduplicate against.

/data/col1/my_mtree/test01: precomp: 100 GB postcomp: 100 GB compression ratio: 1x
/data/col1/my_mtree/test03: precomp: 100 GB postcomp:  50 GB compression ratio: 2x
/data/col1/my_mtree/test04: precomp: 100 GB postcomp:  25 GB compression ratio: 4x
/data/col1/my_mtree/test04: precomp: 100 GB postcomp:  25 GB compression ratio: 4x
/data/col1/my_mtree/test05: precomp: 100 GB postcomp:   1 GB compression ratio: 100x

Resource           Size GiB   Used GiB   Avail GiB   Use%   Cleanable GiB*
----------------   --------   --------   ---------   ----   --------------
/data: pre-comp           -        500           -      -                -
/data: post-comp       1000        201         799    20%                0
----------------   --------   --------   ---------   ----   --------------

Example 1: Status after deleting the first three files:

Resource           Size GiB   Used GiB   Avail GiB   Use%   Cleanable GiB*
----------------   --------   --------   ---------   ----   --------------
/data: pre-comp           -        200           -      -                -
/data: post-comp       1000        201         799    20%              175
----------------   --------   --------   ---------   ----   --------------

 

If cleaning is run after this, it may only be able to reclaim 125 GiB instead of the full 175 GiB cleanable. This is because the last two files share segments with files 1-3. Cleaning cannot recover the other 50 GiB of space because those segments are still in use by files 3-5.

 

Example 2: Using the same starting point as with Example 1. Assume test01 is deleted, then a fastcopy is performed on the entire mtree (so all five files), then test02-test04 are deleted.

Resource           Size GiB   Used GiB   Avail GiB   Use%   Cleanable GiB*
----------------   --------   --------   ---------   ----   --------------
/data: pre-comp           -        800           -      -                -
/data: post-comp       1000        201         799    20%              200
----------------   --------   --------   ---------   ----   --------------

 

The "Size GiB" figure for pre-comp comes from (500 - 100) = 400 * 2 = 800; that is, 500 for the five original files minus 100 for deleting test01 equals 400 GiB. Next, 400 GiB is multiplied by 2 due to the fastcopy on all four remaining files.

 

The post-comp space used is still the same because a fastcopy only adds a tiny amount of space, consisting of the metadata pointers to the original data. The space usage has not changed despite the deletion of test01 because a cleaning has not yet been run.

 

After cleaning is run, the output becomes:

Resource           Size GiB   Used GiB   Avail GiB   Use%   Cleanable GiB*
----------------   --------   --------   ---------   ----   --------------
/data: pre-comp           -        800           -      -                -
/data: post-comp       1000        176         824    18%              200
----------------   --------   --------   ---------   ----   --------------

 

Even though 200 GiB was shown cleanable, only 25 GiB was cleaned. The "Cleanable GiB" showed as 200 because the post-comp file size of test01-test04 added up to 200 GiB. Only test01 was removed, which was 100 GiB, but 75 GiB of which was still in use by the other four files due to deduplication.

 

That may seem odd, since test02-test04 had also been deleted. However, although the system shows test02-test04 as removed, the data segments for those files could not be removed because those files had been fastcopied to another folder. Cleaning can fully recover the space only after all fastcopy versions have also been removed.

 

Another effect that this can have is that the "Cleanable GiB" estimate can sometimes be unexpectedly high. It may even be the same size as the overall storage capacity of the Data Domain. This can lead to confusion about whether to allow cleaning to run in order to avoid losing all data on the system.

 

As of DDOS 7.7, a better and more reliable way of estimating the amount of disk space clean would reclaim is available. It is now possible to accurately determine from the command line the amount of active tier space that the next cleaning will reclaim using the command:

# filesys cleanable-space calculate

 

The process runs the same as a regular cleaning, going through phases 1-4, but skipping phase 5 (copy). The copy phase is the phase that does the work of removing segments identified by the first four phases. As such, it takes the same amount of time as a regular cleaning to complete clean phases 1 through 4 to return a value. This is something that should not be run regularly to get an updated estimate - only when needed. Essentially, 'filesys cleanable-space calculate' runs active tier cleaning without the step that removes data.

 

The process can be monitored as follows:

# filesys cleanable-space watch
Beginning 'filesys cleanable-space calculation' monitoring.  Use Control-C to stop monitoring.

Cleaning: phase 1 of 4 (pre-merge)
  100.0% complete, 96233 GiB free; time: phase  0:02:07, total  0:02:07

Cleaning: phase 2 of 4 (pre-analysis)
  100.0% complete, 96233 GiB free; time: phase  0:06:51, total  0:08:59

Cleaning: phase 3 of 4 (pre-enumeration)
  100.0% complete, 96233 GiB free; time: phase  0:00:20, total  0:09:20

Cleaning: phase 4 of 4 (pre-select)
  100.0% complete, 96233 GiB free; time: phase  0:00:25, total  0:09:46

 

Once calculation is complete, the measurement result can be viewed with:

# filesys cleanable-space status

Cleanable space on active tier is 94649698202 bytes. Last calculated on 2023/08/25 03:29:51
Cleanable space calculation finished at 2023/08/25 03:29:51.

 

So in the above example, if cleaning were to be run it would free up 94,649,698,202 bytes, or 88.1 GiB. Meanwhile at the time of the calculation, the estimate reported by 'df' on the test Data Domain used was 41.9 GiB. As changes are made to the DDFS (new backups, more deletions, snapshots being created and expired) the calculation becomes inaccurate.

 

The calculation process can be stopped if necessary with the below command:

# filesys cleanable-space stop

The 'filesys cleanable-space stop' command stops calculating cleanable space in the system.
Are you sure? (yes|no) [no]: yes

ok, proceeding.

# filesys cleanable-space status
Cleanable space on active tier is 2607064 bytes. Last calculated on 2021/06/27 23:23:05
Cleanable space calculation started at 2021/06/27 23:27:58 and was aborted at 2021/06/27 23:28:19.
Cleaning was aborted by user.