Data Domain: Physical Capacity Measurement or Physical Capacity Reporting Frequently Asked Questions
Summary: Data Domain Operating System (DD OS) Physical Capacity Measurement (PCM) or Physical Capacity Reporting (PCR) frequently asked questions.
Instructions
Version 5.7 of the Data Domain Operating System (DD OS) introduces new functionality known as Physical Capacity Measurement (PCM) or Physical Capacity Reporting (PCR).
This article describes common use cases and questions around this feature. PCM and PCR are used interchangeably in this document.
What is Physical Capacity Measurement (PCM)?
PCM is a new feature supported in DD OS 5.7 and later which allows calculation of accurate hard drive utilization by a directory tree, collection of directory trees, mtree, or a collection of MTrees.
How does this differ from features in previous releases of DDOS?
When a file is ingested on a DDR, we record various statistics about the file. One such statistic is "post-lc bytes" or the physical amount of space taken by a file when written to the system. We can view post-lc bytes for a file or directory tree using the filesys show compression command - for example:
sysadmin@dd9500# filesys show compression /data/col1/jf1 Total files: 4; bytes/storage_used: 1.3 Original Bytes: 4,309,378,324 Globally Compressed: 3,242,487,836 Locally Compressed: 3,293,594,658 Meta-data: 13,897,112
This indicates that the above directory tree contains four files which, in total, used 3,293,594,658 bytes (3.07Gb) of physical space when ingested.
PCM is an effort to avoid inconsistent results caused by the above statistics becoming stale. As PCM can generate reports of hard drive utilization at a specific point in time, the above limitations no longer apply and results are guaranteed to be more accurate.
Are there any known facts or setups in which results calculated by PCM are known to be less than optimally accurate?
PCM does sampling for calculating the disk space used by files in the path set. That means some files may be left out of the enumeration for unique segments, because, for performance reasons, files smaller than a given size are not accounted for. The minimum size of the individual files in the pathsets that PCM jobs consider for enumeration (and hence the ones it accounts for disk space used for) depends on the average file system segment size:
- If the average segment size is 4 K, then files 4 MB and smaller are ignored by PCM
- If the average segment size is 8 K, then files 8 MB and smaller are ignored by PCM
- If the average segment size is 12 K, then files 12 MB and smaller are ignored by PCM
Real-world customer data shows that average segment size hovers around 9 KB for the most common use cases, so an approximate figure for files not being considered when a PCM calculates disk space used would be those smaller than 9 MB.
Are any additional licenses required for PCM?
No - PCM is not a licensed feature and as a result no additional licenses are required to use PCM.
Is PCM support in all platforms?
No - PCM is supported on all Hardware and Virtual Data Domain appliances(DDVE), except on Active Tier on Object Storage (ATOS) DDVEs.
PCM only measures and enumerates the files on the Active Tier: Files sitting in Cloud Units (or for legacy Extended Retention, those in Archiver Units) are not in the size calculation for PCM jobs.
Are there any other pre-requisites required before PCM can be used?
By default PCM is disabled in DDOS 5.7. Before it can be used, it must be enabled and its cache initialized as shown below:
sysadmin@dd9500# compression physical-capacity-measurement enable and-initialize physical-capacity-measurement enabled. Initialization started.
Note: The PCM cache is used to speed future PCM jobs, and initialization of the cache can take considerable time. Despite these PCM jobs can start to be queued whilst the PCM cache is being initialized.
How does PCM calculate usage totals?
PCM uses mtree snapshots to determine physical utilization for a group of files. As a result, when a PCM job starts the following happens:
- A mtree snapshot is created against underlying MTrees.
sysadmin@dd9500# snapshot list mtree /data/col1/jf2 Snapshot Information for MTree: /data/col1/jf2 ---------------------------------------------- Name Pre-Comp (GiB) Create Date Retain Until Status --------------------------------- -------------- ----------------- ----------------- ------- pcr_snap_1440284055_1440360259_19 6.0 Aug 23 2015 13:04 Dec 31 1969 16:00 expired --------------------------------- -------------- ----------------- ----------------- -------
- PCM finds files from the snapshot which are to be in the PCM job (that is in the pathsets or MTrees specified).
- PCM walks the segment tree of these files to essentially build a list of unique segment fingerprints referenced by all the files.
- PCM then finds corresponding segments on disk (within the container set) and calculates the sum of the size of those segments.
-
The sum of the size of these segments represents the current hard drive utilization by the corresponding files.
- In addition to the above, the pre-compressed size of the set of files can be found from the corresponding file metadata.
- Once PCM jobs are complete, underlying PCM snapshots are expired for later removal.
How do PCM jobs work?
PCM jobs are submitted by a user (or by a schedule) and are added to a PCM work queue. Depending on the system workload PCM jobs may then be picked from the queue and started immediately or may be deferred for time.
Examples of why PCM jobs may be deferred are as follows:
- Active tier clean is running on the system - PCM jobs and active tier clean cannot run in parallel. As a result, PCM jobs queued while active tier clean is running, are deferred until active tier clean completes.
- There are already some PCM jobs running against underlying MTrees - PCM uses mtree snapshots and there are strict limits on how many PCM snapshots a given user can create at a given time against a single mtree. If these limits are exceeded by a new PCM job, the job is deferred until the existing job complete. The PCR snapshot limit is 3.
Is it possible to control the resources used by PCM on a system?
PCM uses a throttling mechanism which is similar to that used by active tier clean, that is the PCM throttle can be set from 0 (not aggressive) to 100 (aggressive). Obviously, the higher the throttle, the more resources are used by PCM and the larger impact PCM jobs may have on other workload on the system.
By default the PCM throttle is set to 20, that is:
sysadmin@dd9500# compression physical-capacity-measurement throttle show Throttle is set to 20 percent (default).
PCM throttle can be modified as follows with the change to throttle taking place immediately (that is no DDFS restart is required for PCM to pick up the new throttle setting):
sysadmin@dd9500# compression physical-capacity-measurement throttle set 50 Throttle set to 50 percent.
What are pathsets?
PCM jobs can be run in two ways, that is:
- Against a pre-defined 'path set' (that is user specified collection of directories)
- Against a single mtree
Before jobs can be run against a given path set, the path set must be created or defined as follows:
sysadmin@dd9500# compression physical-capacity-measurement pathset create jfall paths /data/col1/jf1,/data/col1/jf2 Pathset "jfall" created.
Specific directories can be added to or removed from an existing path set as follows:
sysadmin@dd9500# compression physical-capacity-measurement pathset del jfall paths /data/col1/jf2 Path(s) deleted from pathset "jfall". sysadmin@dd9500# compression physical-capacity-measurement pathset add jfall paths /data/col1/jf2 Path(s) added to pathset "jfall".
All pathsets which were created are displayed as follows:
sysadmin@dd9500# compression physical-capacity-measurement pathset show list Pathset Number of paths Measurement-retention (days) --------------- --------------- ---------------------------- jf1 1 180 jf2 1 180 jfall 2 180 phys-gandhi3 1 180 phys-gandhi5-fc 1 180 phys-gandhi5 1 180 phys2-gandhi3 2 180 --------------- --------------- ---------------------------- 7 pathset(s) found.
To view specific paths defined within a path set, the pathset show detailed command is used:
sysadmin@dd9500# compression physical-capacity-measurement pathset show detailed jfall Pathset: jfall Number of paths: 2 Measurement-retention: 180 day(s) Paths: /data/col1/jf1 /data/col1/jf2 sysadmin@dd9500#
To delete a path, set the pathset destroy command can be used:
sysadmin@dd9500# compression physical-capacity-measurement pathset destroy jfall
Note: This removes all history for the given path set.
How is a PCM job started?
A new PCM job is submitted to the PCM work queue by using the sample start command, that is:
sysadmin@dd9500# compression physical-capacity-measurement sample start pathsets jfall Measurement task(s) submitted and will begin as soon as resources are available.
In the above example, a pre-defined path set was used. To submit a PCM job against a single mtree, the mtree is specified, that is:
sysadmin@dd9500# compression physical-capacity-measurement sample start mtrees /data/col1/backup Measurement task(s) submitted and will begin as soon as resources are available.
By default, PCM jobs are submitted with a priority of 'normal'. It is also possible to specify a priority of urgent:
sysadmin@dd9500# compression physical-capacity-measurement sample start pathsets jf1 priority urgent Measurement task(s) submitted and will begin as soon as resources are available.
Jobs with priority of 'urgent' are queued ahead of those with priority of 'normal' (meaning that they are picked up and worked in preference to any submitted jobs of priority 'normal').
A list of submitted or running jobs is displayed using the sample show current command, for example:
sysadmin@dd9500# compression physical-capacity-measurement sample show current Task ID Type Name User State Creation Time Measurement Time Start Time Priority Percent (Submitted Time) Done ----------- ---- ----- -------- --------- ------------------- ------------------- ---------- -------- -------- 47244640259 PS jf2 sysadmin Scheduled 2015/08/23 12:24:12 2015/08/23 12:24:12 -- Urgent 0 47244640258 PS jf1 sysadmin Scheduled 2015/08/23 12:24:09 2015/08/23 12:24:09 -- Urgent 0 47244640257 PS jfall sysadmin Scheduled 2015/08/23 12:23:06 2015/08/23 12:23:06 -- Normal 0 ----------- ---- ----- -------- --------- ------------------- ------------------- ---------- -------- -------- sysadmin@dd9500#
Can PCM jobs be scheduled?
Yes - If a specific PCM job must be run regularly, it is scheduled to run automatically as required. For example:
sysadmin@dd9500# compression physical-capacity-measurement schedule create jf_sched pathsets jfall,jf1,jf2 time 1400 Schedule "jf_sched" created.
Note: Schedules are created to run daily, on specific days of the week, or certain days of each month.
An existing schedule is modified using the schedule modify command:
sysadmin@dd9500# compression physical-capacity-measurement schedule modify jf_sched priority urgent time 1700 day Wed,Fri Schedule "jf_sched" modified.
In addition, an existing schedule may have pathsets added or removed as follows:
sysadmin@dd9500# compression physical-capacity-measurement schedule del jf_sched pathsets jf2 Schedule "jf_sched" modified. sysadmin@dd9500# compression physical-capacity-measurement schedule add jf_sched pathsets jf2 Schedule "jf_sched" modified.
Note: A schedule cannot only contain pathsets OR MTrees (that is the two cannot be mixed):
sysadmin@dd9500# compression physical-capacity-measurement schedule create jf_sched2 mtrees /data/col1/backup time 1400 Schedule "jf_sched2" created. sysadmin@dd9500# compression physical-capacity-measurement schedule add jf_sched2 pathsets jfall **** Failed to add: this schedule is only for mtrees.
To view details of existing schedules, the schedule show all command is used, for example:
sysadmin@dd9500# compression physical-capacity-measurement schedule show all Name: jf_sched Status: enabled Priority: urgent Frequency: weekly on Wed, Fri Time: 17:00 Pathset(s): jfall jf1 jf2 Name: jf_sched2 Status: enabled Priority: normal Frequency: daily Time: 14:00 MTree(s): /data/col1/backup
Existing schedules may be disabled or enabled immediately, that is:
sysadmin@dd9500# compression physical-capacity-measurement schedule disable jf_sched2 Schedule "jf_sched2" disabled. sysadmin@dd9500# compression physical-capacity-measurement schedule enable jf_sched2 Schedule "jf_sched2" enabled.
A schedule may also be destroyed:
sysadmin@dd9500# compression physical-capacity-measurement schedule destroy jf_sched2 Schedule "jf_sched2" destroyed.
Note: This does NOT remove history for the corresponding mtrees/pathsets (it means that new PCM jobs are not automatically scheduled).
How are scheduled jobs started?
When a PCM schedule is added and enabled, this causes a corresponding entry to be added to /etc/crontab, that is:
# # collection.1.crontab.pcr.jf_sched.0 # 00 17 * * Wed,Fri root /ddr/bin/ddsh -a compression physical-capacity-measurement sample start force priority urgent objects-from-schedule jf_sched
Note: The cron job is removed from
/etc/crontab if the schedule is disabled or destroyed.
Can I stop running a PCM job?
Yes - Running PCM jobs can be aborted using either the task id or pathset/mtree names. For example we see that we have two PCM jobs queued:
SE@dd9500## compression physical-capacity-measurement sample show current Task ID Type Name User State Creation Time Measurement Time Start Time Priority Percent (Submitted Time) Done ------------ ---- ----- -------- --------- ------------------- ------------------- ---------- -------- -------- 124554051585 PS jfall sysadmin Scheduled 2015/08/30 16:00:48 2015/08/30 16:00:48 -- Normal 0 124554051586 PS jfall sysadmin Scheduled 2015/08/30 16:01:55 2015/08/30 16:01:55 -- Normal 0 ------------ ---- ----- -------- --------- ------------------- ------------------- ---------- -------- --------
These jobs may be aborted using either the task-id (to stop a single job):
SE@dd9500## compression physical-capacity-measurement sample stop task-id 124554051585 ** This will abort any submitted or running compression physical-capacity-measurement sampling tasks. Do you want to proceed? (yes|no) [no]: yes 1 task(s) aborted.
Leaving us with a single running job:
SE@dd9500## compression physical-capacity-measurement sample show current Task ID Type Name User State Creation Time Measurement Time Start Time Priority Percent (Submitted Time) Done ------------ ---- ----- -------- --------- ------------------- ------------------- ---------- -------- -------- 124554051586 PS jfall sysadmin Scheduled 2015/08/30 16:01:55 2015/08/30 16:01:55 -- Normal 0 ------------ ---- ----- -------- --------- ------------------- ------------------- ---------- -------- --------
Or path set name:
SE@dd9500## compression physical-capacity-measurement sample stop pathsets jfall ** This will abort any submitted or running compression physical-capacity-measurement sampling tasks. Do you want to proceed? (yes|no) [no]: yes 1 task(s) aborted.
Leaving us with no jobs:
SE@dd9500## compression physical-capacity-measurement sample show current No measurement tasks found.
How can details of completed jobs be displayed?
Details of completed jobs are viewed with the sample show history command. For example, to show details for a single path set:
SE@dd9500## compression physical-capacity-measurement sample show history pathset jfall Pathset: jfall Measurement Time Logical Used Physical Used Global-Comp Local-Comp Total-Comp (Pre-Comp) (Post-Comp) Factor Factor Factor (GiB) (GiB) (Reduction %) ------------------- ------------ ------------- ----------- ---------- -------------- 2015/08/23 12:23:06 7.0 4.2 1.70x 0.98x 1.67x (40.24%) 2015/08/23 13:04:20 10.0 6.2 1.63x 0.98x 1.61x (37.84%) 2015/08/26 14:00:01 10.0 6.2 1.63x 0.98x 1.61x (37.84%) 2015/08/27 14:00:01 10.0 6.2 1.63x 0.98x 1.61x (37.84%) 2015/08/28 14:00:02 10.0 6.2 1.63x 0.98x 1.61x (37.84%) 2015/08/29 14:00:02 10.0 6.2 1.63x 0.98x 1.61x (37.84%) 2015/08/30 14:00:01 10.0 6.2 1.63x 0.98x 1.61x (37.84%) ------------------- ------------ ------------- ----------- ---------- -------------- Total number of measurements retrieved = 7.
The detailed-history parameter also shows the start and end times of each job:
SE@dd9500## compression physical-capacity-measurement sample show detailed-history pathset jfall Pathset: jfall Measurement Time Logical Used Physical Used Global-Comp Local-Comp Total-Comp Task ID Task Start Time Task End Time (Pre-Comp) (Post-Comp) Factor Factor Factor (GiB) (GiB) (Reduction %) ------------------- ------------ ------------- ----------- ---------- -------------- ------------ ------------------- ------------------- 2015/08/23 12:23:06 7.0 4.2 1.70x 0.98x 1.67x (40.24%) 47244640257 2015/08/23 12:25:19 2015/08/23 12:25:23 2015/08/23 13:04:20 10.0 6.2 1.63x 0.98x 1.61x (37.84%) 51539607553 2015/08/23 13:05:45 2015/08/23 13:05:48 2015/08/26 14:00:01 10.0 6.2 1.63x 0.98x 1.61x (37.84%) 77309411329 2015/08/26 14:02:50 2015/08/26 14:02:50 2015/08/27 14:00:01 10.0 6.2 1.63x 0.98x 1.61x (37.84%) 85899345921 2015/08/27 14:03:06 2015/08/27 14:03:06 2015/08/28 14:00:02 10.0 6.2 1.63x 0.98x 1.61x (37.84%) 94489280513 2015/08/28 14:02:50 2015/08/28 14:02:51 2015/08/29 14:00:02 10.0 6.2 1.63x 0.98x 1.61x (37.84%) 103079215105 2015/08/29 14:01:40 2015/08/29 14:01:41 2015/08/30 14:00:01 10.0 6.2 1.63x 0.98x 1.61x (37.84%) 115964116993 2015/08/30 14:04:12 2015/08/30 14:04:12 ------------------- ------------ ------------- ----------- ---------- -------------- ------------ ------------------- ------------------- Total number of measurements retrieved = 7.
Note: Either command may be modified to only retrieve results over a specific time period:
SE@dd9500## compression physical-capacity-measurement sample show history pathset jfall last 2days Pathset: jfall Measurement Time Logical Used Physical Used Global-Comp Local-Comp Total-Comp (Pre-Comp) (Post-Comp) Factor Factor Factor (GiB) (GiB) (Reduction %) ------------------- ------------ ------------- ----------- ---------- -------------- 2015/08/29 14:00:02 10.0 6.2 1.63x 0.98x 1.61x (37.84%) 2015/08/30 14:00:01 10.0 6.2 1.63x 0.98x 1.61x (37.84%) ------------------- ------------ ------------- ----------- ---------- -------------- Total number of measurements retrieved = 2.
Or between specific dates or times:
SE@dd9500## compression physical-capacity-measurement sample show history pathset jfall start 08231010 end 08231400 Pathset: jfall Measurement Time Logical Used Physical Used Global-Comp Local-Comp Total-Comp (Pre-Comp) (Post-Comp) Factor Factor Factor (GiB) (GiB) (Reduction %) ------------------- ------------ ------------- ----------- ---------- -------------- 2015/08/23 12:23:06 7.0 4.2 1.70x 0.98x 1.67x (40.24%) 2015/08/23 13:04:20 10.0 6.2 1.63x 0.98x 1.61x (37.84%) ------------------- ------------ ------------- ----------- ---------- -------------- Total number of measurements retrieved = 2.
How long is a specific history kept for a specific pathset/mtree?
By default, the results of each PCM path set are kept for 180 days. This may be changed by modification of the corresponding path set:
SE@dd9500## compression physical-capacity-measurement pathset modify jfall measurement-retention 14 Measurement-retention changed to 14 day(s).
PCM history is held in the systems historical database. As a result, if the historical database is lost or damaged, details of all PCM history are also lost.
Are there any caveats to be aware of when using PCM?
PCM jobs are suspended if the system has less than 10% available space.
PCM jobs are suspended while the cleaning cycle is running.
As previously stated, PCM is a tool used to calculate physical utilization by a specific set of directories/mtrees. When calculating results for a specific path set or set of MTrees, PCM only counts the size of each unique segment of data used by the path set or set of MTrees once.
As a result of this, if files de-duplicating against the same data are in multiple different PCM jobs, each of the segments making up the files may be counted multiple times (once by each PCM job). This means that while the results of each individual PCM job are accurate, the results of multiple PCM jobs cannot be summed to give accurate physical utilization for the sum of pathsets/mtrees in the jobs.
Due to the way PCM jobs are put together, the path set or MTree post-comp space used on disk:
- PCM jobs do not account for dead data (disk space used by deleted files, which space on disk may not have been reclaimed by running GC yet).
- PCM does not account for any data locked in snapshots created for the MTree or pathsets being measured.
For example, I have a mtree called /data/col1/jf1 in which I create three 1Gb files, that is:
!!!! dd9500 YOUR DATA IS IN DANGER !!!! # for i in 1 2 3 ; do
> dd if=/dev/urandom of=/data/col1/jf1/${i} bs=1024k count=1024
> done
I then copy those files to a second mtree (/data/col1/jf2):
!!!! dd9500 YOUR DATA IS IN DANGER !!!! # cp /data/col1/jf1/1 /data/col1/jf2/4 !!!! dd9500 YOUR DATA IS IN DANGER !!!! # cp /data/col1/jf1/2 /data/col1/jf2/5 !!!! dd9500 YOUR DATA IS IN DANGER !!!! # cp /data/col1/jf1/3 /data/col1/jf2/6
Finally, I create a new 1Gb file in /data/col1/jf2:
!!!! dd9500 YOUR DATA IS IN DANGER !!!! # dd if=/dev/urandom of=/data/col1/jf2/7 bs=1024k count=1024 1024+0 records in 1024+0 records out
If local compression of data is disregarded and only de-duplication is considered, it is clear that each mtree used the following amount of physical space when the files were written:
/data/col1/jf1: 3Gb
/data/col1/jf2: 1Gb (for the new file - copied files would have de-duplicated against existing data so would have consumed minimal physical space).
As a result, the sum of physical space utilization by /data/col1/jf1 and /data/col1/jf2 should be around 4Gb.
Three PCM pathsets are created:
jf1 containing /data/col1/jf1
jf2 containing /data/col1/jf2
jfall containing data/col1/jf1 and /data/col1/jf2
The PCM jobs are run and provide output as follows:
Pathset: jf1 Measurement Time Logical Used Physical Used Global-Comp Local-Comp Total-Comp (Pre-Comp) (Post-Comp) Factor Factor Factor (GiB) (GiB) (Reduction %) ------------------- ------------ ------------- ----------- ---------- -------------- 2015/08/23 12:24:09 3.0 3.2 0.96x 0.98x 0.94x (-6.21%) ------------------- ------------ ------------- ----------- ---------- -------------- Pathset: jf2 Measurement Time Logical Used Physical Used Global-Comp Local-Comp Total-Comp (Pre-Comp) (Post-Comp) Factor Factor Factor (GiB) (GiB) (Reduction %) ------------------- ------------ ------------- ----------- ---------- -------------- 2015/08/23 12:24:12 4.0 4.2 0.98x 0.98x 0.96x (-4.14%) ------------------- ------------ ------------- ----------- ---------- --------------
These values are correct as each PCM job is only looking at physical data referenced by the files in its corresponding path set. This means that data for files which were copied is counted twice (once by each PCM job).
It may seem reasonable that to get total physical utilization by the /data/col1/jf1 and /data/col1/jf2 MTrees, we can sum the values of 'physical used' from the above outputs.
To get an accurate value for total physical utilization of /data/col1/jf1 and /data/col1/jf2, it is necessary to run a single PCM job covering both of these MTrees (that is use jfall). This ensures that duplicate segments are only counted once and not twice as in the example above, that is:
Pathset: jfall Measurement Time Logical Used Physical Used Global-Comp Local-Comp Total-Comp (Pre-Comp) (Post-Comp) Factor Factor Factor (GiB) (GiB) (Reduction %) ------------------- ------------ ------------- ----------- ---------- -------------- 2015/08/23 12:23:06 7.0 4.2 1.70x 0.98x 1.67x (40.24%) ------------------- ------------ ------------- ----------- ---------- --------------
In summary, the output of multiple PCM jobs cannot be summed to give accurate physical utilization for a set of pathsets/mtrees. Instead, a single PCM path set should be defined covering all required mtrees/directories as this ensures duplicate data is only counted once. If this is not done and a separate PCM job were to be run for each mtree on a system, for example, then results summed it is entirely possible that the total physical capacity used exceeds the raw capacity of the system.
Can PCM jobs only be submitted by the DDSH command line or can a UI also be used?
In this document, the Data Domain command-line interface (DDSH) is used to configure, submit, and review PCM jobs.