Data Domain: Retention Lock Frequently Asked Questions

What is a retention lock?

What are the different versions of retention lock?

• Governance: The less strict of the two RL functions - Locks against files can be reverted if necessary.
• Compliance: The stricter of the two functions, introduced in DDOS 5.3 - It adheres to several common regulatory standards; that is, locks against files cannot be reverted. The DD must be configured with a "security officer" user to authenticate certain commands. There are various restrictions on other functionality to prevent locked data from being removed or locks being reverted early.

Which data access protocols are supported with retention lock?

• The NFS, CIFS, and DD Boost protocols are fully supported against MTrees using retention lock governance or compliance mode.
• The Virtual Tape Library (VTL) protocol is only supported against MTrees using retention lock governance mode. Automatic retention lock is not supported on Data Domain VTL. See the Data Domain Administration Guide to determine how to unlock retention-locked tapes so that they can be written to.

What regulatory standards does retention lock compliance mode meet?

How is retention lock governance enabled?

1. Add a retention lock governance (RLG) license to the DD.
2. Enable RLG against any required MTree:

   # mtree retention-lock enable mode governance mtree <mtree
   

How is retention lock compliance enabled?

1. There are specific instructions for some newer Data Domain models with iDRAC. (Log in as a registered Dell Support user is required to view this article.)
2. Add a retention lock compliance (RLC) license to the DD.
3. Create a user with the role of "security" (assuming such a user does not exist):

   (ADMIN USER) # user add <username> role security

4. Log in to the DD with the "security" user and enable security user authorization:

   (SECURITY USER) # authorization policy set security-officer enabled

5. Configure the system for retention lock compliance mode. Once the following command runs to completion, the system reboots automatically:

   (ADMIN USER) # system retention-lock compliance configure

6. Once the system has rebooted, enable retention lock compliance mode on the system:

   (ADMIN USER) # system retention-lock compliance enable

    
   Note: For newer DDOS releases, this task must be performed with the Data Domain UI under Administration > Compliance
    
7. Enable retention lock compliance mode against any required MTrees:

   (ADMIN USER) # mtree retention-lock enable mode compliance mtree <mtree>

How is it possible to determine which MTrees have retention lock enabled?

MTrees with retention lock enabled are indicated in the output of 'mtree list'. For example:

# mtree list
Name                                Pre-Comp (GiB)   Status    Tenant-Unit
---------------------------------   --------------   -------   -----------
...
/data/col1/rich-retention-lock                 0.0   RW/RLGE   -
/data/col1/rl_test                             0.0   RW/RLGD   -
/data/col1/rl_test_comp                        0.0   RW/RLCE   -
/data/col1/test                                3.1   RW/RLGE   -
...
---------------------------------   --------------   -------   -----------
...
 RLGE : Retention-Lock Governance Enabled
 RLGD : Retention-Lock Governance Disabled
 RLCE : Retention-Lock Compliance Enabled

Can a MTree with Retention Lock Governance enabled be converted to Retention Lock Compliance?

As stated in the DDOS Administration Guide, this is not possible.

Can a MTree with Retention Lock Compliance enabled be converted to Retention Lock Governance?

As stated in the DDOS Administration Guide, this is not possible.

How are file retention or lock periods set?

Once a retention lock is enabled against a MTree, a minimum and maximum retention period must be set. These periods dictate the minimum and maximum time a file within the MTree can be locked for. For example:

# mtree retention-lock set min-retention-period <period> mtree <mtree>
# mtree retention-lock set max-retention-period <period> mtree <mtree>

Periods can be given in various units as follows:

• 1 min
• 1 hr
• 1 day
• 1 mo
• 1 year

How can existing retention periods be displayed?

This can be done using the following two commands:

# mtree retention-lock show min-retention-period mtree <mtree>
# mtree retention-lock show max-retention-period mtree <mtree>

For example:

# mtree retention-lock show min-retention-period mtree /data/col1/rl_test
Retention-lock min-retention-period of mtree /data/col1/rl_test is: 720 minutes.
# mtree retention-lock show max-retention-period mtree /data/col1/rl_test
Retention-lock max-retention-period of mtree /data/col1/rl_test is: 30 days.

How are files within a MTree with retention lock enabled locked?

• When retention lock is enabled against a MTree, existing files within the MTree are not automatically locked. That is, all preexisting files remain read or write.
• When a new file is written to a MTree with retention lock enabled, the file is not automatically retention locked. That is, the new file remains as read or write.
• Starting with DDOS 6.2.0.20, there is a feature in DDOS called "automatic retention lock," which can set a lock on all written files automatically. See the "Automatic Retention Lock" section further down in this article to learn more.
• To retention lock a specific file, the atime of the file must be modified to match the date, and time the file should be retention locked. That is, the date and time until which the file should remain read-only. Until the time is modified in this way, the file cannot be retention locked (and can be modified or removed).
  • A file's atime can be changed from an NFS or CIFS client using the 'touch' command:

    # touch -a -t <expiry time> <file to be locked>

  • For example, to set atime of /data/col1/rl_test/testfileto 07:05 on 8 June:

    # touch -a -t 06080705 /data/col1/rl_test/testfile

  • The period from current time to future atime must be within the minimum and maximum retention periods for the MTree. Otherwise, an error is generated when modifying the file's atime:

    # touch -a -t 08080705 /data/col1/rl_test/testfile
    touch: setting times of `/data/col1/rl_test/testfile': Permission denied

  • A corresponding message is also shown in the Data Domain File System (DDFS) log files:

    06/07 13:44:57.197 (tid 0x2b28ee5258c0): Attempt to set atime of /data/col1/rl_test/testfile to larger than maximum retention period of mtree.

  • CIFS (Windows) clients do not, by default, include a touch command or utility. However, several such utilities are freely available for download from various third-party websites.
  • Due to version dependencies on the client system FUSE libraries, there are cases in which using "touch -a" to set the file retention lock does not work, as the client OS does not pass on the "atime" modification request. If such a thing happens, a possible workaround is setting both the "atime" and the "mtime" (modification time) in the "touch" command, like "touch -am," which in our testing makes the client OS to pass down the request to modify the "atime." The downside is that the "mtime" is set to the same date and time, which may or may not be an acceptable tradeoff

Which backup applications support automatically retention locking files after writing them to a DD?

Data Domain Retention Lock is compatible with industry-standard, NAS-based Write-Once-Read-Many (WORM) protocols. Integration is qualified with archive applications such as Symantec Enterprise Vault, SourceOne, Cloud Tiering Appliance, or DiskXtender.

For Dell NetWorker, both governance and compliance modes are supported.

As of June 2024, recent Avamar versions support both Data Domain RLC and RLG as part of the Avamar "Immutable Backups" feature. See the article below and the Avamar documentation for details: Avamar and Data Domain: Enabling Avamar Immutable Backups and Data Domain Compliance Mode Retention Lock

It is critical that the steps to enable the Avamar "Immutable Backups" feature are followed strictly. Failure to do so may end up with an Avamar MTree in the DD that cannot be further written to, or that has operational issues which force lengthy recovery. Avamar does not work with DD Automatic Retention Lock (ARL), and ARL must not be enabled for any Avamar MTree on a DD.

Customers using other backup applications that do not natively support DD RL can also develop custom scripts to manually set file retention periods. In this scenario, ensure that custom scripts set files' atimes such that they are unlocked before the backup application attempts to delete the file. Failure to do this can result in the backup application attempting and failing to delete locked files. The files are left on the DD indefinitely consuming disk space. See the Data Domain Administration Guide.

What is an Automatic Retention Lock?

For backup applications that do not natively support the Data Domain retention lock feature, it has always been an issue for customers to leverage the feature. The backup administrator must configure scripts to set the retention lock on newly ingested files for a MTree. The lock must be set so that it expires shortly before the backup administrator expires and deletes the backup.

Contrary to regular RL, Automatic Retention Lock sets a lock on each and every file written to the MTree once the feature has been enabled. This prevents new or modified files from being changed or removed for a given period of time after the cooling-off period configured has passed. If the MTree has files that need be changed at a later point (either backup files, or backup application-related files, such as NetWorker's 'volhdr' or '_nsr_serial') ARL prevents these from being modified. This can cause all sorts of possible issues.

Hence, ARL is not supported for VTL.

For the same reason, ARL is not supported for NetWorker. Use of ARL with NetWorker could result in unintentional data unavailability or even data loss.

Before DDOS 7.8, Automatic Retention Lock cannot be used on DD Boost Logical Storage Units (LSU). Attempting to enable this returns an error stating that it is not supported. From 7.8 onwards, ARL is supported for DD Boost LSUs.

ARL used on target DDs for Managed File Replication (MFR) should have a long enough 'automatic-lock-delay' setting to ensure that clone operations are complete for the backup set before lock is set on the files. Example: A small file that is part of a backup set finishes replicating quickly while a larger file takes longer. The first file is retention locked by the time that the larger file finishes replicating. It then encounters an error when NetWorker attempts to move all the files of the backup set to the final archival directory.

Recent changes in DDOS for fastcopy/filecopy result in the "mtime" of the source file being retained when copying between locations on the same DD. As the ARL delay is determined by reviewing the "mtime" of a file, the copy may be automatically locked on creation if creating it takes longer than the cooling off period for the original file.

On applicable versions, there are additional options in the mtree retention-lock command, as shown below. This feature can be configured through the UI as well by choosing "Automatic" instead of "Manual" in the "Use" option:

# mtree retention-lock set
{min-retention-period | max-retention-period |
automatic-retention-period | automatic-lock-delay} <period>
mtree <mtree-path>

ARL locks a file immediately after a preconfigured cool-off period expires (automatic-lock-delay). The lock occurs if:

• The file is written to an RL-enabled MTree,
• The lock is valid for 'automatic-retention-period' from the moment it was set, and
• The value is within the 'min-retention-period' and 'max-retention-period' values set for the MTree

For more usage and general details about the feature, check the corresponding DDOS Administration Guide. This feature is not well-suited for situations in which the same MTree is used as the target for backups that have differing requirements for lock periods.

What can or cannot be done against a locked file?

• Retention-locked files are read-only and cannot be modified or deleted.
• Once a file's retention period expires, it is "unlocked." When in an unlocked state the file cannot be modified, but it can be deleted. The DD does not automatically delete a file when its retention period expires. Subsequent deletion must be done from a client system or backup application.
• Once set, the retention period for a specific file cannot be reduced (that is, the file's atime cannot be brought forward).
• Retention periods can, however, be increased (atime can be increased up to the maximum retention period for the MTree).
• Ownership and permission settings for a file can continue to be modified while the file is locked.
• It is only possible to rename or delete a directory in a retention lock MTree (RLGE, RLGD, or RLCE) if that directory contains no files. If the directory contains files (even if these files are not retention locked), the rename or delete of the directory fails
• Even if it does not change the file contents itself, renaming locked files is not allowed. Renaming is also not allowed once the file's lock has expired. For files that are no longer locked, the only permitted operation is a delete. This is changed in DDOS 7.7.4. Files that are no longer locked can be renamed.

Is it possible to completely remove the retention lock against a file or set of files?

Yes, it is possible to revert (remove) the retention lock against files in MTrees using governance mode. This is done with the following command:

# mtree retention-lock revert <path>

Once the retention lock against a file is removed, it can be modified or deleted as normal. If this command is run against a directory, all files within that directory and all subdirectories have their retention locks removed.

It is not possible to revert a retention lock against files in MTrees using compliance mode. If this is attempted, the following error is displayed:

# mtree retention-lock revert /data/col1/rl_test_comp/testfile
This operation is not allowed. Mtree is in retention-lock compliance mode.

What happens if a retention-locked file is attempted to be modified or removed?

Any attempt to modify or delete a retention-locked file causes a Permission denied error:

# echo " test2" >> /data/col1/rl_test/testfile
bash: testfile: Permission denied
# rm testfile
rm: remove write-protected regular file `testfile'? y
rm: cannot remove `testfile': Permission denied

The DDFS logs indicate that the operation has failed due to the file being retention locked:

06/07 07:06:59.756 (tid 0x2b5a77605d50): Atime of retention-lock file /data/col1/rl_test/testfile is not expired.
06/07 07:07:42.504 (tid 0x2b5a79111390): Atime of retention-lock file /data/col1/rl_test/testfile is not expired.

Is it possible to list all files which are retention locked?

Yes, this can be performed using the 'mtree retention-lock report generate retention-details' command:

mtree retention-lock report generate retention-details
mtrees {<mtree-list> | all}
[format {text | tsv | csv}]
[output-file <file-name>]
               Report detailed information of
               retention-lock files.

For example, do the following to list details of all locked files in the /data/col1/rl_test mtree:

# mtree retention-lock report generate retention-details mtrees /data/col1/jftest
Report generated on: Fri Jul  1 14:19:31 2016

Report for mtree: /data/col1/jftest
File Path        Mode        Size(Bytes)        Expiration Date
/data/col1/jftest/file1 governance 10521456 Sat Jul  2 22:35:48 2016
/data/col1/jftest/testdir/file2 governance 10521680 Sat Jul  2 22:35:42 2016
/data/col1/jftest/file3 governance 10521820 Sun Jul 10 22:36:09 2016
Total files: 3

Is it possible to completely disable the retention lock for a MTree after it is enabled?

Yes, for MTrees using governance mode this is performed using the 'mtree retention-lock disable mtree <mtree>' command. For example:

# mtree retention-lock disable mtree /data/col1/rl_test
Retention-lock feature is disabled (previously enabled) for mtree /data/col1/rl_test.

Once disabled, 'mtree list' indicates that retention lock was used against the MTree but has since been disabled:

# mtree list
Name                                Pre-Comp (GiB)   Status    Tenant-Unit
---------------------------------   --------------   -------   -----------
...
/data/col1/rl_test                             0.0   RW/RLGD   -
...

Can replication still be used against MTrees which have retention lock enabled?

Yes, retention locked MTrees or files can be replicated using various replication topologies:

• Directory replication:
  • Only supported for files using governance mode
  • Does not replicate minimum and maximum retention periods to the destination system
• MTree replication:
  • Can be used for either governance or compliance mode data
  • Does replicate minimum and maximum retention periods to the destination system
• Collection replication:
  • Can be used for either governance or compliance mode data
  • Does replicate minimum and maximum retention periods to the destination system

For retention locks to be preserved on destination systems:

• Both the source and destination systems must have corresponding retention lock licenses installed.
• If replicating an RLC-enabled MTree, then both source and destination DDs must have RLC configured or an error occurs:
  "A compliance retention-locked mtree cannot be replicated to a destination that is not enabled for compliance retention-lock."
• MTree replication contexts must have 'replication propagate-retention-lock' set to "Enabled."
• For files replicated by directory replication, the corresponding MTrees' minimum and maximum retention periods must be manually set on the destination system.

Are there any other restrictions when replicating retention locked files?

Yes, resyncs of replication contexts (that is, trying to reestablish a previously configured but broken context) where retention locked data is used to can fail.

• If MTree replication is used and the destination MTree contains retention-locked files that do not exist on the source, a resync fails.
• If directory replication is used and the destination has a retention lock that is enabled but the source does not, a resync fails.

MTree replication resyncs succeed in the following scenarios if the destination MTree does not contain retention-locked files not present on the source DD:

• The source MTree does not have retention lock enabled, but the destination does.
• The destination MTree does not have retention lock enabled, but the source does.

It is also not possible to enable retention lock compliance mode on a MTree that is already a member of a MTree replication context. In this scenario:

• The MTree replication context should be broken on both source and destination systems:

  # replication break mtree://<destination system>/data/col1/<mtree>

• Retention Lock Compliance (RLC) mode should be enabled on the source system:

  # mtree retention-lock enable mode compliance mtree <mtree>

• A new MTree replication context should be created on both source and destination systems:

  # replication add source mtree://<source system>/data/col1/<mtree> destination mtree://<destination system/data/col1/<mtree>

• The newly created replication context should be resynced on the source system:

  # replication resync mtree://<destination system>/data/col1/<mtree>

Can retention-locked files be fast copied?

Yes, files that are retention locked can be fast copied as normal. If the destination MTree holding the fast copy has retention lock enabled, the retention lock of the file is preserved against the fast copy. If the destination MTree does not have retention lock enabled, the fast copy is not retention locked.

Are there any other restrictions in system functionality when using retention lock?

The following commands are disallowed on systems using retention lock compliance mode:

# authorization policy set security-officer {enabled | disabled}
# filesys destroy
# user reset

Systems with RLC enabled cannot be booted to single user mode for recovery by technical support without a USB drive and physical access to the system.

The following commands are disallowed against MTrees using retention lock compliance mode:

# mtree delete <mtree>
# mtree retention-lock reset <min-retention-period period | max-retention-period period> mtree <mtree>
# mtree retention-lock disable mtree <mtree>
# mtree retention-lock revert

A provision has been made in DDOS 7.3 and later releases to allow deletion of RLC-enabled MTrees if:

• The DD is running DDOS 7.3 or later.
• The RLCE MTree to be deleted is empty (has zero files and directories).
• An administrator successfully passes security-officer authentication.

In systems configured with Long-Term Retention (Cloud Tier), similarly destructive commands may also be disallowed. For example:

# cloud unit del <cloud unit name>

Is the system clock important on retention lock enabled systems?

Yes, retention lock compliance-enabled systems enable an internal security clock to prevent malicious tampering with the system clock, as setting the clock forward could allow retention-locked files to be deleted early. The times of the security and system clocks are regularly compared. If there is an accumulated two-week skew between them in one calendar year, the DDFS is automatically disabled to prevent access to data on the DD. This can also happen if the system clock is suddenly modified and the time changes by more than two weeks.

How to reenable the DDFS in this scenario:

• Log in to the DD.
• Check that the system clock is set correctly.
• Enable the file system:

  # filesys enable

• When prompted, enter details of a user with the role of security to allow the security clock to be reset.

Can the system clock be synchronized with Active Directory on retention lock compliance-enabled systems?

No, when RLC is enabled, CIFS servers no longer synchronize the system time with Active Directory (AD). If there is a time difference of greater than five minutes between the system and AD, the CIFS server displays an error message when an AD user attempts to log in, or the system attempts to join an AD domain. Configure AD time with NTP to avoid this error.

This is in contrast to systems where retention lock compliance is not enabled but Active Directory is in use. In this situation, it is not recommended to enable NTP as this can cause time setting conflicts between Active Directory and NTP.

What steps can be taken if a DD using retention-locked files fills to capacity?

Assuming there is no cleanable space on the DD (clean has run but the system is full) its contents should be reviewed to determine if:

• There are any files that are not retention locked which can be removed.
• There are any files that are locked using governance mode which can have their locks reverted and be removed.

Once this is done, clean should be run again to physically free space on the system. If no physical data can be deleted from the system, additional physical storage should be added to the DD and the file system expanded (assuming the current DD or configuration supports increased storage).

If the only files on the system are locked using compliance mode, it is not possible to revert the locks and remove these files. As a result, space cannot be freed unless:

• The retention period is reached for some or all the files. After this point, they can be deleted and clean run as described above.
• The system is reinstalled from a USB drive, which causes loss of all data on the DD.
• More physical storage is added to the system as described above.