Data Domain: Retention Lock Compliance Enabled System Shows Alert, or FS May Be Disabled, with "Security Clock Drift is Detected" Message

When RLC is enabled, locked files cannot be deleted, or modified until their corresponding locks have been expired. For RL Governance systems, locks may be "reverted" if proper authentication is provided and the DD date and time is kept correct and not modified maliciously.

On RLC systems, the ability to manually change date and time is limited. Having NTP configured to a trusted time source before enabling RLC is mandatory. Even an NTP server may be compromised and changed to return invalid date and time to servers. This is done as a means to make retention lock files' locks expire early for an attacker to delete or modify existing data.

To partially protect against these possible attack vectors, DD systems with RL Compliance enabled keep close track of system date and time, and record every time jump which may occur. Once the cumulative time drift gets to a given amount, the system raises a warning for the administrator to check date and time. Eventually if clock skew and time drift go too high without changes by the administrator, the file system process is disabled as a safeguard.

An alert is posted when clock skew has gone more than 14 days accumulated clock drift (by default) presents the following:

# alerts show current
Id       Post Time                  Severity   Class        Object              Message
------   ------------------------   --------   ----------   -----------------   ------------------------------------------------------------------------------------------------------------------------------------------------------------------------
m0-95    Mon May  9 11:47:19 2022   CRITICAL   Filesystem                       EVT-FILESYS-00025: Security clock skew violation of 14 day 23 hr 22 min detected. Filesys will be disabled as skew exceeds the configured threshold 14 day 0 hr 0 min.
------   ------------------------   --------   ----------   -----------------   ------------------------------------------------------------------------------------------------------------------------------------------------------------------------

The file system is brought down automatically (log view of debug messages.engineering):

May  9 12:00:29 dd6900 ddfs[10015]: INFO: ddr_statelib: ddr_sd_proc_in_shutdown(DISABLED, "warm_startup: Security clock violation occurs.")
May  9 12:00:29 dd6900 ddfs[10015]: ERROR: warm_startup: Security clock violation occurs.
May  9 12:01:34 dd6900 sms: NOTICE: sm_fs_trigger_enable: filesys: changing state The filesystem is disabled and shutdown. [warm_startup: Security clock violation occurs.]  -> enable (1:1->2)

At any time, the status of the clock drift and skew may be checked from the DD CLI for any RL Compliance-enabled system:

# system retention-lock compliance status
System configuration state               : configured
Retention-lock compliance status         : enabled
Current system time                      : 2022/05/09 14:44:01
Last recorded system time for compliance     : 2022/05/06 17:18:45
Accumulated clock variance                   : 14 day 23 hr 22 min (since 2021/10/2 16:8:10)

RLC enabled systems monitor date and time for security reasons. Changes in date and time are cumulative. If past a given threshold, an alert is raised. If exceeding fourteen days consecutively, the file system is shut down for security.

Clock drift is not necessarily indicative of an attack on the DD to prematurely alter or delete RLC locked data. Usually, clock drift is the result of how DD OS monitors changes in time in current releases. For example, if the DD file system process goes down for more than 15 minutes, this is treated as (and accounted for) clock skew.

When time drift and clock skew alerts get reported, double check your configured time server to confirm it is not subjected to malicious changes. Also, use the command that is mentioned above (system retention-lock compliance status) to check for current skew and over time. At any time, the alert may be cleared using the UI or the CLI. For example, from the CLI, and for the alert presented at the top of this article, it is:

# alert clear alert-id m0-95

If file system process happens to already be down due to excessive skew and drift (greater than 14 days cumulative), see if there is recent or weird changes in date and time, or (as it is the most likely), file system process is down for periods longer than 15 minutes. If after all checks are completed and there is no perceived risk, clear the alert for clock drift and skew. Once cleared, run "filesys enable" command to bring the file system process up again.

If in doubt or need any help, contact Dell Technologies Customer Support.