PowerProtect Data Manager: Backup with Retention Lock Mode enabled failed due to a time difference between Data Domain and PPDM

PPDM backup with Retention Lock Mode enabled failed with below warning in vmdm.log. "Failure while trying to apply retention lock to discovered copy with ID: 2bc30abb-ae95-578d-a63a-1c053970f13e, Detail: The retention lock of Storage Unit name: ppdm-stu, Id: 14a3a424-f3da-4cc6-a592-da84524b16a1 is not enabled. Cannot set retention lock on copy Id: 2b5e8f10-7e72-5097-86b3-dce5b8bb66c3., Remedy :Review the storage connection or storage unit configuration."

vmdm.log: 2024-12-09T02:22:52.870Z INFO [] [vpe-completion-3] [00000000-0000-4000-a000-000000000000][][][TRACE_ID:bf0cde6dc47dc8e0;JOB_ID:851517b98b0b69a4][] [c.e.b.d.d.DDBoostAdapter.connect(386)] -  Connected to PowerProtect DD System, hostname: 10.0.0.1, connDesc: -1383149567, instanceDesc: 816500736 ddosVersion: Data Domain OS 7.10.1.20-1090468  ddsModel: DD6400-MC
vmdm.log: 2024-12-09T02:22:52.874Z INFO [] [vpe-completion-3] [00000000-0000-4000-a000-000000000000][][][TRACE_ID:bf0cde6dc47dc8e0;JOB_ID:851517b98b0b69a4][] [c.e.b.d.s.r.DataDomainRetentionLockServiceImpl.setRetentionTime(120)] -  Successfully connecting to PowerProtect DD System with connectionDescription: -1383149567
vmdm.log: 2024-12-09T02:22:52.874Z INFO [] [vpe-completion-3] [00000000-0000-4000-a000-000000000000][][][TRACE_ID:bf0cde6dc47dc8e0;JOB_ID:851517b98b0b69a4][] [c.e.b.d.s.r.DataDomainRetentionLockServiceImpl.retentionLockAllFilesInCopiesOnDataDomain(171)] -  Starting to set retention lock copies for copy set on PowerProtect DD System: copySetId 2bc30abb-ae95-578d-a63a-1c053970f13e, retention time 2024-12-12T00:00:00Z
vmdm.log: 2024-12-09T02:22:52.878Z INFO [] [vpe-completion-3] [00000000-0000-4000-a000-000000000000][][][TRACE_ID:bf0cde6dc47dc8e0;JOB_ID:851517b98b0b69a4][] [ddboost.log(41)] -  severity:3 msg:[13563] [140001547364096] ddp_disconnect: Attempting to disconnect for connection descriptor:  connd[-1383149567]
......
vmdm.log: 2024-12-09T02:22:52.878Z INFO [] [vpe-completion-3] [00000000-0000-4000-a000-000000000000][][][TRACE_ID:bf0cde6dc47dc8e0;JOB_ID:851517b98b0b69a4][] [ddboost.log(41)] -  severity:3 msg:[13563] [140001547364096] clnt_async_destroy: Source IP:port(10.0.0.2:34224)
vmdm.log: 2024-12-09T02:22:52.878Z INFO [] [vpe-completion-3] [00000000-0000-4000-a000-000000000000][][][TRACE_ID:bf0cde6dc47dc8e0;JOB_ID:851517b98b0b69a4][] [ddboost.log(41)] -  severity:3 msg:[13563] [140001547364096] clnt_async_destroy: RPC_CANTRECV will fail all the pending jobs and close socket
vmdm.log: 2024-12-09T02:22:52.878Z INFO [] [vpe-completion-3] [00000000-0000-4000-a000-000000000000][][][TRACE_ID:bf0cde6dc47dc8e0;JOB_ID:851517b98b0b69a4][] [ddboost.log(41)] -  severity:3 msg:[13563] [140001547364096] ddp_disconnect: ddcl_disconnect succeeded for all physical connections
vmdm.log: 2024-12-09T02:22:52.879Z WARN [] [vpe-completion-3] [00000000-0000-4000-a000-000000000000][][][TRACE_ID:bf0cde6dc47dc8e0;JOB_ID:851517b98b0b69a4][] [c.e.b.v.v.a.VPEProtectOp.completeSessionSuccess(858)] -  Failure while trying to apply retention lock to discovered copy with ID: 2bc30abb-ae95-578d-a63a-1c053970f13e, Detail: The retention lock of Storage Unit name: ppdm-stu, Id: 14a3a424-f3da-4cc6-a592-da84524b16a1 is not enabled. Cannot set retention lock on copy Id: 2b5e8f10-7e72-5097-86b3-dce5b8bb66c3., Remedy :Review the storage connection or storage unit configuration.
com.emc.brs.vmdm.error.RetentionLockException: Unable to apply Retention Lock to copy 'PCS-a232105f-9b71-46ac-a4a6-1caf3ce14f19-50246572-b7f4-cc2b-b986-e3234dc5bbc7', because of invalid target source 'ppdm-stu'.

Run 'mtree list' on Data Domain, the mtree indeed has Retention-Lock Governance Enabled.

sysadmin@ddxxxx# mtree list
Name                                Pre-Comp (GiB)   Status    Tenant-Unit
---------------------------------   --------------   -------   -----------
...
/data/col1/ppdm-stu                 0.0		     RW/RLGE   -
...
---------------------------------   --------------   -------   -----------
...
 RLGE : Retention-Lock Governance Enabled
 RLGD : Retention-Lock Governance Disabled
 RLCE : Retention-Lock Compliance Enabled

The root cause of the issue is that the time difference between Data Domain and PPDM is too large. In this case, the time on Data Domain is 15 minutes ahead of the time on PPDM.

The connectionDescription is set to a negative value "-1383149567" due to this time difference, causing the ddp_disconnect.

Configure NTP on both Data Domain and PPDM server to synchronize the time.

Rerun the backup with Retention Lock Mode enabled.