Data Domain - DD Retention lock and Commvault backups waiting to complete with error "Cannot set the access time of the file"

From Commvault logs 

egrep 'Cannot set the access time of the file' *

5912  16e0  03/14 13:14:28 39714925 89335656-66296274 [MEDIAFS    ] RealMagneticFS Created <\\x.x.x.x\xxxxxxxxx_cvlt_hs_win_30days_repl\xxxxxx_Folder1\CV_MAGNETIC\V_24431218\CHUNK_xxxxx7588> file
5912  16e0  03/14 13:14:28 39714925 89335656-66296274 [DSBACKUP   ] Archive File creation succeeded Type: 16 SubType: 131 Status: 0 EventErrorCode: 0
5912  16e0  03/14 13:14:28 39714925 89335656-66296274 [DSBACKUP   ] Archive File creation succeeded Has been Sent
5912  19e8  03/14 13:14:28 39714925 89335656-66296274 [DSBACKUP   ] Job Update Interval seconds   [300]S
5912  c0c   03/14 13:14:28 39714925 89335656-66296274 [MEDIAFS    ] SetFileValidData is not supported on this path. Code [87]. Current Offset [0x0000000000000000], Chunk increment size [134217728]
5912  33b4  03/14 13:14:30 39714925 89335656-66296274 [DM_RECEIVER] Recieved close for ARCHIVE FILE ID = [224722928]
5912  33b4  03/14 13:14:30 39714925 89335656-66296274 [DM_BASE    ] Going to close chunk for archive File [224722928] The current size of the chunk is [217752447 ]
5912  33b4  03/14 13:14:30 39714925 89335656-66296274 [MEDIAFS    ] RealMagneticFS::FinalizeChunkForBackup: Setting the end of file to offset [0x000000000D000000]
5912  33b4  03/14 13:14:30 39714925 89335656-66296274 [MEDIAFS    ] Cannot set the access time of the file [\\x.x.x.x\xxxxxxxxx_cvlt_hs_win_30days_repl\xxxxx_Folder1\CV_MAGNETIC\V_24431218\CHUNK_569777588]. Time [133555148680000000], Error [5].
5912  33b4  03/14 13:14:30 39714925 89335656-66296274 [MEDIAFS    ] Cannot mark the file [\\x.x.x.x\dc19900dd0100_cvlt_hs_win_30days_repl\xxxxxx_Folderx\CV_MAGNETIC\V_24431218\CHUNK_569777588] as read only.
5912  33b4  03/14 13:14:30 39714925 89335656-66296274 [MEDIAFS    ] CloseSpecific Failed to close the Media Handle - VolId=24431218
5912  33b4  03/14 13:14:30 39714925 89335656-66296274 [DM_BASE    ] Error handler: Setting JPR for [errorType - 1] [Error-code - 996] [Error-module - 218104071] [Event-ID - 1040187703] [Handler type - 218104061]
5912  33b4  03/14 13:14:30 39714925 89335656-66296274 [DM_BASE    ] setting job pending reason [job-ID - 39714925] [job type - 1] [Argument count - 3]
5912  33b4  03/14 13:14:30 39714925 89335656-66296274 [DM_BASE    ] **ERROR** Failed to close the Chunk on Media. ArchiveFile=224722928 ChunkId= 569777588
5912  33b4  03/14 13:14:30 39714925 89335656-66296274 [DM_RECEIVER] DataReceiver::Close: Error: DataWriter Close Failed. ArchFileId [224722928]
5912  33b4  03/14 13:14:30 39714925 89335656-66296274 [DSBACKUP   ] Failed to Close Archive File
5912  33b4  03/14 13:14:30 39714925 89335656-66296274 [DSBACKUP   ] ERROR! Failed to close Archive file Type: 16 SubType: 132 Status: 102 EventErrorCode: 117440531
5912  33b4  03/14 13:14:30 39714925 89335656-66296274 [DSBACKUP   ] ERROR! Failed to close Archive file Has been Sent
5912  33b4  03/14 13:14:30 39714925 89335656-66296274 [DSBACKUP   ] [error] occured [handled - yes]

From DD ddfs.info logs 

First grep for the chunk in DDFS and then get the file handle. 
grep "CHUNK_xxxxx7588" ddfs.info.7
03/13 22:26:36.837110 [7facfe393c50] FM fm_dm1_open:3690 - Create: /data/col1/xxxxxxxx_cvlt_win_30days_repl/xxxx_Folder1/CV_MAGNETIC/V_24429117/CHUNK_xxxxx7588, FH: ---->5700bc:0:xxx:0:xxx:xxxxx:0:<--------:0:(0:99436), tag: 0, cp_uuid = c85bf3507bxxxxxx:67d1086dbxxxxx


Grep for file handle. 
grep "5700bc:xxxxxxxxxxxxxxx" ddfs.info.7
03/13 22:26:36.837110 [7facfe393c50] FM fm_dm1_open:3690 - Create: /data/col1/xxxxxxxx_cvlt_hs_win_30days_repl/xxxxxx_Folder1/CV_MAGNETIC/V_24429117/CHUNK_xxxxx7588, FH: 5700bc:0:xxxxx:0:xxxx:xxx:0:(0:99436), tag: 0, cp_uuid = c85bf3507xxxxx:67d1086dbxxxxx
03/13 22:27:52.381294 [7facfb801050] Attempt to set atime of file 5700bc:0:xx:x:xxxxx:xxxxx:0 to less than minimum retention period of mtree

Commvault is trying to set atime to a value less than (Current time) + (Retention-lock min-retention-period).
 

Either lower the min retention period or set Commvault to set the atime to something later than the present value set.
 

">

">

Retention Lock, Compliance Lock, Software WORM, WORM Copy → All these refers to same CommVault software level WORM protection. It prevents users from deleting jobs, client, backups set, subclient…and so on, data bearing entities when they have valid jobs, and also it prevents reducing retention. 
Before 11.30 → you can use either Java Console to enable this at storage policy copy level by selecting WORM copy option or use Enable Retention Lock workflow to enable WORM copy option on all copies of a selected storage pool.
11.30 onwards → we renamed this option as "Compliance Lock" to avoid confusion with "WORM storage" and exposed it in both Command Center and Java Console under storage pool properties. Simply select the toggle "Compliance Lock" on the storage pool to enable this. 
 
WORM Storage → Refers to hardware level (storage vendor side) WORM protection. With this option data is locked at the storage side to prevent any direct deletions and/or modifications from the storage side.
We automatically enable "Compliance Lock" when WORM Storage is enabled to prevent retention changes to align with storage side lock. 
Before 11.30 → use "Enable WORM Storage" workflow to enable this.
From 11.30 → we exposed this option in both Command Center and Java Console under storage pool properties. Simply select toggle "WORM Storage" on the storage pool to enable this.