Data Domain - DD Retention Lock causes Commvault backups error cannot set file access time
Summary: Commvault backups stall with errors 'Cannot set file access time' and 'Cannot mark file' when Data Domain retention lock is enabled.
Symptoms
From Commvault logs
egrep 'Cannot set the access time of the file' *
5912 16e0 03/14 13:14:28 39714925 89335656-66296274 [MEDIAFS ] RealMagneticFS Created <\\x.x.x.x\xxxxxxxxx_cvlt_hs_win_30days_repl\xxxxxx_Folder1\CV_MAGNETIC\V_24431218\CHUNK_xxxxx7588> file
5912 16e0 03/14 13:14:28 39714925 89335656-66296274 [DSBACKUP ] Archive File creation succeeded Type: 16 SubType: 131 Status: 0 EventErrorCode: 0
5912 16e0 03/14 13:14:28 39714925 89335656-66296274 [DSBACKUP ] Archive File creation succeeded Has been Sent
5912 19e8 03/14 13:14:28 39714925 89335656-66296274 [DSBACKUP ] Job Update Interval seconds [300]S
5912 c0c 03/14 13:14:28 39714925 89335656-66296274 [MEDIAFS ] SetFileValidData is not supported on this path. Code [87]. Current Offset [0x0000000000000000], Chunk increment size [134217728]
5912 33b4 03/14 13:14:30 39714925 89335656-66296274 [DM_RECEIVER] Recieved close for ARCHIVE FILE ID = [224722928]
5912 33b4 03/14 13:14:30 39714925 89335656-66296274 [DM_BASE ] Going to close chunk for archive File [224722928] The current size of the chunk is [217752447 ]
5912 33b4 03/14 13:14:30 39714925 89335656-66296274 [MEDIAFS ] RealMagneticFS::FinalizeChunkForBackup: Setting the end of file to offset [0x000000000D000000]
5912 33b4 03/14 13:14:30 39714925 89335656-66296274 [MEDIAFS ] Cannot set the access time of the file [\\x.x.x.x\xxxxxxxxx_cvlt_hs_win_30days_repl\xxxxx_Folder1\CV_MAGNETIC\V_24431218\CHUNK_569777588]. Time [133555148680000000], Error [5].
5912 33b4 03/14 13:14:30 39714925 89335656-66296274 [MEDIAFS ] Cannot mark the file [\\x.x.x.x\dc19900dd0100_cvlt_hs_win_30days_repl\xxxxxx_Folderx\CV_MAGNETIC\V_24431218\CHUNK_569777588] as read only.
5912 33b4 03/14 13:14:30 39714925 89335656-66296274 [MEDIAFS ] CloseSpecific Failed to close the Media Handle - VolId=24431218
5912 33b4 03/14 13:14:30 39714925 89335656-66296274 [DM_BASE ] Error handler: Setting JPR for [errorType - 1] [Error-code - 996] [Error-module - 218104071] [Event-ID - 1040187703] [Handler type - 218104061]
5912 33b4 03/14 13:14:30 39714925 89335656-66296274 [DM_BASE ] setting job pending reason [job-ID - 39714925] [job type - 1] [Argument count - 3]
5912 33b4 03/14 13:14:30 39714925 89335656-66296274 [DM_BASE ] **ERROR** Failed to close the Chunk on Media. ArchiveFile=224722928 ChunkId= 569777588
5912 33b4 03/14 13:14:30 39714925 89335656-66296274 [DM_RECEIVER] DataReceiver::Close: Error: DataWriter Close Failed. ArchFileId [224722928]
5912 33b4 03/14 13:14:30 39714925 89335656-66296274 [DSBACKUP ] Failed to Close Archive File
5912 33b4 03/14 13:14:30 39714925 89335656-66296274 [DSBACKUP ] ERROR! Failed to close Archive file Type: 16 SubType: 132 Status: 102 EventErrorCode: 117440531
5912 33b4 03/14 13:14:30 39714925 89335656-66296274 [DSBACKUP ] ERROR! Failed to close Archive file Has been Sent
5912 33b4 03/14 13:14:30 39714925 89335656-66296274 [DSBACKUP ] [error] occured [handled - yes]
From DD ddfs.info logs
First grep for the chunk in DDFS and then get the file handle.
grep "CHUNK_xxxxx7588" ddfs.info.7
03/13 22:26:36.837110 [7facfe393c50] FM fm_dm1_open:3690 - Create: /data/col1/xxxxxxxx_cvlt_win_30days_repl/xxxx_Folder1/CV_MAGNETIC/V_24429117/CHUNK_xxxxx7588, FH: ---->5700bc:0:xxx:0:xxx:xxxxx:0:<--------:0:(0:99436), tag: 0, cp_uuid = c85bf3507bxxxxxx:67d1086dbxxxxx
Grep for file handle.
grep "5700bc:xxxxxxxxxxxxxxx" ddfs.info.7
03/13 22:26:36.837110 [7facfe393c50] FM fm_dm1_open:3690 - Create: /data/col1/xxxxxxxx_cvlt_hs_win_30days_repl/xxxxxx_Folder1/CV_MAGNETIC/V_24429117/CHUNK_xxxxx7588, FH: 5700bc:0:xxxxx:0:xxxx:xxx:0:(0:99436), tag: 0, cp_uuid = c85bf3507xxxxx:67d1086dbxxxxx
03/13 22:27:52.381294 [7facfb801050] Attempt to set atime of file 5700bc:0:xx:x:xxxxx:xxxxx:0 to less than minimum retention period of mtree
Cause
Commvault is trying to set atime to a value less than (Current time) + (Retention-lock min-retention-period).
Resolution
Either lower the min retention period or set Commvault to set the atime to something later than the present value set.
Additional Information
https://community.commvault.com/self-hosted-q-a-2/difference-between-retention-lock-enabling-worm-compliance-lock-6540 [community.commvault.com] ">
https://documentation.commvault.com/2023e/expert/configuring_worm_storage_lock_and_compliance_lock.html [documentation.commvault.com] ">
Retention Lock, Compliance Lock, Software WORM, WORM Copy → All these refers to same CommVault software level WORM protection. It blocks from deleting jobs, clients, backup sets, or subclients with valid jobs and also prevents reducing retention periods.
Before version 11.30, use the Java Console to enable WORM at the storage policy copy level. Alternatively, use the Retention Lock workflow for all copies in a storage pool.
From version 11.30 onward, the option was renamed 'Compliance Lock' to avoid confusion with WORM storage and is available in Command Center and Java Console under storage pool properties. Select the toggle "Compliance Lock" on the storage pool to enable this.
WORM Storage → Refers to hardware level (storage vendor side) WORM protection. With this option data is locked at the storage side to prevent any direct deletions and modifications from the storage side.
We automatically enable "Compliance Lock" when WORM Storage is enabled to prevent retention changes to align with storage side lock.
Before, 11.30 → use "Enable WORM Storage" workflow to enable this.
From, 11.30 → we exposed this option in both Command Center and Java Console under storage pool properties. Select click "WORM Storage" on the storage pool to enable this.