Data Domain - DD Retention Lock 導致 Commvault 備份錯誤,無法設定檔案存取時間
Summary: 啟用 Data Domain 保留鎖定時,Commvault 備份停止,並出現「無法設定檔案存取時間」和「無法標記檔案」錯誤。
Symptoms
從 Commvault 記錄
egrep 'Cannot set the access time of the file' *
5912 16e0 03/14 13:14:28 39714925 89335656-66296274 [MEDIAFS ] RealMagneticFS Created <\\x.x.x.x\xxxxxxxxx_cvlt_hs_win_30days_repl\xxxxxx_Folder1\CV_MAGNETIC\V_24431218\CHUNK_xxxxx7588> file
5912 16e0 03/14 13:14:28 39714925 89335656-66296274 [DSBACKUP ] Archive File creation succeeded Type: 16 SubType: 131 Status: 0 EventErrorCode: 0
5912 16e0 03/14 13:14:28 39714925 89335656-66296274 [DSBACKUP ] Archive File creation succeeded Has been Sent
5912 19e8 03/14 13:14:28 39714925 89335656-66296274 [DSBACKUP ] Job Update Interval seconds [300]S
5912 c0c 03/14 13:14:28 39714925 89335656-66296274 [MEDIAFS ] SetFileValidData is not supported on this path. Code [87]. Current Offset [0x0000000000000000], Chunk increment size [134217728]
5912 33b4 03/14 13:14:30 39714925 89335656-66296274 [DM_RECEIVER] Recieved close for ARCHIVE FILE ID = [224722928]
5912 33b4 03/14 13:14:30 39714925 89335656-66296274 [DM_BASE ] Going to close chunk for archive File [224722928] The current size of the chunk is [217752447 ]
5912 33b4 03/14 13:14:30 39714925 89335656-66296274 [MEDIAFS ] RealMagneticFS::FinalizeChunkForBackup: Setting the end of file to offset [0x000000000D000000]
5912 33b4 03/14 13:14:30 39714925 89335656-66296274 [MEDIAFS ] Cannot set the access time of the file [\\x.x.x.x\xxxxxxxxx_cvlt_hs_win_30days_repl\xxxxx_Folder1\CV_MAGNETIC\V_24431218\CHUNK_569777588]. Time [133555148680000000], Error [5].
5912 33b4 03/14 13:14:30 39714925 89335656-66296274 [MEDIAFS ] Cannot mark the file [\\x.x.x.x\dc19900dd0100_cvlt_hs_win_30days_repl\xxxxxx_Folderx\CV_MAGNETIC\V_24431218\CHUNK_569777588] as read only.
5912 33b4 03/14 13:14:30 39714925 89335656-66296274 [MEDIAFS ] CloseSpecific Failed to close the Media Handle - VolId=24431218
5912 33b4 03/14 13:14:30 39714925 89335656-66296274 [DM_BASE ] Error handler: Setting JPR for [errorType - 1] [Error-code - 996] [Error-module - 218104071] [Event-ID - 1040187703] [Handler type - 218104061]
5912 33b4 03/14 13:14:30 39714925 89335656-66296274 [DM_BASE ] setting job pending reason [job-ID - 39714925] [job type - 1] [Argument count - 3]
5912 33b4 03/14 13:14:30 39714925 89335656-66296274 [DM_BASE ] **ERROR** Failed to close the Chunk on Media. ArchiveFile=224722928 ChunkId= 569777588
5912 33b4 03/14 13:14:30 39714925 89335656-66296274 [DM_RECEIVER] DataReceiver::Close: Error: DataWriter Close Failed. ArchFileId [224722928]
5912 33b4 03/14 13:14:30 39714925 89335656-66296274 [DSBACKUP ] Failed to Close Archive File
5912 33b4 03/14 13:14:30 39714925 89335656-66296274 [DSBACKUP ] ERROR! Failed to close Archive file Type: 16 SubType: 132 Status: 102 EventErrorCode: 117440531
5912 33b4 03/14 13:14:30 39714925 89335656-66296274 [DSBACKUP ] ERROR! Failed to close Archive file Has been Sent
5912 33b4 03/14 13:14:30 39714925 89335656-66296274 [DSBACKUP ] [error] occured [handled - yes]
從 DD ddfs.info 記錄
First grep for the chunk in DDFS and then get the file handle.
grep "CHUNK_xxxxx7588" ddfs.info.7
03/13 22:26:36.837110 [7facfe393c50] FM fm_dm1_open:3690 - Create: /data/col1/xxxxxxxx_cvlt_win_30days_repl/xxxx_Folder1/CV_MAGNETIC/V_24429117/CHUNK_xxxxx7588, FH: ---->5700bc:0:xxx:0:xxx:xxxxx:0:<--------:0:(0:99436), tag: 0, cp_uuid = c85bf3507bxxxxxx:67d1086dbxxxxx
Grep for file handle.
grep "5700bc:xxxxxxxxxxxxxxx" ddfs.info.7
03/13 22:26:36.837110 [7facfe393c50] FM fm_dm1_open:3690 - Create: /data/col1/xxxxxxxx_cvlt_hs_win_30days_repl/xxxxxx_Folder1/CV_MAGNETIC/V_24429117/CHUNK_xxxxx7588, FH: 5700bc:0:xxxxx:0:xxxx:xxx:0:(0:99436), tag: 0, cp_uuid = c85bf3507xxxxx:67d1086dbxxxxx
03/13 22:27:52.381294 [7facfb801050] Attempt to set atime of file 5700bc:0:xx:x:xxxxx:xxxxx:0 to less than minimum retention period of mtree
Cause
Commvault 嘗試將 atime 設定為小於 (目前時間) + (Retention-lock min-retention-period)。
Resolution
縮短最短保留期間,或設置 Commvault 將 atime 設置為晚於當前值設置的某個值。
Additional Information
https://community.commvault.com/self-hosted-q-a-2/difference-between-retention-lock-enabling-worm-compliance-lock-6540 [community.commvault.com] “>
https://documentation.commvault.com/2023e/expert/configuring_worm_storage_lock_and_compliance_lock.html [documentation.commvault.com] ”>
保留鎖定, 法規遵循鎖定, 軟體 WORM, WORM 複製→ 所有這些都是指相同的 CommVault 軟體層級 WORM 保護。它會阻止刪除具有有效作業的作業、用戶端、備份集或子用戶端,還會阻止縮短保留期。
在版本 11.30 之前,使用 Java 控制台在存儲策略複製級別啟用 WORM。或者,請對儲存集區中的所有複本使用保留鎖定工作流程。
從版本 11.30 開始,該選項被重命名為”Compliance Lock以避免與 WORM 儲存混淆,且可在 Command Center 和 Java 主控台的儲存集區屬性下取得。選取切換開關 "Compliance Lock" 以啟用此功能。
WORM 儲存→指硬體層級 (儲存裝置廠商端) WORM 保護。使用此選項,數據將鎖定在存儲端,以防止存儲端直接刪除和修改。
我們自動啟用 "Compliance Lock" 啟用 WORM 儲存裝置以防止保留變更與儲存側鎖定一致時。
之前,11.30 →使用 "Enable WORM Storage" 工作流以啟用此功能。
從 11.30 →,我們在 Command Center 和 Java Console 的儲存集區屬性下公開了此選項。選取按一下儲存集區上的「WORM 儲存」以啟用此功能。