Avamar: Data Domain Integration: After Upgrading Data Domain to 6.1.x, the Avamar Snapshots Stored on Data Domain Do Not Expire
Summary: After upgrading Data Domain to 6.1.x, the Avamar snapshots stored on Data Domain do not expire. No other changes were made to the environment.
Symptoms
The Data Domain shows that there are no expired snapshots. This can be verified by doing the following:
1. On the Avamar Utility Node, determine the MTree name by running the "avmaint hfscreate" command on the Avamar grid, and prepending the string "/data/col1/avamar-" as follows:
Example:
avmaint hfscreate
1501099628
The resulting MTree name: /data/col1/avamar-1501099628
2. From the Data Domain, using the following command, get a list of the snapshots associated with the Avamar MTree:
snapshot list mtree /data/col1/<avamar-mtree-name>
Example:
snapshot list mtree /data/col1/avamar-1501099628
Snapshot Information for MTree: /data/col1/avamar-1501099628
----------------------------------------------
Name Pre-Comp (GiB) Create Date Retain Until Status
----------------- -------------- ----------------- ------------ ------
cp.20170802130330 501241.1 Aug 2 2017 09:04
cp.20170802131127 501355.0 Aug 2 2017 09:11
cp.20170803120133 503440.7 Aug 3 2017 08:02
cp.20170803120726 503554.7 Aug 3 2017 08:07
cp.20170804120142 496207.0 Aug 4 2017 08:02
cp.20170804120836 496321.0 Aug 4 2017 08:09
cp.20170805130259 523295.5 Aug 5 2017 09:03
cp.20170805130955 523409.5 Aug 5 2017 09:10
cp.20170806130127 541524.5 Aug 6 2017 09:01
cp.20170806130719 541638.5 Aug 6 2017 09:07
cp.20170807130120 438037.9 Aug 7 2017 09:01
cp.20170807130712 438151.9 Aug 7 2017 09:07
----------------- -------------- ----------------- ------------ ------
Snapshot Summary
-------------------
Total: 12
Not expired: 12
Expired: 0
The following events can be seen in DDR maintenance logs from Avamar (/usr/local/avamar/var/ddrmaintlogs/ddrmaint.log*):
Aug 7 09:07:45 avamar ddrmaint.bin[122469]: Error: cp-delete::expire_checkpoint_snapshot - Failed to expire checkpoint: cp.20170806130719, ddr: test-dd.company.com, ddr-index: 1, DDR result code: 5075, desc: the user has insufficient access rights
Aug 7 09:07:45 avamar ddrmaint.bin[122469]: Error: <4740>Datadomain checkpoint delete operation failed.
...
Aug 6 09:07:18 avamar ddrmaint.bin[176316]: Info: Data Domain configured in Stand-Alone mode.
Aug 6 09:07:18 avamar ddrmaint.bin[176316]: Info: cp-delete::execute_delete_cp - Deleting DDR Checkpoint for dpnid:<dpnid> on ddr:test-dd.company.com cp: cp.20170805130259
Aug 6 09:07:18 avamar ddrmaint.bin[176316]: Info: Setting default storage unit to 'avamar-1501099628' for handle 1
Aug 6 09:07:18 avamar ddrmaint.bin[176316]: Warning: Calling DDR_EXPIRE_SNAPSHOT returned result code:5075 message:the user has insufficient access rights
Aug 6 09:07:18 avamar ddrmaint.bin[176316]: Error: cp-delete::expire_checkpoint_snapshot - Failed to expire checkpoint: cp.20170805130259, ddr: dd.emc.com, ddr-index: 1, DDR result code: 5075, desc: the user has insufficient access rights
Aug 6 09:07:18 avamar ddrmaint.bin[176316]: Error: <4740>Datadomain checkpoint delete operation failed.
Aug 6 09:07:18 avamar ddrmaint.bin[176316]: Info: ============================= cp-delete finished in 1 seconds
Aug 6 09:07:18 avamar ddrmaint.bin[176316]: Info: ============================= cp-delete cmd finished =============================
The following events can be seen in the ddfs.info file (/ddr/var/log/debug/ddfs.info*) on the Data Domain:
08/07 08:01:06.322 (tid 0x7fe7037cd930): ddboost-<test-avamar.company.com-37933>: ddboost_api ERROR: ddp_snapshot_expire() failed for SUName avamar-1501099628, snapshot: cp.20170802130330, retention: -1, flags: 0 Err: 5075-Update retention of snapshot [cp.20170802130330] on Storage Unit [avamar-1501099628(nfs: Operation not permitted)
10/11 11:19:55.468 (tid 0x7f27454a63f0): ddboost-<test-avamar.company.com-60566>: test-avamar.company.com Local Time: Wed Oct 11 11:19:55 2017
10/11 11:19:55.471 (tid 0x7f2cd150e830): OST_FH_PERM FAIL on storage-unit=avamar-1501099628 op=NFSPROC3_DDP_LOOKUP[27] client=test-avamar.company.com uid=500:uid or gid does not match
09/26 09:17:46.762 (tid 0x7f60d7a08410): ddboost-<test-avamar.company.com-37933>: ddboost_api ERROR: ddp_snapshot_list() failed, Err: 5009-Get snapshot list on Storage Unit [avamar-1501099628] failed (nfs: I/O error)
09/26 09:18:46.472 (tid 0x7f7e1ab92d50): ddboost-<test-avamar.company.com-37933>: ddboost_api ERROR: ddp_snapshot_list() failed, Err: 5009-Get snapshot list on Storage UnitCause
The DD Boost User for a particular storage unit was created in an earlier version of DDOS with an operating system group of users, rather than admin. This would have occurred when Avamar first connected to the Data Domain and created the storage unit, setting the permissions similar to those below:
<ddboostuser> users 430 Apr 26 2016 <avamar-mtree-name>
From DDOS 6.1.x, some storage-unit operations such as deleting a storage unit or expiring snapshots, require that the owner of the storage unit be part of the admin group (instead of the users group).
When this is not the case, these operations fail, and every day, two new Data Domain snapshots, corresponding to Avamar checkpoints, are left without expiring.
The system is not affected if the DD Boost User associated with the storage unit was changed to an admin role afterwards.
user group and permissions.
Resolution
Check the registry settings by doing the following on the Data Domain:
1. Use the following command to retrieve the Avamar DD Boost User
ddboost storage-unit show
Name Pre-Comp (GiB) Status User Report Physical
Size (MiB)
----------------- -------------- ------ --------------- ---------------
avamar-1501099628 10808220.0 RW ddboost-avamar -
d025 457051.7 RW ddboost-avamar -
rman_dd 240902.7 RW ddboost-rman -
mssql 142474.8 RW ddboost-avamar -
----------------- -------------- ------ --------------- ---------------
From this output, the Avamar DD Boost User is ddboost-avamar.
2. Retrieve and make note of the user id (UID) of the DD Boost User:
user show list
Example:
User list from node "localhost".
Name Uid Role Last Login From Last Login Time Status Disable Date
-------- --- ----- --------------- ------------------------ ------- ------------
sysadmin 100 admin 10.x.x.x Tue Oct 10 14:44:52 2017 enabled never
ddboost-avamar 500 admin 10.x.x.x Wed Oct 11 11:07:49 2017 enabled never
-------- --- ----- --------------- ------------------------ ------- ------------
2 users found.
In this instance, the UID is 500 (but this can vary).
3. Review the registry settings by doing one of the following:
- Searching the
autosupport- Access
/ddvar/support(using aCIFSshare orNFSexport) or see article Data Domain: How to Gather Autosupports
- Access
- Using the CLI command:
reg show protocol.ost
users group is 100 (not to be confused with the UID 100 for sysadmin)
An example where the registry settings are incorrect (group id is 100):
reg show protocol.ost
... protocol.ost.stu_user.<avamar-mtree-name>= 500:100 protocol.ost.uid500:100 = <ddboostusername> protocol.ost.user.<ddboostusername> = 500:100 ...
If the settings are incorrect (as in the example above where the GID is set to 100), Create a Service Request referencing this knowledge article. The issue can then be reviewed, verified, and resolved.