Avamar: Data Domain Integration: Maintenance failures due to an empty ddr_info file
Summary: Avamar Maintenance failures due to an empty ddr_info file
Symptoms
1. All Avamar maintenance activities (checkpoint, hfscheck, and garbage collection) fail with MSG_ERR_DDR_ERROR:
This output can be verified using the status.dpn command:
Checkpoint failed with result MSG_ERR_DDR_ERROR : cp.20140626152425 started Thu Jun 26 11:24:26 2014 ended Thu Jun 26 11:26:18 2014, completed 40887 of 40888 stripes
Last GC: finished Thu Jun 26 08:01:44 2014 after 46m 42s >> recovered 49.44 GB (OK)
Last hfscheck failed with result MSG_ERR_DDR_ERROR : started Thu Jun 26 10:12:15 2014 ended Thu Jun 26 11:24:23 2014, nodes queried/replied/total(3/3/3)
2. The following entries are present in the /usr/local/avamar/var/ddrmaintlogs/ddrmaint.log (on the Avamar Utility Node):
Error: cplist::enumerate_ddrconfig Error reading ddr_info: Unable to parse ddr_info <avamar> tag not found
Info: cplist::open_all_ddrs: ddr_info file from the persistent store has no ddr_config entries or all entries were skipped.
Error: cplist::body - ddr_info file from the persistent store has no ddr_config entries
Error: <4750>Datadomain get checkpoint list operation failed.
ddr_info" message. All other messages can be seen in relation to other issues.
3. Checkpoints are listed as 'invalid' from the cplist output:
cplist
cplist: ERROR: ddrmaint: <4750>Datadomain get checkpoint list operation failed.
cp.20130104160214 Fri Jan 4 11:02:14 2013 invalid --- --- nodes 1/1 stripes 10859
cp.20130104163558 Fri Jan 4 11:35:58 2013 invalid --- --- nodes 1/1 stripes 10859
4. The ddr_info file on the utility node is emptied out and has zero size:
ls -l /usr/local/avamar/var/ddr_info
-rw-r--r-- 1 admin admin 0 Apr 2 10:21
Cause
Avamar maintenance activities read the /usr/local/avamar/var/ddr_info file located on the Avamar Utility Node. If this file does not contain the same contents as that of the persistent store, maintenance fails.
The same file is read and modified while the Data Domain (DD) is edited using Avamar UI.
Resolution
Engage Avamar Technical Support to further troubleshoot this issue by creating a Service Request, referencing this knowledge article.