Avamar: Data Domain Integration: Cloud Tier - Marked backups are not going to cloud due to unconfigured schedule
Summary: The backups were marked but no files were found in the cloud.
Symptoms
On an Avamar grid integrated with Data Domain, after Cloud Tiering is configured the backups are marked, but remain on the Data Domain while the Cloud location stays empty.
Backups are marked but show no indication of being moved to the Cloud.
Avamar:
admin@AVE01:~/#: mccli backup show --name=/clients/10.0.10.10/VirtualMachines/VMTEST01
0,23000,CLI command completed successfully.
Created LabelNum Size Retention Hostname Location Consistent Level Tier
------------------- -------- ------- --------- ---------------- -------- ---------------- ------
2019-11-17 06:08 PM 130 86.4 GB DW AVE01.test.local Local app Active
2019-11-16 06:08 PM 129 86.4 GB D AVE01.test.local Local app Active
2019-11-15 06:08 PM 128 86.4 GB D AVE01.test.local Local app Active
--------------- snip ---------------
2019-10-23 07:08 PM 105 86.4 GB D AVE01.test.local Local app Active
2019-10-22 07:07 PM 104 86.4 GB D AVE01.test.local Local app Active
2019-10-21 07:07 PM 103 86.4 GB DW AVE01.test.local Local app Active
2019-10-17 07:08 PM 102 86.4 GB D AVE01.test.local Local app Marked
2019-10-16 07:07 PM 101 86.4 GB D AVE01.test.local Local app Marked
2019-10-15 07:09 PM 100 86.4 GB D AVE01.test.local Local app Marked
--------------- snip ---------------
2019-08-22 07:07 PM 50 86.4 GB D AVE01.test.local Local app Marked
2019-08-21 07:06 PM 49 86.4 GB D AVE01.test.local Local app Marked
2019-08-20 07:08 PM 48 86.4 GB D AVE01.test.local Local app Marked
Data Domain:
The "data-movement show history" command shows that the latest activity happened some time ago:
sysadmin@dd3300# data-movement show history
Start Time End Time Bytes Moved Bytes Moved Files Moved Elapsed Time State
(Pre-comp) (Post-comp) (hh:mm:ss)
----------------- ----------------- ----------- ----------- ----------- --------------- ---------
Sep 24 2019 16:55 Sep 24 2019 16:55 0 0 0 0 days 00:00:00 Completed
Sep 24 2019 16:53 Sep 24 2019 16:53 0 0 0 0 days 00:00:00 Completed
----------------- ----------------- ----------- ----------- ----------- --------------- ---------
Cause
The data-movement schedule on Data Domain was not configured, so the data did not get migrated to the cloud unit.
This can be confirmed by running the following commands on the Data Domain:
sysadmin@dd3300# data-movement schedule show
Data-movement is not scheduled.
sysadmin@dd3300# data-movement status
Data-movement to cloud tier:
----------------------------
Data-movement was started on Sep 24 2019 16:55 and completed on Sep 24 2019 16:55
Moved (post-comp): None, (pre-comp): None,
Files inspected: 0, Files eligible: 0, Files moved: 0, Files failed: 0
Data-movement recall:
---------------------
No recall operations found.
Use 'path all' option to list all recall jobs (up to the maximum supported limit).
Resolution
Configure the data-movement schedule.
1. Set the schedule: (In this example, it is set up to run every day at 8PM):
sysadmin@dd3300# data-movement schedule set to-tier cloud days sun-sat time 20:00
Data-movement schedule has been set.
Data-movement is scheduled to run on day(s) "sun,mon,tue,wed,thu,fri,sat" at "20:00" hrs.
2. If the customer allows, start the data movement (otherwise wait for the schedule):
sysadmin@dd3300# data-movement start
Data-movement started.
Run "data-movement watch" to monitor progress.
As stated in the output above, If started manually, the "data-movement watch" command can be run to monitor progress. The commands below provide additional information.
After a few minutes, check the status:
sysadmin@dd3300# data-movement status
data-movement status
Data-movement to cloud tier:
----------------------------
Data-movement:
3% complete; Elapsed time: 0:01:31
Moved (post-comp): 308.88 KiB, (pre-comp): 717.19 MiB,
Files inspected: 251, Files eligible: 23, Files moved: 0, Files failed: 0
Data-movement recall:
---------------------
No recall operations found.
Use 'path all' option to list all recall jobs (up to the maximum supported limit).
The following command provides more details:
sysadmin@dd3300# data-movement status to-tier cloud detailed
Data-movement to cloud tier:
----------------------------
Data-movement:
3% complete; Elapsed time: 0:25:42
Moved (post-comp): 9.06 MiB, (pre-comp): 3.54 GiB,
Files inspected: 253, Files eligible: 25, Files moved: 0, Files failed: 0
Data-movement status for MTrees:
--------------------------------
MTree Files Files Files Files Bytes Moved Bytes Moved Destination
Inspected Eligible Moved Failed (Pre-comp) (Post-comp) Cloud Unit
----------------------------- --------- -------- ----- ------ ----------- ----------- -----------
/data/col1/avamar-1562000000* 253 25 0 0 86.42 GiB 9.06 MiB Avamar
----------------------------- --------- -------- ----- ------ ----------- ----------- -----------
(*) Data-movement is in progress for the marked MTrees.
Files currently being moved:
----------------------------
Path Name File Size Logical Destination Elapsed Time
Bytes Moved Cloud Unit hh:mm:ss
----------------------------------------------------------------------------------------------------------------------------------------------------- --------- ----------- ----------- ------------
/data/col1/avamar-1562000000/cur/DELETED/1574121600/1e428c0110b9162726f5217463a247c4728985d8/1D556114FA39B9C/251A33C32514398ED2325F2F5B304D4A9EBC936A 1.61 KiB 583.00 B Avamar 00:04:36
/data/col1/avamar-1562000000/cur/DELETED/1574121600/1e428c0110b9162726f5217463a247c4728985d8/1D556114FA39B9C/B6FE71BED477868E7EE213BD9E0FAEFEDE55468B 6.39 GiB 574.52 MiB Avamar 00:04:36
/data/col1/avamar-1562000000/cur/DELETED/1574121600/1e428c0110b9162726f5217463a247c4728985d8/1D556114FA39B9C/D56D73E317F1F82912DE4F4B8364E2AC353F6666 80.01 GiB 3.02 GiB Avamar 00:24:26
/data/col1/avamar-1562000000/cur/DELETED/1574121600/1e428c0110b9162726f5217463a247c4728985d8/1D556114FA39B9C/7605BFA0D16EA64711A0D54A5145A7B4DB5A1CE1 2.43 KiB 1.39 KiB Avamar 00:15:07
----------------------------------------------------------------------------------------------------------------------------------------------------- --------- ----------- ----------- ------------
3. After some hours, or any time after a scheduled data movement policy, recheck the activity status:
sysadmin@dd3300# data-movement status
Data-movement to cloud tier:
----------------------------
Data-movement:
3% complete; Elapsed time: 1:17:06
Moved (post-comp): 813.55 MiB, (pre-comp): 13.84 GiB,
Files inspected: 259, Files eligible: 31, Files moved: 11, Files failed: 0
Data-movement recall:
---------------------
No recall operations found.
Use 'path all' option to list all recall jobs (up to the maximum supported limit).
The following command provides more details:
sysadmin@dd3300# data-movement status to-tier cloud detailed
Data-movement to cloud tier:
----------------------------
Data-movement:
3% complete; Elapsed time: 1:17:16
Moved (post-comp): 813.55 MiB, (pre-comp): 13.84 GiB,
Files inspected: 259, Files eligible: 31, Files moved: 11, Files failed: 0
Data-movement status for MTrees:
--------------------------------
MTree Files Files Files Files Bytes Moved Bytes Moved Destination
Inspected Eligible Moved Failed (Pre-comp) (Post-comp) Cloud Unit
----------------------------- --------- -------- ----- ------ ----------- ----------- -----------
/data/col1/avamar-1562000000* 262 34 14 0 86.42 GiB 813.55 MiB Avamar
----------------------------- --------- -------- ----- ------ ----------- ----------- -----------
(*) Data-movement is in progress for the marked MTrees.
Files currently being moved:
----------------------------
Path Name File Size Logical Destination Elapsed Time
Bytes Moved Cloud Unit hh:mm:ss
----------------------------------------------------------------------------------------------------------------------------------------------------- --------- ----------- ----------- ------------
/data/col1/avamar-1562000000/cur/DELETED/1574208000/1e428c0110b9162726f5217463a247c4728985d8/1D556DA640A1786/A88DCF1006786F4B2EB9BC5B733345CDAD991683 3.24 KiB 2.19 KiB Avamar 00:14:51
/data/col1/avamar-1562000000/cur/DELETED/1574208000/1e428c0110b9162726f5217463a247c4728985d8/1D556DA640A1786/4E6B85E1E62C6E62F5BCFB969EB82E8E7F8FCE29 1.73 KiB 702.00 B Avamar 00:05:50
/data/col1/avamar-1562000000/cur/DELETED/1574121600/1e428c0110b9162726f5217463a247c4728985d8/1D556114FA39B9C/D56D73E317F1F82912DE4F4B8364E2AC353F6666 80.01 GiB 7.43 GiB Avamar 01:15:55
/data/col1/avamar-1562000000/cur/DELETED/1574208000/1e428c0110b9162726f5217463a247c4728985d8/1D556DA640A1786/B410BEB61B20C68EF1048BBF5454FD5877CB23EF 10.50 KiB 9.45 KiB Avamar 00:05:04
----------------------------------------------------------------------------------------------------------------------------------------------------- --------- ----------- ----------- ------------
4. After the Avamar Garbage Collection has completed (likely the next day as part of the maintenance schedule), confirm that the backups are now tiered:
admin@AVE01:~/#: mccli backup show --name=/clients/10.0.10.10/VirtualMachines/VMTEST01
0,23000,CLI command completed successfully.
Created LabelNum Size Retention Hostname Location Consistent Level Tier
------------------- -------- ------- --------- ---------------- -------- ---------------- ------
2019-11-20 06:08 PM 133 86.4 GB D AVE01.test.local Local app Active
2019-11-19 06:07 PM 132 86.4 GB D AVE01.test.local Local app Active
2019-11-18 06:07 PM 131 86.4 GB D AVE01.test.local Local app Active
--------------- snip ---------------
2019-10-24 07:08 PM 106 86.4 GB D AVE01.test.local Local app Active
2019-10-23 07:08 PM 105 86.4 GB D AVE01.test.local Local app Active
2019-10-22 07:07 PM 104 86.4 GB D AVE01.test.local Local app Active
2019-10-21 07:07 PM 103 86.4 GB DW AVE01.test.local Local app Marked
2019-10-17 07:08 PM 102 86.4 GB D AVE01.test.local Local app Cloud
2019-10-16 07:07 PM 101 86.4 GB D AVE01.test.local Local app Cloud
2019-10-15 07:09 PM 100 86.4 GB D AVE01.test.local Local app Cloud
--------------- snip ---------------
2019-08-25 07:07 PM 53 86.4 GB DW AVE01.test.local Local app Cloud
2019-08-24 07:07 PM 52 86.4 GB D AVE01.test.local Local app Cloud
2019-08-23 07:07 PM 51 86.4 GB D AVE01.test.local Local app Cloud
5. After being configured and run, it is now possible to verify task completion:
sysadmin@dd3300# data-movement show history
Start Time End Time Bytes Moved Bytes Moved Files Moved Elapsed Time State
(Pre-comp) (Post-comp) (hh:mm:ss)
----------------- ----------------- ----------- ----------- ----------- --------------- ---------
Nov 20 2019 20:00 Nov 20 2019 20:03 6.22 GiB 22.01 MiB 17 0 days 00:03:02 Completed
Nov 19 2019 20:00 Nov 19 2019 20:11 6.20 GiB 1.29 GiB 17 0 days 00:11:36 Completed
Nov 18 2019 17:21 Nov 19 2019 03:52 4.81 TiB 29.59 GiB 1026 0 days 10:31:01 Completed
Sep 24 2019 16:55 Sep 24 2019 16:55 0 0 0 0 days 00:00:00 Completed
Sep 24 2019 16:53 Sep 24 2019 16:53 0 0 0 0 days 00:00:00 Completed
----------------- ----------------- ----------- ----------- ----------- --------------- ---------
sysadmin@dd3300# data-movement status
Data-movement to cloud tier:
----------------------------
Data-movement was started on Nov 20 2019 20:00 and completed on Nov 20 2019 20:03
Moved (post-comp): 22.01 MiB, (pre-comp): 6.22 GiB,
Files inspected: 14227, Files eligible: 17, Files moved: 17, Files failed: 0
Data-movement recall:
---------------------
No recall operations found.
Use 'path all' option to list all recall jobs (up to the maximum supported limit).
sysadmin@dd3300# data-movement status to-tier cloud detailed
Data-movement to cloud tier:
----------------------------
Data-movement was started on Nov 20 2019 20:00 and completed on Nov 20 2019 20:03
Moved (post-comp): 22.01 MiB, (pre-comp): 6.22 GiB,
Files inspected: 14227, Files eligible: 17, Files moved: 17, Files failed: 0
Data-movement status for MTrees:
--------------------------------
MTree Files Files Files Files Bytes Moved Bytes Moved Destination
Inspected Eligible Moved Failed (Pre-comp) (Post-comp) Cloud Unit
---------------------------- --------- -------- ----- ------ ----------- ----------- -----------
/data/col1/avamar-1562250622 14227 17 17 0 6.23 GiB 22.01 MiB Avamar
---------------------------- --------- -------- ----- ------ ----------- ----------- -----------
(*) Data-movement is in progress for the marked MTrees.
Files currently being moved:
----------------------------
Path Name File Size Logical Destination Elapsed Time
Bytes Moved Cloud Unit hh:mm:ss
--------- --------- ----------- ----------- ------------
--------- --------- ----------- ----------- ------------