Avamar: Data Domain Integration: Replication does not appear to be progressing
Summary: When a backup in an Avamar and Data Domain integrated environment is replicated, the "Progress Bytes" in the UI Activity window do not appear to update as often as expected.
Symptoms
When monitoring the replication progress in the activity window of the Avamar UI:
Replication does not appear to progress.
-- Or --
Replication appears to stall for a long period of time and then appears to "jump" significantly:
...
2023-11-18 02:03:30 avtar Info <8688>: Status 2023-11-18 02:03:30, 1 files, 1,950 GB (1 files, 26.62 GB, 1.37% new) 398MB 0% CPU
2023-11-18 02:03:30 avtar Info <8688>: Status 2023-11-18 02:03:30, 1 files, 1,950 GB (1 files, 26.62 GB, 1.37% new) 398MB 0% CPU - (target server)
2023-11-18 02:08:30 avtar Info <8688>: Status 2023-11-18 02:08:30, 1 files, 1,950 GB (1 files, 26.62 GB, 1.37% new) 398MB 0% CPU
2023-11-18 02:13:30 avtar Info <8688>: Status 2023-11-18 02:13:30, 1 files, 1,950 GB (1 files, 26.62 GB, 1.37% new) 398MB 0% CPU
2023-11-18 02:18:30 avtar Info <8688>: Status 2023-11-18 02:18:30, 1 files, 1,950 GB (1 files, 26.62 GB, 1.37% new) 398MB 0% CPU
2023-11-18 02:18:30 avtar Info <8688>: Status 2023-11-18 02:18:30, 1 files, 1,950 GB (1 files, 26.62 GB, 1.37% new) 398MB 0% CPU - (target server)
2023-11-18 02:23:30 avtar Info <8688>: Status 2023-11-18 02:23:30, 1 files, 1,950 GB (1 files, 26.62 GB, 1.37% new) 398MB 0% CPU
2023-11-18 02:28:30 avtar Info <8688>: Status 2023-11-18 02:28:30, 1 files, 1,950 GB (1 files, 26.62 GB, 1.37% new) 398MB 0% CPU
...
2023-11-18 03:48:31 avtar Info <8688>: Status 2023-11-18 03:48:31, 1 files, 1,950 GB (1 files, 26.62 GB, 1.37% new) 398MB 0% CPU
2023-11-18 03:48:31 avtar Info <8688>: Status 2023-11-18 03:48:31, 1 files, 1,950 GB (1 files, 26.62 GB, 1.37% new) 398MB 0% CPU - (target server)
2023-11-18 03:53:31 avtar Info <8688>: Status 2023-11-18 03:53:31, 1 files, 1,950 GB (1 files, 26.62 GB, 1.37% new) 398MB 0% CPU
2023-11-18 03:58:31 avtar Info <8688>: Status 2023-11-18 03:58:31, 2 files, 2,080 GB (2 files, 40.77 GB, 1.96% new) 398MB 0% CPU
2023-11-18 04:03:31 avtar Info <8688>: Status 2023-11-18 04:03:31, 2 files, 2,080 GB (2 files, 40.77 GB, 1.96% new) 398MB 0% CPU
2023-11-18 04:03:31 avtar Info <8688>: Status 2023-11-18 04:03:31, 2 files, 2,080 GB (2 files, 40.77 GB, 1.96% new) 398MB 0% CPU - (target server)
...
2023-11-18 04:33:32 avtar Info <8688>: Status 2023-11-18 04:33:32, 2 files, 2,080 GB (2 files, 40.77 GB, 1.96% new) 398MB 0% CPU
2023-11-18 04:33:32 avtar Info <8688>: Status 2023-11-18 04:33:32, 2 files, 2,080 GB (2 files, 40.77 GB, 1.96% new) 398MB 0% CPU - (target server)
2023-11-18 04:38:32 avtar Info <8688>: Status 2023-11-18 04:38:32, 2 files, 2,080 GB (2 files, 40.77 GB, 1.96% new) 398MB 0% CPU
...
Cause
This is working by design. The replication is not stalled.
The status information from avtar reports the last updated replication file transfer that the sessionprogress object holds.
The message is reported on every 5 minutes, but the status is updated (accumulated) when each file transfer is completed to the Data Domain.
From the example above:
2023-11-18 02:03:30 avtar Info <8688>: Status 2023-11-18 02:03:30, 1 files, 1,950 GB (1 files, 26.62 GB, 1.37% new)
......
2023-11-18 03:53:31 avtar Info <8688>: Status 2023-11-18 03:53:31, 1 files, 1,950 GB (1 files, 26.62 GB, 1.37% new)
This represents the previous file transfer status, and a new (next one) file is being processed.
When completed (in this example after 1h50), the next file transfer starts:
2023-11-18 03:58:31 avtar Info <8688>: Status 2023-11-18 03:58:31, 2 files, 2,080 GB (2 files, 40.77 GB, 1.96% new)
Resolution
Assuming replication completes without issue, no further action is required.
The data transfer status can be reviewed by running the following command on the Data Domain:
ddboost file-replication show active