Avamar: Replication fails with INTERNAL CYCLE ERROR

An Avamar grid may experience issues replicating backups for one or more clients.

This particular situation was encountered on a VSphere Data Protection (VDP) system. The vdp-client-activity.log provides the following information: 

2016-10-03 04:11:03 avtar Info <6654>: Replicating backup 244, Label "somebackup-1475002800005", 26.27 GB 
2016-10-03 04:16:01 avtar Info <8688>: Status 2016-10-03 04:16:01, 26.00 GB (0 bytes, 0.00% new) 638MB 0% CPU client04_UATRUr8adn0H5ZysJuc9xw#244::/VMFiles/1/ 
2016-10-03 04:16:04 avtar Info <7694>: Server(vdp05.customer.com) not responding (possible network congestion?) (300 seconds) 
2016-10-03 04:16:04 avtar Info <7694>: Server(vdp06.customer.com) not responding (possible network congestion?) (300 seconds) 
2016-10-03 04:21:01 avtar Info <8688>: Status 2016-10-03 04:21:01, 26.00 GB (0 bytes, 0.00% new) 638MB 0% CPU client04_UATRUr8adn0H5ZysJuc9xw#244::/VMFiles/1/ 
2016-10-03 04:21:03 avtar Error <5445>: INTERNAL CYCLE ERROR: bad response in repldata DST_ADD_DATA: ADD_HASH_DATA=2 serial=114 seq=0 flags=R:N:1 kind=7 rsp=MSG_ERR_INVALID_PARAMETERS 
2016-10-03 04:16:04 avtar Info <7694>: Server(vdp05.customer.com) not responding (possible network congestion?) (300 seconds) 
2016-10-03 04:16:04 avtar Info <7694>: Server(vdp06.customer.com) not responding (possible network congestion?) (300 seconds) 
2016-10-03 04:47:43 avtar Info <7694>: Server(vdp05.customer.com) not responding (possible network congestion?) (300 seconds) 

The repeated messages 'possible network congestion' do not necessarily mean that there is a connectivity problem. They indicate that the target grid is not responding in the expected way.

The error message provides more information by stating that there is a problem with source grids requests to ADD_DATA to the destination VDP "DST_ADD_DATA: ADD_HASH_DATA=2."

Whilst investigating this issue, it was not possible to identify the root cause of the problem. "MSG_ERR_INVALID_PARAMETERS" however the key phrase INTERNAL CYCLE ERROR is indicative of potential client cache corruption.

The following method can be used to resolve this behavior.
(If this exact problem occurs repeatedly, contact the Dell Technologies Support team.)

1. Make a list of the clients whose backups failed with the errors described above.

2. Copy the existing replication group to create a "test" replication group. Alternatively, modify an existing replication group temporarily and ensure that it only contains the problem clients.

3. Create or edit the file /usr/local/avamar/var/client/avtar.cmd  on the source grid.

4. Add the entry '--nocache' to the file.

This prevents replication jobs from leveraging the client cache files.

If the client cache is corrupt, not using them can determine if they are the cause of the replication failure.

5. Run the replication job containing the 'problem clients' and allow it to complete.

If the replication fails, this article does not apply, and further investigation is needed. 

If the replication succeeds, the client caches must be regenerated for the problematic clients.

6. To regenerate the client caches:

a. Edit /usr/local/avamar/var/client/avtar.cmd again.

b. Remove the '--nocache' entry and replace it with '--clearcache'.

7. Rerun the test replication group so that the client caches are re-created. Wait for the job to be complete.

8. Delete the avtar.cmd file or edit usr/local/avamar/var/client/avtar.cmd and remove the '--clearcache' entry.  

9. Revert to the original replication group configuration which contains all the clients that must be replicated.

10. Allow scheduled replication to run as normal. The clients whose caches were regenerated should now replicate.