constant 10007, 10052,10056, and 10059 errors on VM backups

For months my environment was running rather well. Had an ES30 45 TB raw expansion shelf installed in July, no spikes caused by end users and very rarely did I have a job fail. Back in early November that changed when we had a network outage that caused issues for days. Since then, I have had about 8-12 Virtual Servers fail to backup daily with error codes 10007, 10052, and 10056.

Haven’t been able to pinpoint an exact cause of the issue. Different data stores, hosts, and proxies have been involved. Sometimes it’s the same VMs that fail but on occasion it’s different ones.

When I get a VM that has a 10056 error, I have had to do a storage vMotion and later consolidate to get it to backup. Some with 10007 and 10052 errors will back up just fine in the morning and then I need to manually correct the expiration date since DWMY retention is being applied.

I do have a ticket open with support but I wanted to chime in with other people to get some ideas on some things to check out.

Rebuilt the proxy servers. Validated they had access to all data stores and hosts.

Moved the vCenter VM to run at an earlier time

Turned off Distributed Resource Scheduler (DRS) in vSphere on the three proxy servers

3 proxy servers to handle roughly 60-65 VMs.

vCenter at version 6.0

Avamar at 7.5.0.183 and the proxies at the same version.

*Note added 10059 this morning since that error popped up on 4 VMs.