Avamar: Intermittent Virtual Machine Image Backup Failure with Error "vSphere Task failed"

Backup fails at the snapshot creation phase with the following errors:

2023-08-06 23:06:32 avvcbimage Error <44034>: vSphere Task failed (snapshot error=45): 'There are currently 2 usable fault domains. The operation requires 1 more usable fault domains.'.
2023-08-06 23:06:32 avvcbimage Error <44020>: Snapshot 'Avamar-169136317118b22ef41ebc7a87342f2790f581058cfa460bfd' creation for VM '[VxRail-Virtual-SAN-Datastore-xxxx-xxxx-xx] xxx-xxxx-xxx/vm_name.vmx' task creation encountered a problem

This issue occurs sporadically with random VMs that reside on a vSAN datastore managed by VxRail.

The same error message as above is observed in the vSphere UI:

Create Snapshot message: There are currently 2 usable fault domains. The operation requires 1 more usable fault domains.
--> An error occurred while taking a snapshot: Out of resources.

VxRail uses the concept of "Fault Domains." Fault Domains in vSAN are an optional configuration to improve the resilience of a vSAN cluster. It ensures that data in the cluster remains available if failure or unavailability of a single host, or multiple hosts.

As per the following VMware KB: Unable to snapshot VMs when less than the minimum required hosts of a specific storage policy are online , vSAN requires a minimum number of hosts to satisfy its storage policy. If one of these hosts becomes unavailable, vSAN does not allow the provisioning of new objects because there are not enough resources to satisfy the applied storage policy.

The root cause analysis of this backup failure indicates that the vSAN Witness Host is in the backup policy. Backing up the Witness Host is not recommended, as snapshot operations can cause the above discrepancy, which affects other running backups with the above errors.

Exclude the vSAN Witness Host from the backup policy.