Avamar - VM backup fails when esxi version is 5.x due VixDiskLib_QueryAllocatedBlocks API failure

VM image backups are failing when CBT is enabled on an unsupported ESXi version 5.x.

I. The backup log shows the following line indicating that the target VM is running on an unsupported VMware version:

avvcbimage Info <40638>: VM Host Name=esxi.example.com(5.5 build-2068190)

II. The backup log show CBT is enabled, but backup is going to be a full CBT level zero backup: 

avvcbimage Info <11988>: A reference to a valid prior backup is not available so this will be a full level zero backup.

OR 

avvcbimage Info <14681>: Change block tracking (CTK) is supported for this VM but has been declined. This will increase proxy CPU and IOP consumption.

III. The backup log shows that a successful snapshot can be created:

avvcbimage Info <14632>: Snapshot 'Avamar-16763996243af4ead6dea4e3e0d24ea86d327bebed43a19e42' creation for VM '[datastore1] vmname/vmname.vmx' task completed, moref=snapshot-91273 

IV. The backup log shows the disk connected using hotadd or NBD. For example:

avvcbimage Info <40634>: Connected with hotadd transport to virtual disk [datastore1] vmname/vmname-000001.vmdk. Time to connect: 16300 msecs  

OR  

avvcbimage Info <40634>: Connected with nbd transport to virtual disk [datastore)] vmname/vmname-000001.vmdk. Time to connect: 1025 msecs

V. The backup log shows that the VixDiskLib_QueryAllocatedBlocks API call is made and fails with "Illegal message" and "NBD_ERR_GENERIC" errors:

avvcbimage Info <16041>: VDDK:VixDiskLib: VixDiskLib_QueryAllocatedBlocks: Query allocated blocks.
avvcbimage Warning <16041>: VDDK:[NFC ERROR]NfcFssrvrProcessErrorMsg: received NFC error 2 from server: Illegal message during fssrvr session, id = 49
avvcbimage Info <16041>: VDDK:DISKLIB-LIB_MISC   : DiskLibGetAllocatedSectorChunksInRangeInt: failed to get allocated sector bitmap with 'NBD_ERR_GENERIC' (290).
avvcbimage Info <16041>: VDDK:VixDiskLib: Detected DiskLib error 290 (NBD_ERR_GENERIC).
avvcbimage Info <16041>: VDDK:VixDiskLib: VixDiskLibQueryBlockList: Fail to query blocks (offset 0, chunkSize 128, chunkNumber 524288). Error 1 (Unknown error) (DiskLib error 290: NBD_ERR_GENERIC) at 591.

VI. The backup log shows graceful termination due to VixDiskLib_Read failure from Step V. 

avvcbimage Error <0000>: [IMG0008] VixDiskLib_Read() (datastore)] vmname/vmname.vmdk) at offset 0 length 128 sectors returned (1) (1)Unknown error
avvcbimage Info <9772>: Starting graceful (staged) termination, VixDiskLib_Read returned an error (wrap-up stage)

"The VMware ESXi version is unsupported. Avamar 19.2 and higher use a new API called VixDiskLib_QueryAllocatedBlocks to query for in-use blocks of a virtual disk. However, this API is not backwards compatible with ESXi 5.x versions."

The VMware server should be upgraded to supported version. ESXi 5.5 is not supported as indicated on this VMware page: End of General Support for vSphere 5.5 (51491) (vmware.com).

As a workaround, you can set the advanced flag '--enable_query_allocated_block' to false.

Here are two methods to do this:


Method 1 (over CMD file)
1. On the Proxy, edit the following file: /usr/local/avamarclient/var/avvcbimageAll.cmd and modify or add these lines: 

# If enabled, use QueryAllocatedBlock instead of QueryChangedDiskAreas for full backup/in-use query
#    Default is true.
--enable_query_allocated_block=false

Method 2 (by dataset in AUI) 
1. Edit Dataset being used for by the backup policy.
2. Select the Windows VMware Image plug-in  >  Show Free Form >  ADD 
Key: 
[avvcbimage]enable_query_allocated_block   
Value: 
False 
3. Select Linux VMware Image plug-in >  Show Free Form >  ADD 
Key: 
[avvcbimage]enable_query_allocated_block   
Value: 
False 

See Example below: