PowerProtect: TSDM backups fail with ABV0016 against a specific ESXi host

The DPD service has dependencies with the vCenter vSAN Health Service, even when vSAN storage is not configured in the environment. The vCenter vSAN Health Service provides the ESXi version details to determine the number of concurrent Snapshot Data Mover (SDM) sessions supported by the ESXi host. The number of concurrent SDM backup sessions per ESXi host is 20 with vSphere version 7.0 U3d (ESXi and vCenter). The vSphere versions prior to vSphere 7.0 U3d (legacy versions) could only perform 10 concurrent SDM backup sessions.  

If the vCenter vSAN Health Service cache has outdated version information or a null entry, it categorizes the ESXi host as legacy. The default PowerProtect Data Manager throttle sends 18 SDM backup sessions per ESXi host. The ESXi host processes the initial 10 SDM backups, but the vCenter would fail any subsequent SDM backups indicating the DPD limit was exceeded.

There are enhancements in the vSphere 8.0 U1 that automatically refresh the vSAN Health service cache in certain conditions. It is recommended to upgrade to vSphere 8.0 U1 to take advantage of these enhancements and prevent certain vSAN Health service cache symptoms. 

VMware Workaround:
Ensure the vCenter server and all the ESXi hosts in the cluster are at version 7.0 U3d or newer. Clear the cache by restarting the vCenter Server Appliance (VCSA) vSAN Health service over an SSH session:

service-control --restart vsan-health

If required, contact VMware Support for further insight or assistance with the vSAN Health service.

PowerProtect Data Manager Workaround:
There is a method in PowerProtect Data Manager to decrease the throttle value for TSDM backup sessions against all ESXi hosts. This can be used as a temporary workaround while upgrading the ESXi hosts or working with VMware Support.

• Ensure that no protection jobs are running
• Log in to PowerProtect Data Manager and elevate to sudo: sudo su -
• Make a copy of the vmdirect.env file: cp vmdirect.env vmdirect.env.orig
• Open environment vmdirect.env file: vi /opt/emc/vmdirect/unit/vmdirect.env
• Change the value for the environment variable "MAX_SDM_BACKUP_SESSIONS_PER_HOST" to 10 (instead of 18) and save the file.
• Restart the VISD service: 

  systemctl restart visd.service

Example output:

my-ppdm:/opt/emc/vmdirect/unit # cp vmdirect.env vmdirect.env.orig
my-ppdm:/opt/emc/vmdirect/unit # ls -l
total 20
-rwxr-xr-x 1 root root  397 Nov  2 22:48 visd.service
-rwxr-xr-x 1 root root 2225 Nov  2 22:48 vmdirect.env
-rwx------ 1 root root 2225 Apr 21 09:39 vmdirect.env.orig
-rwxr-xr-x 1 root root  360 Nov  2 22:48 vpod-nas.service
-rwxr-xr-x 1 root root  334 Nov  2 22:48 vpod.service
my-ppdm:/opt/emc/vmdirect/unit # vi vmdirect.env
my-ppdm:/opt/emc/vmdirect/unit # cat vmdirect.env | grep SDM
SDM_ENABLED_EXTERNAL_PROXIES=false
export SDM_ENABLED_EXTERNAL_PROXIES
MAX_SDM_SESSIONS_PER_HOST=20
export MAX_SDM_SESSIONS_PER_HOST
MAX_SDM_BACKUP_SESSIONS_PER_HOST=10
export MAX_SDM_BACKUP_SESSIONS_PER_HOST
MAX_ESX_SDM_TOTAL_DISKS=40
export MAX_ESX_SDM_TOTAL_DISKS
my-ppdm:/opt/emc/vmdirect/unit # systemctl restart visd.service
my-ppdm:/opt/emc/vmdirect/unit # systemctl status visd.service
● visd.service - VMDirect Infrastructure Services Daemon
   Loaded: loaded (/usr/lib/systemd/system/visd.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri YYYY-04-21 09:42:38 PDT; 7s ago
Main PID: 67488 (visd)
    Tasks: 16 (limit: 512)
   Memory: 23.2M
      CPU: 317ms
   CGroup: /system.slice/visd.service
           └─67488 /opt/emc/vmdirect/bin/visd -user admin -group app -p 9097 -nodemgmt -proxymgmt -vmdeployment -installclient -l

Apr 21 09:42:38 my-ppdm.my-domain.com systemd[1]: Started VMDirect Infrastructure Services Daemon.
Apr 21 09:42:38 my-ppdm.my-domain.com visd[67488]: INFO:   Found GID 2000 (app) for group 'app'.
Apr 21 09:42:38 my-ppdm.my-domain.com visd[67488]: NOTICE: Effective group ID is set to 2000 (app).
Apr 21 09:42:38 my-ppdm.my-domain.com visd[67488]: INFO:   Found UID 500 (admin) for user 'admin'.
Apr 21 09:42:38 my-ppdm.my-domain.com visd[67488]: NOTICE: Effective user ID is set to 500 (admin).
my-ppdm:/opt/emc/vmdirect/unit #