Avamar: VMware Image Backup Fails With Resource Temporarily Unavailable

• Deploying proxy version 19.3 using proxy deployment manager.
• When Using proxy version 19.3, random VM image backups fail with the below error messages:

  [VcbImageBackupAssistThread]  ERROR: <0237> Cannot create thread (11) Resource temporarily unavailable
  avvcbimage Info <43574>: vcApiVersion 6.5
  avvcbimage Info <43575>: VSphereHTTP::vcVersion65 will be true
  avvcbimage Info <42137>: job type = 4
  .
  avvcbimage Info <16041>: VDDK:VixDiskLib: Advanced transport plugin /usr/lib/vmware-vix-disklib/lib64/libdiskLibPlugin.so was successfully loaded into vixDiskLib.
  avvcbimage Info <16041>: VDDK:VixDiskLib: Enabling advanced transport modes.
  [VcbImageBackupAssistThread]  ERROR: <0001> threadimpl::runthread Exception Resource temporarily unavailable
  avvcbimage Info <9772>: Starting graceful (staged) termination, MCS cancel (wrap-up stage)
  

• Proxy tasks count exceeded the value (512) set for TasksMax, even though the setting is infinity in the file /etc/systemd/system/multi-user.target.wants/avagent.service as below:

  <Proxy_Name>:~ # less /etc/systemd/system/multi-user.target.wants/avagent.service
  .
  [Service]
  ExecStart=/etc/init.d/avagent start
  ExecStop=/etc/init.d/avagent stop
  KillMode=process
  Restart=on-abort
  RemainAfterExit=yes
  TasksMax=infinity
  

• Also, when checking task counting from Service vami-sfcb.service, it shows the correct tasks count as below:

  <Proxy_Name>:~ # systemctl status vami-sfcb.service
    vami-sfcb.service - LSB: Small Footprint CIM Broker Service
     Loaded: loaded (/etc/init.d/vami-sfcb; disabled; vendor preset: disabled)
     Active: active (running) since <DAY> <yyyy-mm-dd hh:mm:ss UTC; # days ago
       Docs: man:systemd-sysv-generator(8)
      Tasks: 16 (limit: 512)
     CGroup: /system.slice/vami-sfcb.service
                6873 /opt/vmware/sbin/vami-sfcbd -d
                6878 /opt/vmware/sbin/vami-sfcbd -d
                6882 /opt/vmware/sbin/vami-sfcbd -d
                6883 /opt/vmware/sbin/vami-sfcbd -d
                6892 /opt/vmware/sbin/vami-sfcbd -d
                6901 /opt/vmware/sbin/vami-sfcbd -d
               11366 /usr/local/avamarclient/bin/avagent.bin --listenport=28009...
  

• But when checking task counting number from avagent.service, it shows "0" value, which means it is not counting tasks correctly as below:

  <Proxy_Name>:~ # systemctl status avagent.service
    avagent.service - Avamar Unix backup/restore agent (avagent)
     Loaded: loaded (/usr/lib/systemd/system/avagent.service; enabled; vendor preset: disabled)
     Active: active (exited) since <Day> <yyyy-mm-dd hh:mm:ss UTC; # ago
   Main PID: 1286 (code=exited, status=0/SUCCESS)
      Tasks: 0
     CGroup: /system.slice/avagent.service
               1476 /usr/local/avamarclient/bin/avagent.bin --listenport=28009 ...
  

1. On the proxy, verify that the TasksMax value is infinity in the file /etc/systemd/system/multi-user.target.wants/avagent.service as below:

<Proxy_Name>:~ # less /etc/systemd/system/multi-user.target.wants/avagent.service
.
[Service]
ExecStart=/etc/init.d/avagent start
ExecStop=/etc/init.d/avagent stop
KillMode=process
Restart=on-abort
RemainAfterExit=yes
TasksMax=infinity

2. Enable Service vami-sfcb on the proxy as below:

<Proxy_Name>:~ # systemctl enable vami-sfcb
Synchronizing state of vami-sfcb.service with SysV init with /usr/lib/systemd/systemd-sysv-install...
Executing /usr/lib/systemd/systemd-sysv-install enable vami-sfcb
Created symlink from /etc/systemd/system/multi-user.target.wants/vami-sfcb.service to /usr/lib/systemd/system/vami-sfcb.service.

3. Reboot the proxy using reboot command.
4. Verify that the service avagent.service is showing the correct task counting and not zero as below:

<Proxy_Name>:~ # systemctl status avagent.service
  avagent.service - Avamar Unix backup/restore agent (avagent)
   Loaded: loaded (/usr/lib/systemd/system/avagent.service; enabled; vendor preset: disabled)
   Active: active (exited) since <Day> <yyyy-mm-dd hh:mm:ss UTC; # ago
 Main PID: 1286 (code=exited, status=0/SUCCESS)
    Tasks: 16
   CGroup: /system.slice/avagent.service
             1476 /usr/local/avamarclient/bin/avagent.bin --listenport=28009 ...