Avamar: Backups fail with "An operation on a socket could not be performed because the system lacked sufficient buffer space ..."

Windows File System backups may fail with errors similar to the following:

2018-11-29 13:17:41 avtar Info <5694>: - Failed initial handshake, trying again
2018-11-29 13:17:41 avtar Info <6063>: - Communication error: Could not create connection to Server
2018-11-29 13:17:41 avtar Info <5557>: No connections available
2018-11-29 13:17:41 avtar FATAL <8941>: Fatal server connection problem, aborting initialization. Verify correct server address and login credentials.

When debugging is enabled either in the dataset or in the avtar.cmd file in the \Program Files\avs\var directory (--debug flag), the following messages may be seen:

2018/11/29-18:17:41.74900 [avtar]  Windows ::connect() code 10055: An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full
2018/11/29-18:17:41.74900 [avtar]  sockimpl::init connect returns -1 errno=2 No such file or directory
2018/11/29-18:17:41.74900 [avtar]  sslsockimpl::open  failure
2018/11/29-18:17:41.74900 [avtar]  tcpsockimpl::close shutdown error -1 errno 2 code 10057: A request to send or receive data was disallowed because the socket is not connected and (when sending on a datagram socket using a sendto call) no address was supplied

 

It is caused by port exhaustion in the Windows operating system. When application(s) makes too many outgoing connections in a short time frame or does not dispose outgoing connections properly, it depletes the available ports that the avtar process requires.

To confirm this, open a PowerShell or Windows command prompt and run the following command:

netstat -ab

https://stackoverflow.com/questions/4415175/an-operation-on-a-socket-could-not-be-performed-because-the-system-lacked-suffi

http://blog.whitesites.com/blog.aspx?key=635140180116307813

https://support.microsoft.com/en-us/help/929851/the-default-dynamic-port-range-for-tcp-ip-has-changed-in-windows-vista

https://www.aip.im/2012/10/how-to-change-the-number-of-dynamic-ports/