Avamar: Backup failing due to DNS query taking too much time - avtar FATAL <8941>: Fatal server connection problem, aborting initialization. Verify correct server address and login credentials.

Summary: The purpose of this KB article is to explain the special scenario where the avtar handshake fail but the ping works and the required TCP port numbers are are also open.

This article applies to This article does not apply to This article is not tied to any specific product. Not all product versions are identified in this article.
Check out other resources

Symptoms



In this scenario we found that some clients were affected while some other were not.
For the affected client the issue happen with both Win FS backups as well as VSS backups.

From the avtar logs we can see the following error message: 
avtar FATAL <8941>: Fatal server connection problem, aborting initialization. Verify correct server address and login credentials.
avtar Info <5694>: - Failed initial handshake, trying again
avtar Info <5562>: - - Connect: Trying 10.xx.xx.xx:29000
Adding log debugging we can see the extra bit of information in the log:


[avtar]  sslcertificate::verify_certificate_ip CN Name='Avamar Server RSA TLS'
    [avtar]  sslcertificate::verify_cnname Performing CN Name validation for Avamar Server RSA TLS
    [avtar]  uwrapper::gethostbyaddr DnsQuery(dns)  returned (9002) for 10.xx.xx.xx
    [avtar]  sslcertificate::verify_certificate_ip CN Name='<avamar-server-fqdn>'
    [avtar]  sslcertificate::verify_cnname Performing CN Name validation for <avamar-server-fqdn>
    [avtar]  uwrapper::gethostbyaddr DnsQuery(dns)  returned (9002) for 10.xx.xx.xx
    [avtar]  sslcertificate::verify_certificate_ip CN Name field did not match Hostname - Checking SA Names
    [avtar]  sslcertificate::verify_certificate_ip rawIPAddrLen = 4
    [avtar]  sslcertificate::verify_certificate_ip Comparing 10.xx.xx.xx with 10.xx.xx.xx
    [avtar]  sslcertificate::verify_certificate_ip Certificate successfully verified
    [avtar]  <-- SSL
    [avtar]  <-- TLS 1.2 Handshake, ServerHelloDone
    [avtar]  --> SSL
    [avtar]  --> TLS 1.2 Handshake, ClientKeyExchange
    [avtar]  --> SSL
    [avtar]  --> TLS 1.2 ChangeCipherSpec
    [avtar]  --> SSL
    [avtar]  --> TLS 1.2 Handshake, Finished
>[avtar]  sslsockimpl::open  connect failure  (setrslt 1)  (conrslt 0)
>[avtar]  Printing ssl error stack
    [avtar]  certlock::~certlock() success to remove SSL cert lock 'C:\Program Files\avs\etc\.tmp\.certlock'
    [avtar]  sslsockimpl::save_server_cert saving cert='C:\Program Files\avs\etc\servercert.pem'
    [avtar]  sslsockimpl::save_server_cert cipher='AES256-SHA'
>[avtar]  sslsockimpl::open  failure
> avtar Info <5694>: - Failed initial handshake, trying again
For the troubleshooting purpose we checked and confirmed that the TCP ports 28001, 28002, 27000 and 29000 were all open and within the correct TCP directions as per Avamar security guide, ping and DNS resolution were also working fine.



 

Cause

This issue is due to the long response time in the DNS queries, in fact we can see that the "DnsQuery(dns)  returned " was taking more than 10 seconds every time, note the handshake process run a number of query attempts before it fails completely.

As per example: 
2019/03/05-10:22:41.39299 [avtar]  sslcertificate::verify_cnname Performing CN Name validation for Avamar Server RSA TLS
2019/03/05-10:22:53.30100 [avtar]  uwrapper::gethostbyaddr DnsQuery(dns)  returned (9002) for 10.xx.xx.xx
And the entire handshake process would require about 50 seconds to complete and fail:


2019/03/05-10:22:14.89800 [avtar]  sslsockimpl::open  initclient success
....
2019/03/05-10:23:05.41599 [avtar]  sslsockimpl::open  failure

For comparison here is an example from a working client where we see that the the "DnsQuery" is returned in less than 1 second:

2019/03/05-11:56:06.97600 [avtar]  sslcertificate::verify_cnname Performing CN Name validation for <avamar-server-fqdn>
2019/03/05-11:56:07.79600 [avtar]  uwrapper::gethostbyaddr DnsQuery(dns)  returned (9002) for 10.xx.xx.xx
And the entire handshake process would complete in about 13 seconds:


2019/03/05-11:55:54.12400 [avtar]  sslsockimpl::open  initclient success
...
2019/03/05-11:56:07.82899 [avtar]  sslsockimpl::open  initclient success, cipher: AES256-SHA
In a summary, the root cause is identified as DnsQuery spent too much time on the affected client machines.

Resolution

In order to address this type of issue the System admin needs to take actions on the DNS server to resolve this delay.

Since the issue is outside the scope of the Avamar backup product the Sys admin needs to be engaged.

Additional Information

In case you experience the same errors but in your case the DNS respose in less than a second then you might be experiencing a different type of issue.
Please check first in the Dell EMC Knowledge-Base to see if any another KB article may help you resolve this issue otherwise contact the Avamar support team.

Affected Products

Avamar

Products

Avamar, Avamar Client for Windows
Article Properties
Article Number: 000055434
Article Type: Solution
Last Modified: 11 Oct 2024
Version:  3
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.