Avamar: VM image backups stopped responding
Summary: VM image backups stopped responding.
Symptoms
-
From Avamar Activity Monitor, it shows all VM backups in hung state
-
Backup log shows a warning, but does not seem to stop the vCenter connection. This is where the log stop moving forward forever:
2019-09-14 01:28:30 avvcbimage Info <18664>: Login(https://vc.sample.com:443/sdk) Datacenter: 'Production DataCenter' 2019-09-14 01:28:30 avvcbimage Info <19728>: - connected to 'VirtualCenter' - version: 'VMware vCenter Server 6.5.0 build-14389939', apiVersion:'6.5' 2019-09-14 01:28:30 avvcbimage Info <16005>: Login(https://vc.sample.com:443/sdk) problem with reused sessionID='5e11ee9f4d2b1064869dcfc75245c638c1971407' contacting data center 'Production DataCenter'. 2019-09-14 01:28:30 avvcbimage Warning <0000>: [IMG0014] Problem logging into URL 'https://vc.sample.com:443/sdk' with session cookie. 2019-09-14 01:28:30 avvcbimage Info <16021>: Logging into URL 'https://vc.sample.com:443/sdk' with user 'AD_Domain\avmrbkp' credentials. 2019-09-14 01:30:38 avvcbimage Info <19726>: Login(https://vc.sample.com:443/sdk) Datacenter: 'Production DataCenter', connected to 'VirtualCenter' - version: 'VMware vCenter Server 6.5.0 build-14389939' apiVersion: '6.5' 2019-09-14 01:30:38 avvcbimage Info <42219>: vCenter login using username and password.
-
Issue remains the same after stopped and started vCenter connection and also tried reboot vCenter Server
-
Run "java -jar proxycp.jar --listpermission --vm sampleVM" to check vCenter account permission, it gives the following result:
Error while fetching VM : Exception caught trying to invoke method RetrieveServiceContent; nested exception is: java.net.ConnectException: Connection timed out (Connection timed out)
-
Telnet vCenter server at 443, it worked fine:
/> telnet vc.sample.com 443 Trying 172.24.32.102... Connected to vc.sample.com. Escape character is '^]'.
-
But when we ran cur command to connect to the vCenter server, it appears a different IP address than telnet the same FQDN:
/> curl -kv https://vc.sample.com:443/sdk * Hostname was NOT found in DNS cache * Trying 172.24.32.172...
Cause
This was due to there are two DNS entries for the same vCenter server, the first works and the second does not work.
nslookup vc.sample.com Server: xx.xx.xx.xx Address: xx.xx.xx.xx#53 Name: vc.sample.com Address: 172.24.32.102 Name: vc.sample.com Address: 172.24.32.172
Resolution
Remove the second dns entry from the DNS server resolves the problem.