2018-10-31 10:23:22 avtar Info <8468>: Sending wrapup message to parent
2018-10-31 10:23:22 avtar Info <5314>: Command failed (2 errors, exit code 10008: cannot establish connection with server (possible network or DNS failure))
I checked ports from my server to avamar servers and all these are ok:
myserver avamar agent -> avamar_nodes: 27000 (if not encrypted communication)
Problem solved. Wrong Avamar client version caused this annoying problem. Really not easy to figure out that handshake failed is because of mismatched client and avamar grid version. Also mismatched version registers with grid, but no error message about incompatibility. And at least grid could send update command back to registered outdated client. That's just my recommendation how to improve it.
We have the same problem. Randomly started happening last week. Support looked at the grid for 3 hours this afternoon. No joy or explanation. They say it's a network problem and I should check with the network team... yes... that would be me then!
I rebooted the grid - came back up fine but all 8 storage nodes were not reachable on the LAN so I logged into the console of each node and started a ping to 8.8.8.8 and then our monitoring system instantly picked up the nodes pinging again. Very strange...
That was 5pm this afternoon but it seems the nodes have lost networking again (not reachable) now the scheduled backups are running and have failed... once again.
2018-10-31 11:19:57 avtar Info <8468>: Sending wrapup message to parent
2018-10-31 11:19:57 avtar Info <5314>: Command failed (2 errors, exit code 10008: cannot establish connection with server (possible network or DNS failure))
And this is from another server where it's working:
2018-10-31 15:46:47 avtar Info <6555>: Initializing connection
2018-10-31 15:46:47 avtar Info <5552>: Connecting to Avamar Server (avamarserver2.company.com)
2018-10-31 15:46:47 avtar Info <5554>: Connecting to one node in each datacenter
2018-10-31 15:46:47 avtar Info <5993>: - Connect: Connected to x.x.x.57:29000, Priv=0, SSL Cipher=AES256-SHA
2018-10-31 15:46:47 avtar Info <5559>: - Trying connection x.x.x.58:29000
2018-10-31 15:46:47 avtar Info <5560>: - Connected to node x.x.x.58:29000, Priv=0, SSL Cipher=AES256-SHA
2018-10-31 15:46:47 avtar Info <5581>: Logging in on connection 0 with Session Ticket
2018-10-31 15:46:47 avtar Info <5582>: Avamar Server login successful
So it is connecting to a node to port 29000 and I tested that connected on the not-working server and it's open:
[root@server01 log]# telnet node1 29000
Trying x.x.x.45...
Connected to node1.
Escape character is '^]'.
There is firewall running on server1 - but it's set to accept all connections. Connection from server1 to node1 is working, but handshake failed could mean that node1 connect back to server1 and this is refused?
Tomas-TS
3 Posts
8943
0
Posted November 7th, 2018 06:00
Problem solved. Wrong Avamar client version caused this annoying problem. Really not easy to figure out that handshake failed is because of mismatched client and avamar grid version. Also mismatched version registers with grid, but no error message about incompatibility. And at least grid could send update command back to registered outdated client. That's just my recommendation how to improve it.