3 Posts
0
9417
Failed initial handshake, trying again
Hello, can you please help me with Avamar backup not working. This is what I get when I initiate backup:
...
2018-10-31 10:23:21 avtar Info <6555>: Initializing connection (Avamar Deduplication Engine v2.0.0)
2018-10-31 10:23:22 avtar Info <5552>: Connecting to Avamar Server (avamarserver1.company.com)
2018-10-31 10:23:22 avtar Info <5554>: Connecting to one node in each datacenter
2018-10-31 10:23:22 avtar Info <5694>: - Failed initial handshake, trying again
2018-10-31 10:23:22 avtar Info <5694>: - Failed initial handshake, trying again
2018-10-31 10:23:22 avtar Info <5694>: - Failed initial handshake, trying again
2018-10-31 10:23:22 avtar Info <6063>: - Communication error: Could not create connection to Server
2018-10-31 10:23:22 avtar Info <5557>: No connections available
2018-10-31 10:23:22 avtar FATAL <8604>: Fatal server connection problem, aborting initialization. Verify correct server address and login credentials.
2018-10-31 10:23:22 avtar FATAL <8941>: Fatal server connection problem, aborting initialization. Verify correct server address and login credentials.
2018-10-31 10:23:22 avtar Info <6149>: Error summary: 2 errors: 8604, 8941
2018-10-31 10:23:22 avtar Info <8468>: Sending wrapup message to parent
2018-10-31 10:23:22 avtar Info <5314>: Command failed (2 errors, exit code 10008: cannot establish connection with server (possible network or DNS failure))
I checked ports from my server to avamar servers and all these are ok:
myserver avamar agent -> avamar_nodes: 27000 (if not encrypted communication)
myserver avamar agent -> avamar_nodes: 29000 (if encrypted communication)
myserver avamar agent -> avamar server: 28001
myserver avamar agent -> avamar server: 29000
myserver avamar agent -> avamar_datadomain: 111
Another thing is I can't get debug messages despite I set debug=true in my correct dataset.
Thank you
Tomas-TS
3 Posts
0
November 7th, 2018 06:00
Problem solved. Wrong Avamar client version caused this annoying problem. Really not easy to figure out that handshake failed is because of mismatched client and avamar grid version. Also mismatched version registers with grid, but no error message about incompatibility. And at least grid could send update command back to registered outdated client. That's just my recommendation how to improve it.
chris_mcmorris
1 Message
0
November 1st, 2018 19:00
We have the same problem. Randomly started happening last week. Support looked at the grid for 3 hours this afternoon. No joy or explanation. They say it's a network problem and I should check with the network team... yes... that would be me then!
I rebooted the grid - came back up fine but all 8 storage nodes were not reachable on the LAN so I logged into the console of each node and started a ping to 8.8.8.8 and then our monitoring system instantly picked up the nodes pinging again. Very strange...
That was 5pm this afternoon but it seems the nodes have lost networking again (not reachable) now the scheduled backups are running and have failed... once again.
Support really need to sort this out...
DELL-Sam L
Moderator
Moderator
•
6.9K Posts
0
November 6th, 2018 12:00
Hello tomas-ts,
Have you tried the steps that are in this post https://community.emc.com/message/803236 to see if it can resolve your issue?
Please let us know if you have any other questions.
Tomas-TS
3 Posts
0
November 7th, 2018 00:00
Yes I tried that, ports are open, but can't switch dataset to debug mode, so can't see where it's failing.
2018-10-31 11:19:56 avtar Info <6555>: Initializing connection (Avamar Deduplication Engine v2.0.0)
2018-10-31 11:19:56 avtar Info <5552>: Connecting to Avamar Server (avamarserver1.company.com)
2018-10-31 11:19:56 avtar Info <5554>: Connecting to one node in each datacenter
2018-10-31 11:19:56 avtar Info <5694>: - Failed initial handshake, trying again
2018-10-31 11:19:57 avtar Info <5694>: - Failed initial handshake, trying again
2018-10-31 11:19:57 avtar Info <5694>: - Failed initial handshake, trying again
2018-10-31 11:19:57 avtar Info <6063>: - Communication error: Could not create connection to Server
2018-10-31 11:19:57 avtar Info <5557>: No connections available
2018-10-31 11:19:57 avtar FATAL <8604>: Fatal server connection problem, aborting initialization. Verify correct server address and login credentials.
2018-10-31 11:19:57 avtar FATAL <8941>: Fatal server connection problem, aborting initialization. Verify correct server address and login credentials.
2018-10-31 11:19:57 avtar Info <6149>: Error summary: 2 errors: 8604, 8941
2018-10-31 11:19:57 avtar Info <8468>: Sending wrapup message to parent
2018-10-31 11:19:57 avtar Info <5314>: Command failed (2 errors, exit code 10008: cannot establish connection with server (possible network or DNS failure))
And this is from another server where it's working:
2018-10-31 15:46:47 avtar Info <6555>: Initializing connection
2018-10-31 15:46:47 avtar Info <5552>: Connecting to Avamar Server (avamarserver2.company.com)
2018-10-31 15:46:47 avtar Info <5554>: Connecting to one node in each datacenter
2018-10-31 15:46:47 avtar Info <5993>: - Connect: Connected to x.x.x.57:29000, Priv=0, SSL Cipher=AES256-SHA
2018-10-31 15:46:47 avtar Info <5559>: - Trying connection x.x.x.58:29000
2018-10-31 15:46:47 avtar Info <5560>: - Connected to node x.x.x.58:29000, Priv=0, SSL Cipher=AES256-SHA
2018-10-31 15:46:47 avtar Info <5581>: Logging in on connection 0 with Session Ticket
2018-10-31 15:46:47 avtar Info <5582>: Avamar Server login successful
So it is connecting to a node to port 29000 and I tested that connected on the not-working server and it's open:
[root@server01 log]# telnet node1 29000
Trying x.x.x.45...
Connected to node1.
Escape character is '^]'.
There is firewall running on server1 - but it's set to accept all connections. Connection from server1 to node1 is working, but handshake failed could mean that node1 connect back to server1 and this is refused?