Highlighted
tomas-ts
1 Copper

Failed initial handshake, trying again

Jump to solution

Hello, can you please help me with Avamar backup not working. This is what I get when I initiate backup:

...

2018-10-31 10:23:21 avtar Info <6555>: Initializing connection (Avamar Deduplication Engine v2.0.0)

2018-10-31 10:23:22 avtar Info <5552>: Connecting to Avamar Server (avamarserver1.company.com)

2018-10-31 10:23:22 avtar Info <5554>: Connecting to one node in each datacenter

2018-10-31 10:23:22 avtar Info <5694>: - Failed initial handshake, trying again

2018-10-31 10:23:22 avtar Info <5694>: - Failed initial handshake, trying again

2018-10-31 10:23:22 avtar Info <5694>: - Failed initial handshake, trying again

2018-10-31 10:23:22 avtar Info <6063>: - Communication error: Could not create connection to Server

2018-10-31 10:23:22 avtar Info <5557>: No connections available

2018-10-31 10:23:22 avtar FATAL <8604>: Fatal server connection problem, aborting initialization. Verify correct server address and login credentials.

2018-10-31 10:23:22 avtar FATAL <8941>: Fatal server connection problem, aborting initialization. Verify correct server address and login credentials.

2018-10-31 10:23:22 avtar Info <6149>: Error summary: 2 errors: 8604, 8941

2018-10-31 10:23:22 avtar Info <8468>: Sending wrapup message to parent

2018-10-31 10:23:22 avtar Info <5314>: Command failed (2 errors, exit code 10008: cannot establish connection with server (possible network or DNS failure))

I checked ports from my server to avamar servers and all these are ok:

myserver avamar agent -> avamar_nodes: 27000 (if not encrypted communication)

myserver avamar agent -> avamar_nodes: 29000 (if encrypted communication)

myserver avamar agent -> avamar server: 28001

myserver avamar agent -> avamar server: 29000

myserver avamar agent -> avamar_datadomain: 111

Another thing is I can't get debug messages despite I set debug=true in my correct dataset.

Thank you

Tags (3)
0 Kudos
1 Solution

Accepted Solutions
tomas-ts
1 Copper

Re: Failed initial handshake, trying again

Jump to solution

Problem solved. Wrong Avamar client version caused this annoying problem. Really not easy to figure out that handshake failed is because of mismatched client and avamar grid version. Also mismatched version registers with grid, but no error message about incompatibility. And at least grid could send update command back to registered outdated client. That's just my recommendation how to improve it.

0 Kudos
4 Replies
chris_mcmorris
1 Copper

Re: Failed initial handshake, trying again

Jump to solution

We have the same problem. Randomly started happening last week. Support looked at the grid for 3 hours this afternoon. No joy or explanation. They say it's a network problem and I should check with the network team... yes... that would be me then!

I rebooted the grid - came back up fine but all 8 storage nodes were not reachable on the LAN so I logged into the console of each node and started a ping to 8.8.8.8 and then our monitoring system instantly picked up the nodes pinging again. Very strange...

That was 5pm this afternoon but it seems the nodes have lost networking again (not reachable) now the scheduled backups are running and have failed... once again.

Support really need to sort this out...

0 Kudos
Moderator
Moderator

Re: Failed initial handshake, trying again

Jump to solution

Hello tomas-ts,

Have you tried the steps that are in this post https://community.emc.com/message/803236 to see if it can resolve your issue?

Please let us know if you have any other questions.

DELL-Sam L
Dell | Social Outreach Services - Enterprise
Download the Dell Quick Resource Locator app today to access PowerEdge support content on your mobile device! (iOS, Android, Windows)

0 Kudos
tomas-ts
1 Copper

Re: Failed initial handshake, trying again

Jump to solution

Yes I tried that, ports are open, but can't switch dataset to debug mode, so can't see where it's failing.

2018-10-31 11:19:56 avtar Info <6555>: Initializing connection (Avamar Deduplication Engine v2.0.0)

2018-10-31 11:19:56 avtar Info <5552>: Connecting to Avamar Server (avamarserver1.company.com)

2018-10-31 11:19:56 avtar Info <5554>: Connecting to one node in each datacenter

2018-10-31 11:19:56 avtar Info <5694>: - Failed initial handshake, trying again

2018-10-31 11:19:57 avtar Info <5694>: - Failed initial handshake, trying again

2018-10-31 11:19:57 avtar Info <5694>: - Failed initial handshake, trying again

2018-10-31 11:19:57 avtar Info <6063>: - Communication error: Could not create connection to Server

2018-10-31 11:19:57 avtar Info <5557>: No connections available

2018-10-31 11:19:57 avtar FATAL <8604>: Fatal server connection problem, aborting initialization. Verify correct server address and login credentials.

2018-10-31 11:19:57 avtar FATAL <8941>: Fatal server connection problem, aborting initialization. Verify correct server address and login credentials.

2018-10-31 11:19:57 avtar Info <6149>: Error summary: 2 errors: 8604, 8941

2018-10-31 11:19:57 avtar Info <8468>: Sending wrapup message to parent

2018-10-31 11:19:57 avtar Info <5314>: Command failed (2 errors, exit code 10008: cannot establish connection with server (possible network or DNS failure))

And this is from another server where it's working:

2018-10-31 15:46:47 avtar Info <6555>: Initializing connection

2018-10-31 15:46:47 avtar Info <5552>: Connecting to Avamar Server (avamarserver2.company.com)

2018-10-31 15:46:47 avtar Info <5554>: Connecting to one node in each datacenter

2018-10-31 15:46:47 avtar Info <5993>: - Connect: Connected to x.x.x.57:29000, Priv=0, SSL Cipher=AES256-SHA

2018-10-31 15:46:47 avtar Info <5559>: - Trying connection x.x.x.58:29000

2018-10-31 15:46:47 avtar Info <5560>: - Connected to node x.x.x.58:29000, Priv=0, SSL Cipher=AES256-SHA

2018-10-31 15:46:47 avtar Info <5581>: Logging in on connection 0 with Session Ticket

2018-10-31 15:46:47 avtar Info <5582>: Avamar Server login successful

So it is connecting to a node to port 29000 and I tested that connected on the not-working server and it's open:

[root@server01 log]# telnet node1 29000

Trying x.x.x.45...

Connected to node1.

Escape character is '^]'.

There is firewall running on server1 - but it's set to accept all connections. Connection from server1 to node1 is working, but handshake failed could mean that node1 connect back to server1 and this is refused?

0 Kudos
tomas-ts
1 Copper

Re: Failed initial handshake, trying again

Jump to solution

Problem solved. Wrong Avamar client version caused this annoying problem. Really not easy to figure out that handshake failed is because of mismatched client and avamar grid version. Also mismatched version registers with grid, but no error message about incompatibility. And at least grid could send update command back to registered outdated client. That's just my recommendation how to improve it.

0 Kudos