Start a Conversation

Unsolved

This post is more than 5 years old

5126

May 25th, 2017 06:00

VM Backups fail intermittently

Hello,

VM Backups seems to be frozen on activity screen, no move, nothing and failed after timeout...

In logs on proxy, I notice these errors :

2017-05-25 13:39:30 avtar Info <18854>: Using Secure Session Ticket Format

2017-05-25 13:39:30 avtar Info <9772>: Starting graceful (staged) termination, Fatal server error (wrap-up stage)

2017-05-25 13:39:30 avtar Error <8609>: Connection killed from GSAN.

2017-05-25 13:39:30 avtar FATAL <5704>: Fatal Server Error occurred (MSG_ERR_AUTH_FAIL), aborting execution (SECURETICKETLOGIN=452 serial=1 seq=0 flags=R:H:0 kind=0 rsp=MSG_ERR_AUTH_FAIL)

2017-05-25 13:39:30 avtar Error <5126>: Login error 5: Authorization failure (Session Ticket login)

2017-05-25 13:39:30 avtar FATAL <8941>: Fatal server connection problem, aborting initialization. Verify correct server address and login credentials.

2017-05-25 13:39:30 avtar Info <6149>: Error summary: 4 errors: 8941, 5704, 5126, 8609

2017-05-25 13:39:30 avtar Info <8469>: Progress avtar, no wrapup.

2017-05-25 13:39:30 avtar Info <5314>: Command failed (4 errors, exit code 10008: cannot establish connection with server (possible network or DNS failure))

Do you have an idea ?

498 Posts

May 25th, 2017 08:00

refresh your connection to what ever vm software you are using.

for example VMWare

I have had to do this on occasion when the VM guys do something to VMware

Administration/Services tab

Utility node

find you vm connection and do a restart

might or might no help, but worth a try

42 Posts

May 25th, 2017 11:00

Restart services, restart Avamar, resync vCenter... Nothing works...

6 Posts

May 26th, 2017 06:00

Hello,

This issue may be related to the network settings between the Avamar Proxy and the Avamar Server.

The logs extract you provided is a bit limited, to look at the big picture we would need the see the full backup log.

Please review the "Avamar for VMware Guide" along with the "Avamar Security guide" where all the network requirements and basic configurations are described, links available on he support site.

If the issue persists I recommend to open an Avamar SR ticket, you will be better assisted.

42 Posts

May 27th, 2017 02:00

I agree with you about the imited log, but there is no more on logs...

After this message, Avamar server still think that backup is running.

I also agree about networks settings but why sometimes it works and sometimes not ?

And last, I've opened a ticket.

Regards

May 30th, 2017 02:00

Hello,

Check following things -

> VC connection is up. This can be checked from Administration GUI. If it is up and other backups from same VC are running fine, no need to restart the connection.

> From proxy, ping VC and Avamar. Check hosts file on proxy is updated with FQDN of Avamar server and DD if it is in use.

> On Proxy, run "yast", check network settings to ensure correct DNS and Gateway IPs are used.

Hope this will resolve the issue.

Regards,

Pawan

5 Practitioner

 • 

274.2K Posts

August 22nd, 2017 09:00

We're finding the exact same error is occurring against Linux and Windows filesystem backups, so i do not believe this is application specific, but rather something inside GSAN.

Each client can manually run successfully, but if the group policy is executed then we get failures with the exact same messages as described above.

This is a new ave deployment v7.4.1-58.

Any assistance would be greatly appreciated.

498 Posts

August 22nd, 2017 09:00

how many servers are in the policy?

I had issues and was told to keep a policy down to 100.

too many starting at once can overwhelm the VM server.

so I have mine at about 100 and start them at 30 minute intervals (only so many can run at once any way)

5 Practitioner

 • 

274.2K Posts

August 22nd, 2017 09:00

Thank you J.H., we have about 25 Linux clients running filesystem backups.  But, i don't believe this is specific to any application (VMware, Filesystem, 3rd Party) given the identical messages in the log, but rather something going on inside the GSAN where the job request cannot close due a certificate/credentials failure.

October 8th, 2017 14:00

I had the same problem and opened an SR.  The resolution is that the Avamar proxy name needs to be added to DNS (forward and reverse) or the hosts file of all Avamar Utility/Storage nodes.  This is because of a new security feature on version 7.3 that does mutual authentication and that requires name resolution from the grid nodes back to the client IP.  I'm not sure if this is only part of the Avamar Security Package or if it's part of the base product as we have the Security Package installed.

I added hosts entries for all proxies on the 1 x utility, 3 x storage nodes, and 1 x data domain (using the Ethernet, Settings, Hosts Mapping).

On the plus side, you can go ahead and setup the DDBOOST ACL on the Data Domain after doing this.

October 8th, 2017 15:00

There is also an internal article for this on the support site:

Backup could fail with 'Timed Out - Response' or MSG_ERR_AUTH_FAIL due to DD Boost Token slowness

Article Number 000499805

October 10th, 2017 13:00

If you have a multiple ethernet interface scenario on the Avamar VM Proxy (vCenter on one, Avamar/DD on another), you have to the IP that is on the Aavamar/DD network to the hosts file on the Avamar nodes and also into the Data Domain (can be done via GUI on Ethernet screen).

1 Message

May 7th, 2019 13:00

What is the Fix for this issue ? Finally Support engineers fix the issue? Please explain about the resolution steps so that everyone will get the befit. We have placed a Ticket too .. but still they are keep on asking for logs and still issue persist since past 3 weeks.

No Events found!

Top