Highlighted
2 Iron

VM Backups fail intermittently

Hello,

VM Backups seems to be frozen on activity screen, no move, nothing and failed after timeout...

In logs on proxy, I notice these errors :

2017-05-25 13:39:30 avtar Info <18854>: Using Secure Session Ticket Format

2017-05-25 13:39:30 avtar Info <9772>: Starting graceful (staged) termination, Fatal server error (wrap-up stage)

2017-05-25 13:39:30 avtar Error <8609>: Connection killed from GSAN.

2017-05-25 13:39:30 avtar FATAL <5704>: Fatal Server Error occurred (MSG_ERR_AUTH_FAIL), aborting execution (SECURETICKETLOGIN=452 serial=1 seq=0 flags=R:H:0 kind=0 rsp=MSG_ERR_AUTH_FAIL)

2017-05-25 13:39:30 avtar Error <5126>: Login error 5: Authorization failure (Session Ticket login)

2017-05-25 13:39:30 avtar FATAL <8941>: Fatal server connection problem, aborting initialization. Verify correct server address and login credentials.

2017-05-25 13:39:30 avtar Info <6149>: Error summary: 4 errors: 8941, 5704, 5126, 8609

2017-05-25 13:39:30 avtar Info <8469>: Progress avtar, no wrapup.

2017-05-25 13:39:30 avtar Info <5314>: Command failed (4 errors, exit code 10008: cannot establish connection with server (possible network or DNS failure))

Do you have an idea ?

Reply
12 Replies
Highlighted
3 Cadmium

Re: VM Backups fail intermittently

refresh your connection to what ever vm software you are using.

for example VMWare

I have had to do this on occasion when the VM guys do something to VMware

Administration/Services tab

Utility node

find you vm connection and do a restart

might or might no help, but worth a try

Reply
Highlighted
2 Iron

Re: VM Backups fail intermittently

Restart services, restart Avamar, resync vCenter... Nothing works...

Reply
Highlighted
1 Copper

Re: VM Backups fail intermittently

Hello,

This issue may be related to the network settings between the Avamar Proxy and the Avamar Server.

The logs extract you provided is a bit limited, to look at the big picture we would need the see the full backup log.

Please review the "Avamar for VMware Guide" along with the "Avamar Security guide" where all the network requirements and basic configurations are described, links available on he support site.

If the issue persists I recommend to open an Avamar SR ticket, you will be better assisted.

Reply
Highlighted
2 Iron

Re: VM Backups fail intermittently

I agree with you about the imited log, but there is no more on logs...

After this message, Avamar server still think that backup is running.

I also agree about networks settings but why sometimes it works and sometimes not ?

And last, I've opened a ticket.

Regards

Reply
Highlighted
1 Copper

Re: VM Backups fail intermittently

What is the Fix for this issue ? Finally Support engineers fix the issue? Please explain about the resolution steps so that everyone will get the befit. We have placed a Ticket too .. but still they are keep on asking for logs and still issue persist since past 3 weeks.

Reply
Highlighted

Re: VM Backups fail intermittently

Hello,

Check following things -

> VC connection is up. This can be checked from Administration GUI. If it is up and other backups from same VC are running fine, no need to restart the connection.

> From proxy, ping VC and Avamar. Check hosts file on proxy is updated with FQDN of Avamar server and DD if it is in use.

> On Proxy, run "yast", check network settings to ensure correct DNS and Gateway IPs are used.

Hope this will resolve the issue.

Regards,

Pawan

Reply
Highlighted
1 Copper

Re: VM Backups fail intermittently

We're finding the exact same error is occurring against Linux and Windows filesystem backups, so i do not believe this is application specific, but rather something inside GSAN.

Each client can manually run successfully, but if the group policy is executed then we get failures with the exact same messages as described above.

This is a new ave deployment v7.4.1-58.

Any assistance would be greatly appreciated.

Reply
Highlighted
3 Cadmium

Re: VM Backups fail intermittently

how many servers are in the policy?

I had issues and was told to keep a policy down to 100.

too many starting at once can overwhelm the VM server.

so I have mine at about 100 and start them at 30 minute intervals (only so many can run at once any way)

Reply
Highlighted
1 Copper

Re: VM Backups fail intermittently

Thank you J.H., we have about 25 Linux clients running filesystem backups.  But, i don't believe this is specific to any application (VMware, Filesystem, 3rd Party) given the identical messages in the log, but rather something going on inside the GSAN where the job request cannot close due a certificate/credentials failure.

Reply