Error Code 10055 fault InvalidLogin does not have reason

Sometimes the error is not what you think it is.

What popped into my head was LOAD.

when you start your vm backups - HOW MANY are in the policy?

I had some issues at the start (don't remember the error) and found out some info

I should only have about 100 vm's in a policy.

Stager the start times of the policies

So VMbackup1 starts at 7 pm

VMbackup2 starts a 8 pm

VMbackup3 starts a 9 pm

This will put less of a load on the VCenter when backup policies start by only adding 100 jobs at a time.

Not sure if that is your issue, but worth a try to look into it.

As you said a RE-Run works fine.  And you only have the issue at Peak Loads.

We are having the exact same issue. Have you had any resolution for this?

Like J.H., we divided our VM backup window in smaller batches throughout the night.  Most VMs backup in just a few minutes, but when we kicked off 150 VMs at once, we'd start to see snapshot timeouts and hotadd failures almost immediately.  Even though vCenter isn't reporting authentication failures from Avamar, you might see excessive connections and/or connection failures to the vCenter DB - this was the key to us that we had too many sessions trying to authenticate to vCenter, all within 5 seconds of each other.

If you're using a vCenter server appliance, you'll want to pay special attention to this, as VMware tuned the 5.x appliances fairly low, in terms of memory and maximum connections.  The 6.x appliances seem better off, but still seem a little under-sized, which you'll see during lots of backups, Storage vMotions, etc.

The simple solution for us was to group VMs in smaller batches, then kick them off throughout the evening.  Using Data Protection Advisor, we plotted a simple graph of the job schedule, then moved VMs into groups with staggered start times until we were balanced out, again, just like J.H. describes. Since then, we've had no issue due to load.

Let us know if that helps!

Karl

Thank you. This does help. EMC support is failing me at this point.

No luck yet.

I've spread the jobs out but still end up with 15 to 20 failures per night.

I have uploaded the VMware VPXD and SSO logs along with my Avamar logs to find a correlation. I did have a conference call with VMware support and EMC. During that call, the L2 engineer said they have seen this and referred VMware support to a known issue that they supposedly will not address.

I'll be sure to update if I find anything.

Karl,

how many concurrent VM backup sessions did you have ? I am using DD as my back-end store (100 streams configured for Avamar to use because it's DD9500 with 1.8k streams limit).  So when my job starts i have 1300 VMs that get submitted at once, 100 run concurrently and then the rest sits in the queue. I am not seeing these issues.

Our original backend was a DD860, which we replaced with a DD4200.  I never saw any problems from Avamar/DD (with either the DD860 or the DD4200), it was always only an issue with vCenter.  We would kick off the entire 600 VM shooting matching, but only the first 100-ish VMs would progress; the rest returned connection issues.  When we dropped the list to 200 VMs, we had more VMs that would queue longer, but then we saw more snapshot timeouts.  It wasn't until we dropped under a hundred VMs or so that we stopped seeing snapshot timeouts.

We first saw the snapshot errors on vCenter 5.0/Avamar 6.1 and 7.0 (when we switched to Avamar Gen4S nodes and DD4200), then saw primarily the hotadd issues on 7.0.  No problems ever since, but we're grouped into smaller batches during our backup window.

interesting, did not see these issues with vCenter 5.5/ Avamar 7.0 and up to vCenter 6/Avamar 7.2.  

My VM's land on DD as well.

it was an issue with VCenter not the DD.

I had too many jobs trying to talk to VCenter at once.  It slowed down to the point that Avamar timed out on the jobs.

Thanks guys for the input. We have an Avamar Gen4S /DD 2500. For weeks now it’s been flawless using a vCenter Appliance 6.0.0.20000 Build Number 3634791. Just the past few days we’ve had 70+ failures a night as mentioned in this thread. (“Nothing has changed” as far as we know) We are backing up about 500+ VM’s nightly with the main backup group having 400+ members. All the groups were kicking off at the same time. So as of today I have split the 400+ group into 3 groups of about 135 VMs each and then staggered the start time 2 hours from each other. I’ll see how things go tonight.

Glad this was able to help you!

Be sure to reply back with any changes or experiences that might help the next person experiencing this.

Thanks!

If you note significant numbers of sessions failing with snapshot creation errors, L2 might referr to something like a "transactional limit" or "session state limit".  There's a tunable parameter to increase this, and VMware might suggest increasing it, restarting vCenter and re-running your backups.  While this only reduced our failures by a handful, it was very informative for showing us we needed more CPU on our vCenter instance.

Let us know if you learn anything that helps your situation!

Splitting up my backup jobs and staggering the start times had no effect on the amount of failures. I’ve escalated the case within EMC and we are digging into it further.

Just today, they have found a correlation in the vpxd logs, the sso logs and the Avamar logs and have deemed it to be a VMware issue. VMware is now escalating it on their side.