Error Code 10055 fault InvalidLogin does not have reason

Ok. After a couple of WebEx’s with EMC here is the latest.

When looking at the VMs log for the failed backup(s) there are the usual errors and exceptions in Red and this is what we are generally drawn to. However right above one of these entries is something one of the EMC guys found:

2016-10-19 10:03:26 avvcbimage Warning <16041>: VDDK:VixDiskLibVim: Login failure. Callback error 3014 at 2439.
2016-10-19 10:03:26 avvcbimage Info <16041>: VDDK:VixDiskLibVim: Failed to find the VM. Error 3014 at 2511.
2016-10-19 10:03:26 avvcbimage Info <16041>: VDDK:VixDiskLibVim: VixDiskLibVim_FreeNfcTicket: Free NFC ticket.
2016-10-19 10:03:26 avvcbimage Info <16041>: VDDK:VixDiskLib: Error occurred when obtaining NFC ticket for: servername/serverdisk.vmdk. Error 3014 (Insufficient permissions in the host operating system) (fault InvalidLogin, type VmodlVimFaultInvalidLogin, reason: (none given), translated to 3014) at 4526.

2016-10-19 10:03:26 avvcbimage Info <16041>: VDDK:VixDiskLib: VixDiskLib_OpenEx: Cannot open disk servername/serverdisk.vmdk. Error 3014 (Insufficient permissions in the host operating system) (fault InvalidLogin, type VmodlVimFaultInvalidLogin, reason: (none given), translated to 3014) at 4680.

2016-10-19 10:03:26 avvcbimage Info <16041>: VDDK:VixDiskLib: VixDiskLib_Open: Cannot open disk servername/serverdisk.vmdk. Error 3014 (Insufficient permissions in the host operating system) at 4718.

2016-10-19 10:03:26 avvcbimage Error <0000>: Failed to connect to virtual disk servername/serverdisk.vmdk (3014) (3014) Insufficient permissions in the host operating system

2016-10-19 10:03:26 avvcbimage Info <19644>: Connecting virtual disk servername/serverdisk.vmdk

Avamar can’t login to vCenter for some reason, the backup fails. This error is present in all of the failed backups.

So we started to dig into the vpxd.log on the vCenter (appliance in our case) and grepped for “failed” and the AD service account we use for Avamar.

We found over 100+ entries like this:

2016-10-18T23:18:50.794-07:00 error vpxd[7FA110D49700] Failed to authenticate user (actual domain and user names removed to make the security guy happy)

Some of these entries had an time stamp that exactly matched the Avamar client logs for the “VDDK:VixDiskLibVim: Login failure”

When this was found the EMC engineer asked if we were backing up our vCenter appliance with the other production VMs? Yes we are. The next question was are you also backing up your domain controllers with the production VMs? Yes.

He stated that due to the Avamar taking quiesced snapshots there may be a few seconds during which that is happening that either the vCenter server or the AD DC’s are not accepting the login. This would explain why in the AM we can rerun any one of the failed backups and they complete without issue. No other snapshots going on. SAN is quite, everything is responding well.

For tonight’s backup window we removed the vCenter appliance and all of the AD DCs from the backups. We’ll see what happens overnight.

I’ll update you all tomorrow.

Did you ever find a solution with VMware? We have had a couple of cases open with VMware and they haven't found a solution yet.

Here's what they had to say.

IDM maintains a TenantCache which stores information about the tenant so that IDM can avoid repeated queries to vmdir. The key to this cache is the tenant name, as supplied by the login request. In this case, we were logging into "abcvsphere.local". However, when the customer created this tenant, they provided the name "ABCvsphere". As such, "ABCvsphere" was stored in vmdir as the tenant name. When writing to the cache, the tenant name stored in vmdir is used as the key, in this case "ABCvsphere". I'll walk through an example to illustrate what happened:

- In many cases this will result in a POST to /sts/STSService/odc.local. Notice the tenant name is "abcvsphere.local"

- IDM tries to query the TenantCache with key "abcvsphere.local". The cache is empty, this is expected since it is the first login.

- IDM queries vmdir several times to get the tenant information. Most LDAP queries are case insensitive, so lookups for tenant "abcvsphere.local" will return the data for "ODC.local."

- Now that IDM has the tenant information, it will be cached with key "ABCvsphere.local". The key to the cache is the tenant name as stored in vmdir.

- Subsequent logins will result in IDM querying the cache with key "abcvsphere.local". By default, Java String comparisons are case sensitive, so the cache will appear empty, and IDM will repeat the same process of querying vmdir.

- Since every authentication request results in a cache miss, many extra LDAP connections are created for each request. Eventually, so many LDAP connections are created that the machine exhausts the ephemeral port space. This in turn results in some authentication requests failing since IDM cannot establish a connection to vmdir.

The fix for the port exhaustion is supposed to be available in v6.0u3 out in January.

Have you tried using administrator@vsphere.local or another vsphere account instead of a AD account for Avamar authentication to vCenter?  One time I ran into an intermittent issue similar to what is described in this post and that was a sufficient resolution for me.

proxycp.jar has an option to test vCenter connection and I used it to troubleshoot the issue.  The issue I ran into when I used an AD account the proxycp.jar vcenter connection consistently failed about half the times I tried it.  vSphere account worked every time.

The 6.5 VDDK release notes mention a new feature for reusing vCenter server sessions.  Possibly this future VMWare feature is designed to resolve the issue mentioned in this post.  VDDK is the API that Avamar uses for VMWare backups.

We have encountered this same issue (error code = 10055), we rebooted the host at vcenter and it corrected the problem. I hope this helps.