Skip to main content
Start a Conversation

Unsolved

This post is more than 5 years old

caii-marcosa

8 Posts

36369

March 29th, 2010 13:00

vmware image backups failing - error code 10013

I'm trying to get the VMware image backups working...

And in fact, if I shut down the VM I'm trying to back up, it works just fine.

However, if the VM is running, I get errors in vSphere saying it's unable access vmname/vmname.vmx... and also vmname/vmname.nvram

Then it fails. The Activity log in the avamar administrator shows "Cancelled" and "10013" as the error code.

I tried manually copying this file to another location on the same datastore using the vSphere client, and it also fails with the same error (in vSphere)... "unable to access file..."

If I manually snapshot a VM using the vSphere client, the snapshot works just fine.

Any help on this would be appreciated.

rpervan

266 Posts

April 27th, 2010 01:00

Symptom:
The Avamar VMware image level backup fails with the following error message in the backup client session log:
avvcbimage Error <0000>: vmx file is suspiciously small (under 30 bytes), please examine the log on the Avamar Administrator for root cause analysis

High Level Cause:
The high level cause for why this message is reported is that the .vmx file was not able to be backed up.

Four Possible Root Causes:
There are 4 different possible root causes for why the .vmx file was not able to be backed up.  It is important to examine the system to determine which of these 4 possible root causes might be applicable:

Possible Root Cause #1: Failed to get VMware vCenter connection
Description:
One possible reason why the .vmx file did not get backed up is because the Avamar Administrator Server (also known as the Management Console Server, or MCS) was not able to connect to the VMware vCenter server to coordinate the backup of the .vmx file.
In v5.0.0.4xx, the MCS requests that vCenter allocate a pool of 3 connections for its use.  These connections are required for short periods of time at various points throughout the image level backup for each backup proxy server, for example, to create the snapshot, to initiate the backup, to delete the snapshot, etc.  If the Avamar system is performing image level backups concurrently through multiple different backup proxy servers, it's possible that 3 connections might not be adequate.
There is a hard limit on the number of available connections to a vCenter. The limit is 15 if the vCenter is running on a 32 bit Windows OS and 30 if running on a 64 bit OS. Instances of the standard VMware vSphere Client GUI consume connections from this limited pool so Avamar consumption of connections is subject to interaction with customer usage patterns. Source:  http://www.vmware.com/pdf/vsphere4/r40/vsp_40_config_max.pdf
Symptom:
The symptom of this issue is that the following message will be reported in the /usr/local/avamar/var/mc/server_log/mcserver.log.* files on the Avamar grid's utility node:
WARNING: FAILED TO GET VC CONNECTION

Workaround/Solution:
a) In v5.0.0.4xx, the workaround is to increase the "connection_pool_size" parameter in the /usr/local/avamar/var/mc/server_data/prefs/mcserver.xml file on the Avamar grid's utility node.  We recommend increasing this value from the default of 3 to 5.
The trade-off of making this value too large is that this consumes connections that might be required by other systems that connect to the vCenter server.  Therefore, do not increase this value beyond 5 unless you know that the value needs to be increased.
b) In the upcoming v5.0 SP1 release, the default value will be set to 5 connections.  If the MCS runs into an issue where it fails to get a vCenter connection, then the MCS will automatically request additional connections, as required.
Possible Root Cause #2: Unable to retrieve .vmx file because "Datacenters" is not at root level of vCenter node.
Description:
Even if the MCS was able to connect to the vCenter server, then the reason why the .vmx file might not get backed up is because of either or both of the following possible issues:
a) If the Datacenter where the .vmx file is located is not at the root vCenter folder, then the .vmx file will not get backed up.  This is described in Avamar bug 17366.  The issue is that the v5.0.0.4xx version of the Avamar software assumes that the Datacenter is located directly at the root level of the vCenter node.
b) If the root folder internal name, as shown in the vCenter Managed Object Browser, is not "Datacenters", then the .vmx file will not get backed up.  This is described in Avamar bug 17591.  This issue could also be caused if, for example, the Datacenter name is not in English but is in another language.

Workaround/Solution:
a) In v5.0.0.4xx, the workaround is to apply hotfix 17760 to both the MCS and the MC client.
b) Upgrade the Avamar grid to the upcoming v5.0 SP1 release.
Possible Root Cause #3: Unable to retrieve .vmx file because of special characters in datastore, folder, and/or datacenter names
Description:
Another possible cause for why Avamar is unable to retrieve the .vmx file for backup is because there might be "special characters" in the datastore, folder, and/or datacenter names.  This is a limitation of the vCenter server, and is tracked by VMware under VMware Support Request SR# 1483856391.  This issue is also being tracked under Avamar bug 17608.
The following is a list of the "special characters" (in the format of "character"/"escape sequence" format) that will cause the .vmx file to fail to be backed up:
&   %26
+   %2B
/   %2F
=   %3D
?   %3F
%   %25
\   %5C
~   %7E
]   %5D

Workaround/Solution:
a) The workaround is to rename the datastore, folder, and/or datacenter names to avoid the use of these special characters.
b) The solution is to apply a fix from VMware when it is available.
Possible Root Cause #4: Unable to retrieve .vmx file because the ESX server has an exclusive lock on the file
Description:
In ESX 4.0, the ESX server maintains an exclusive lock on the .vmx and nvram files.  Therefore, if the vCenter attempts to use another ESX server (other than the one that has the exclusive lock on the file) to back up the files, the other ESX server will not be able to read the .vmx and nvram files to back them up.  Because of the algorithm used by vCenter to select the ESX server that is used to attempt to read the .vmx and nvram files, the probability of encountering this issue increases as the number of ESX servers managed by vCenter increases.
This issue is tracked by VMware under VMware Support Request SR# 1483856391.  This issue is also being tracked under Avamar bug 17608.

Workaround/Solution:
a) The workaround is pass the following option into the avvcbimage plugin: --x22=8192.  This can be passed in through the dataset by setting:
Attribute: [avvcbimage]x22
Value    : 8192
Alternatively, you can create an avvcbimage.cmd file in the Avamar var directory on the client that contains one line:
--x22=8192

This allows the backup to complete even if the vmx file (that contains the VM configuration) is not backed up.

Because the VM configuration information is NOT backed up (if the .vmx file is 0 bytes), the following will apply during the restore process: The restore for this backup will work for restore to original and restore to existing, but not for restore to new. To do restore to new, you need to manually create a VM with the same configuration as the original VM, and do restore to existing to the newly created VM.
Be advised that restore to a manually created VM will issue new virtual NIC MAC addresses and new virtual disk serial numbers which may cause license activation issues with VMs running Windows.
b) The solution is to apply a fix from VMware when it is available, and to upgrade the Avamar grid to v5.0 SP1.  Avamar v5.0 SP1 has been modified to include retries for retrieving the vmx file, which will be required to accommodate the upcoming fix from VMware.

manivas

18 Posts

May 25th, 2010 22:00

This issue is tracked by VMware under VMware Support Request SR#  1483856391.  This issue is also being tracked under Avamar bug 17608.

Any update on this?.

markaldridge

9 Posts

May 26th, 2010 08:00

I have started the RPQ process with EMC to qualify the ISCSI storage. In the mean time we wait for the VMware patch im afraid.

matej_felle

41 Posts

June 9th, 2010 01:00

We have the same problem at our customer .....

This is VMware ESX 4 issue ....  It will be fix in VMware ESX 4 update 2 ....

BR

jboitel

18 Posts

June 17th, 2010 09:00

I updated my ESX4 to update2 and now, the image-level backup doesn't work !

When I backup a VM,  I can see a "reconfigure" of my VM in Virtual Center, and after nothing ! no snapshot...

The Avamar activity show a "Time Out Response !"

Before, in ESX4 Update 1, the backups were almost OK (sometimes, I had locked vmx, but it worked better).

Somebody who has an idea ???

rpervan

266 Posts

June 21st, 2010 05:00

Possible causes for "vmx file is suspiciously small (under 30 bytes)" error message in Avamar VMware image level backup client session log
                
ID: esg111924
Use Count: 18
Date Created: 03/09/2010
Date Modified: 06/18/2010
Product(s): Avamar Server
Category(ies): Debugging / Troubleshooting
Status: Approved
Related Bugs: 17366, 17591, 17608
SOLUTION
Symptom:
The Avamar VMware image level backup fails with the following error message in the backup client session log:
avvcbimage Error <0000>: vmx file is suspiciously small (under 30 bytes), please examine the log on the Avamar Administrator for root cause analysis

High Level Cause:
The high level cause for why this message is reported is that the .vmx file was not able to be backed up.

Four Possible Root Causes:
There are 4 different possible root causes for why the .vmx file was not able to be backed up.  It is important to examine the system to determine which of these 4 possible root causes might be applicable:

Possible Root Cause #1: Failed to get VMware vCenter connection
Description:
One possible reason why the .vmx file did not get backed up is because the Avamar Administrator Server (also known as the Management Console Server, or MCS) was not able to connect to the VMware vCenter server to coordinate the backup of the .vmx file.
In v5.0.0.4xx, the MCS requests that vCenter allocate a pool of 3 connections for its use.  These connections are required for short periods of time at various points throughout the image level backup for each backup proxy server, for example, to create the snapshot, to initiate the backup, to delete the snapshot, etc.  If the Avamar system is performing image level backups concurrently through multiple different backup proxy servers, it's possible that 3 connections might not be adequate.
There is a hard limit on the number of available connections to a vCenter. The limit is 15 if the vCenter is running on a 32 bit Windows OS and 30 if running on a 64 bit OS. Instances of the standard VMware vSphere Client GUI consume connections from this limited pool so Avamar consumption of connections is subject to interaction with customer usage patterns. Source:  http://www.vmware.com/pdf/vsphere4/r40/vsp_40_config_max.pdf
Symptom:
The symptom of this issue is that the following message will be reported in the /usr/local/avamar/var/mc/server_log/mcserver.log.* files on the Avamar grid's utility node:
WARNING: FAILED TO GET VC CONNECTION

Workaround/Solution:
a) In v5.0.0.4xx, the workaround is to increase the "connection_pool_size" parameter in the /usr/local/avamar/var/mc/server_data/prefs/mcserver.xml file on the Avamar grid's utility node.  We recommend increasing this value from the default of 3 to 5.
The trade-off of making this value too large is that this consumes connections that might be required by other systems that connect to the vCenter server.  Therefore, do not increase this value beyond 5 unless you know that the value needs to be increased.
b) In the upcoming v5.0 SP1 release, the default value will be set to 5 connections.  If the MCS runs into an issue where it fails to get a vCenter connection, then the MCS will automatically request additional connections, as required.
Possible Root Cause #2: Unable to retrieve .vmx file because "Datacenters" is not at root level of vCenter node.
Description:
Even if the MCS was able to connect to the vCenter server, then the reason why the .vmx file might not get backed up is because of either or both of the following possible issues:
a) If the Datacenter where the .vmx file is located is not at the root vCenter folder, then the .vmx file will not get backed up.  This is described in Avamar bug 17366.  The issue is that the v5.0.0.4xx version of the Avamar software assumes that the Datacenter is located directly at the root level of the vCenter node.
b) If the root folder internal name, as shown in the vCenter Managed Object Browser, is not "Datacenters", then the .vmx file will not get backed up.  This is described in Avamar bug 17591.  This issue could also be caused if, for example, the Datacenter name is not in English but is in another language.

Workaround/Solution:
a) In v5.0.0.4xx, the workaround is to apply hotfix 17760 to both the MCS and the MC client.
b) Upgrade the Avamar grid to the upcoming v5.0 SP1 release.
Possible Root Cause #3: Unable to retrieve .vmx file because of special characters in datastore, folder, and/or datacenter names
Description:
Another possible cause for why Avamar is unable to retrieve the .vmx file for backup is because there might be "special characters" in the datastore, folder, and/or datacenter names.  This is a limitation of the vCenter server, and is tracked by VMware under VMware Support Request SR# 1483856391.  This issue is also being tracked under Avamar bug 17608.
The following is a list of the "special characters" (in the format of "character"/"escape sequence" format) that will cause the .vmx file to fail to be backed up:
&   %26
+   %2B
/   %2F
=   %3D
?   %3F
%   %25
\   %5C
~   %7E
]   %5D

Workaround/Solution:
a) The workaround is to rename the datastore, folder, and/or datacenter names to avoid the use of these special characters.
b) The solution is to apply a fix from VMware when it is available.
Possible Root Cause #4: Unable to retrieve .vmx file because the ESX server has an exclusive lock on the file
Description:
In ESX 4.0, the ESX server maintains an exclusive lock on the .vmx and nvram files.  Therefore, if the vCenter attempts to use another ESX server (other than the one that has the exclusive lock on the file) to back up the files, the other ESX server will not be able to read the .vmx and nvram files to back them up.  Because of the algorithm used by vCenter to select the ESX server that is used to attempt to read the .vmx and nvram files, the probability of encountering this issue increases as the number of ESX servers managed by vCenter increases.
This issue is tracked by VMware under VMware Support Request SR# 1483856391.  This issue is also being tracked under Avamar bug 17608.

Workaround/Solution:
a) The workaround is pass the following option into the avvcbimage plugin: --x22=8192.  This can be passed in through the dataset by setting:
Attribute: [avvcbimage]x22
Value    : 8192
Alternatively, you can create an avvcbimage.cmd file in the Avamar var directory on the client that contains one line:
--x22=8192

This allows the backup to complete even if the vmx file (that contains the VM configuration) is not backed up.

Because the VM configuration information is NOT backed up (if the .vmx file is 0 bytes), the following will apply during the restore process: The restore for this backup will work for restore to original and restore to existing, but not for restore to new. To do restore to new, you need to manually create a VM with the same configuration as the original VM, and do restore to existing to the newly created VM.
Be advised that restore to a manually created VM will issue new virtual NIC MAC addresses and new virtual disk serial numbers which may cause license activation issues with VMs running Windows.
b) The solution is to apply a fix from VMware when it is available, and to upgrade the Avamar grid to v5.0 SP1.  Avamar v5.0 SP1 has been modified to include retries for retrieving the vmx file, which will be required to accommodate the upcoming fix from VMware.

nwilliams2

2 Posts

July 8th, 2010 13:00

Confirmed that this worked for me.

The workaround is pass the following option into the avvcbimage plugin:  --x22=8192

Hormigo

131 Posts

July 8th, 2010 15:00

Hi,

Resolved all problems with this configuration:

Update all for:

VMWare vSphere 4 Update 2 (build 258672)

Avamar Server 5.0.2.41

Avamar VM Proxy 5.0.105-169  (Included with SP2 on the Grid)

Regards,

Hormigo

Allbertik

38 Posts

January 31st, 2014 01:00

Hello, i have problem too :

2014-01-31 08:38:43 avvcbimage Error <12016>: vmx file is suspiciously small (under 30 bytes), please examine the log on the Avamar Administrator for root cause analysis (Log #2)

2014-01-31 08:38:43 avvcbimage Error <9760>: Backup of VM metadata failed. (Log #2)

2014-01-31 08:38:47 avvcbimage Error <9768>: Avtar exited with 'code 163: externally cancelled' (Log #2)

If I use parameter x22=8192 don't work too.

Avamar Administrator 6.1.2-47

Avamar Combined Proxy-linux-sles11_64-6.1.102-47

vSphere 5.5.0

pawankumawat

355 Posts

January 31st, 2014 04:00

Hello Baa,

This issue may occur from different causes. Adding parameter x22=8192 may resolve issue if this issue is caused by Possible Root Cause #4: Unable to retrieve .vmx file because the ESX server has an exclusive lock on the file as mentioned in earlier posts here.

I would suggest you to please open a SR with EMC support to get this issue resolved.

Regards,

Pawan

View More

View All

No Events found!

Top