Since implementing Avamar, I'm getting an error on several VMs similar to the following:
Error Code: 10007
I've done some research, but I'm unable to get a clear picture of where the problem lies. It looks like it could be a problem with change block tracking, so I disabled that, created a snapshot of a couple of VMs, consolidated the snapshots and deleted them, but still no joy.
I'm running Avamar 6.1.0 - 402 and ESX 5.1.
If anyone has encountered this and has any suggestions, I would love to hear them. Please let me know if you need any other information. Thanks.
Eric Marriott
A bit more information on this. The log above was obtained from Administrator / Activity
Below is what shows in the event detail under Administrator / Administration / Event Management:
The data section contains the client's hostname. Either the client machine has been assigned a new hostname, or an attempted security breach has been detected. If this client has been assigned a new hostname, first use the Policy dialog to unactivate the client, then re-activate the client, which will update the client list to reflect the new hostname.
I could easily take those steps on a physiscal server, but am unsure how to go about doing this on a wide scale for several VMs, while preserving backups.
The problem with VM backup could be with umlauts characters like (ü, ä, ö) or (~!@$^%(){}[]|,`;#\/:*?<>'"&) or path limitation to the 255 character length.
Could you post the backup log for a job with this error or at least the portion of a log that shows this error? Events alone can be a little misleading but the logs usually tell more of the story. If you don't want to post the log, search it for "error" and "warning" to find possible areas of interest.
I am experiencing the same issue I believe. After upgrading to 6.0.1-66 to 6.1.0-402 I am experiencing errors in vSphere 5.0 U1 & Avamar. It looks like if fails quiescing the server and then runs the snapshot again with quiescing off?
I'm going to leave this thread open because it appears that Brett is having similar issues, but I've been able to sort out the majority of the issues. It appears that the 10007 message is fairly generic in nature. Going to the Administrator / Administration Console and viewing the Event Management log appears to provide some helpful details (you will need to open the error and scroll down to the details section at the bottom).
I've found several things in my case.
1. VM snapshots may be present and this will prevent the backup. Consolidate them, or select Delete All, which will consolidate the snaps and then delete them.
2. Create another snapshot within VMWare and then select the Delete All button. For some reason, this deletes snapshots that don't really appear to be there. Yeah, I'm a bit confused too, but it seemed to work.
3. Make sure there are no Physical RAW device mappings. Sadly, we have a ton of these.
4. Check the free snapshot space and size of any .vmdk files. I had a 400 GB file that picked a fight between Avamar and our EQ SAN. No winner on this one yet.
5. If the snapshot can't be quiesced, check the I/O. There just might be too much activity for the snap to take place, or it could indicate an underlying performance problem with a physical drive / SAN.
Hope this help and thanks to all for the feedback.
I will try the manual create/delete snapshot sometime today. I am noticing this in the Windows event log, but while I do not have a floppy presented to teh VM, it still shows up in Device Manager:
I believe the quiese is enabled for this particular VM, if quiese snapshot is enabled, VMWARE will talk to VSS inside the VM to take the VSS snapshot, but if for any reason VSS inside the VM fails creating the snapshot then Vmware snapshot also fails.
If you are running 6.1 proxy add the below flag and run on-demand backup and tell us the result
It looks like a no quiesce runs automatically after the snapshot fails while trying to quiesce. I am only assuming this based on teh quiesceFS=0 on the very next line where it tries to create teh snapshot again. see the log:
2012-08-09 08:00:56 avvcbimage Info <14631>: create snapshot task still in progress, sleep for 2 sec.
2012-08-09 08:00:58 avvcbimage Info <14631>: create snapshot task still in progress, sleep for 2 sec.
2012-08-09 08:01:00 avvcbimage Warning <0000>: vSphere Task failed (application quiesce): 'An error occurred while quiescing the virtual machine. See the virtual machine's event log for details.'.
2012-08-09 08:01:00 avvcbimage Info <14627>: Creating snapshot 'Avamar-134451361931aa2cdd1597cd8a6ecc0c66ea996ef7f4416edb', quieceFS=0
2012-08-09 08:01:00 avvcbimage Info <14631>: create snapshot task still in progress, sleep for 2 sec.
2012-08-09 08:01:02 avvcbimage Info <14631>: create snapshot task still in progress, sleep for 2 sec.
2012-08-09 08:01:04 avvcbimage Info <14631>: create snapshot task still in progress, sleep for 2 sec.
2012-08-09 08:01:06 avvcbimage Info <14632>: snapshot create completed
2012-08-09 08:01:06 avvcbimage Info <14634>: New snapshot 'Avamar-134451361931aa2cdd1597cd8a6ecc0c66ea996ef7f4416edb', moref is 'snapshot-38298'.
The plugin will retry the snapshot if the first one fails due to some issue with quiescing (VSS failure is the most common cause for Windows guests). This retry will cause your backup to complete with exceptions.
Ideally, it would be best to address the cause of the VSS failure but if this is not possible you can use Amol's suggestion to disable quiescing, meaning the backup is reported as successful instead of completing with exceptions.
I had this same problem with the Avamar jobs completing with exceptions and vcenter throwing some unable to quiesce drive alerts. Our problem was that there was a small builtin partition on each of the vm's that had issues. This partition had to drive letter assigned to it. All I had to do was give it a drive letter. Now, during a backup, there are no errors or alerts in Avamar or vcenter.
rpervan
266 Posts
12260
0
Posted August 7th, 2012 06:00
The problem with VM backup could be with umlauts characters like (ü, ä, ö) or (~!@$^%(){}[]|,`;#\/:*?<>'"&) or path limitation to the 255 character length.
regards,