Highlighted
inkjet2000
1 Nickel

VMDK Backups freezing after snapshot removal

Jump to solution

We're experiencing an issue which is basically summarized in VMWare article 2010953 (http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&externalId=2010953&sl...).

It seems as though bullet two under symptoms stands out the most as I see in all of the backup logs the hotadd method is always selected.

  •    2013-04-11 23:56:26 avvcbimage Info <9675>: Connected with hotadd transport to virtual disk [datastore] /disk/path.vmdk

Our VMWare Datastores are all mounted via the NFS protocol. When taking a backup of the VMDK via the Avamar, we're seeing a large amount of time where the VM is absolutely unresponsive (as high as 120 seconds).

I see that this is due to the Avamar useing the "hot-add" backup method when collecting the VMDK files, and was wondering if EMC Avamar had a work around or hotfix for customers on NFS infrastructures.

The work around in the VMWare article is not acceptable for implementation for several reasons. Is there additional functionality with the Avamar to pull a VMDK with something other than the hot-add method?

We are currently on Avamar 6.1.1-87, ESX 5.1, utilizing NetApp NFS datastores..

Thanks!

Justin

0 Kudos
9 Replies
ionthegeek
4 Ruthenium

Re: VMDK Backups freezing after snapshot removal

Jump to solution

I spoke with the support team and it's possible to disable hotadd and fall back on NBD / NBDSSL but please note that this may have a significant performance impact. You can disable hotadd by specifying the following advanced flag in the dataset:

[avvcbimage]transport = nbd:nbdssl

To set this flag, go to Tools => Manage Datasets, select your dataset and click the Edit button. Go to the Options tab, select the appropriate plug-in from the drop-down list (Windows VMware Image or Linux VMware Image), click the "More" button and fill in the attribute and value.

Enter [avvcbimage]transport for the attribute and nbd:nbdssl for the attribute value. Be careful of typos -- the entries have to be specified exactly and can't have leading or trailing spaces. When you're happy with the setting, click the + sign under the fields and you should see the flag added to the select box. Click OK to save your changes.

inkjet2000
1 Nickel

Re: VMDK Backups freezing after snapshot removal

Jump to solution

For anyone with similar architecture and symptoms, this has enabled us to begin using the VMDK snapshot capabilities without risk of locking a VM. Great answer, thank you!

0 Kudos
dustin_k22
1 Copper

Re: VMDK Backups freezing after snapshot removal

Jump to solution

Ran into the same issue. Avamar Server 6.1.1-87 Avamar VM Proxy 6.1.181-110 vCenter 5.1. When doing the snapshot removal, VM would be down for 120 seconds or more. Didn't have this issue with vCenter 5.0

Applied the above fix, backup now happens real fast and only lose a ping for snapshot and snapshot removal.

Thanks.

0 Kudos
pannujat
1 Copper

Re: VMDK Backups freezing after snapshot removal

Jump to solution

But that means the backups would be very slow. I have 3000 VMs to backup, they would never finish like this? What are other alternatives.

0 Kudos
samatheis
1 Copper

Re: VMDK Backups freezing after snapshot removal

Jump to solution

We've run into this same issue.  I set the flag, but we're still seeing hotadd as an option, and it was selected for the backup:

Dataset Options:

ddr=false

utilize_changed_block_list=true

[avvcbimage]transport=nbd:nbdssl

Did I miss something when entering in the option?

0 Kudos
ionthegeek
4 Ruthenium

Re: VMDK Backups freezing after snapshot removal

Jump to solution

Was it a scheduled backup or an on-demand backup that ran with hotadd? The dataset configuration will not affect on-demand backups.

Did you set the flag for all of your VM Image datasets? If you're using the built-in datasets, keep in mind that there are two -- one for Windows and one for Linux.

0 Kudos
samatheis
1 Copper

Re: VMDK Backups freezing after snapshot removal

Jump to solution

I created a separate dataset with the flag set, confined to the type of VM this is (Windows.)  I had tested with on-demand, but the scheduled backup this morning also had the problem. 

Is it really the hotadd that's the problem?  We see the guest-side issue manifest during the snapshot removal and consolidation, not during the creation or add/removal from the VCB proxy.

0 Kudos
ionthegeek
4 Ruthenium

Re: VMDK Backups freezing after snapshot removal

Jump to solution

Is it really the hotadd that's the problem?  We see the guest-side issue manifest during the snapshot removal and consolidation, not during the creation or add/removal from the VCB proxy.

Is the VM on NFS storage per the VMware KB linked at the top of this thread? If so, this is likely the same issue.

Further troubleshooting will require a look at the logs. If you're comfortable posting them (anonymized) here, please do. If not, I would recommend opening a service request.

0 Kudos
samatheis
1 Copper

Re: VMDK Backups freezing after snapshot removal

Jump to solution

It is on a NFS datastore.  Telling me I need to keep my vcbproxy on the same host as (theoretically) all my VMs is pretty sub-optimal, given the number of hosts and clusters we have involved here.

I've got a SR open on some other issues related to Avamar backups, so I'll tack this one on.

0 Kudos