This post is more than 5 years old
5 Posts
0
9813
VMDK Backups freezing after snapshot removal
We're experiencing an issue which is basically summarized in VMWare article 2010953 (http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&externalId=2010953&sliceId=2&docTypeID=DT_KB_1_1&dialogID=782708998&stateId=1%200%20782738446).
It seems as though bullet two under symptoms stands out the most as I see in all of the backup logs the hotadd method is always selected.
- 2013-04-11 23:56:26 avvcbimage Info <9675>: Connected with hotadd transport to virtual disk [datastore] /disk/path.vmdk
Our VMWare Datastores are all mounted via the NFS protocol. When taking a backup of the VMDK via the Avamar, we're seeing a large amount of time where the VM is absolutely unresponsive (as high as 120 seconds).
I see that this is due to the Avamar useing the "hot-add" backup method when collecting the VMDK files, and was wondering if EMC Avamar had a work around or hotfix for customers on NFS infrastructures.
The work around in the VMWare article is not acceptable for implementation for several reasons. Is there additional functionality with the Avamar to pull a VMDK with something other than the hot-add method?
We are currently on Avamar 6.1.1-87, ESX 5.1, utilizing NetApp NFS datastores..
Thanks!
Justin
ionthegeek
2K Posts
2
April 12th, 2013 13:00
I spoke with the support team and it's possible to disable hotadd and fall back on NBD / NBDSSL but please note that this may have a significant performance impact. You can disable hotadd by specifying the following advanced flag in the dataset:
[avvcbimage]transport = nbd:nbdssl
To set this flag, go to Tools => Manage Datasets, select your dataset and click the Edit button. Go to the Options tab, select the appropriate plug-in from the drop-down list (Windows VMware Image or Linux VMware Image), click the "More" button and fill in the attribute and value.
Enter [avvcbimage]transport for the attribute and nbd:nbdssl for the attribute value. Be careful of typos -- the entries have to be specified exactly and can't have leading or trailing spaces. When you're happy with the setting, click the + sign under the fields and you should see the flag added to the select box. Click OK to save your changes.
inkjet2000
5 Posts
0
April 22nd, 2013 11:00
For anyone with similar architecture and symptoms, this has enabled us to begin using the VMDK snapshot capabilities without risk of locking a VM. Great answer, thank you!
dustin_k22
1 Message
0
December 18th, 2013 13:00
Ran into the same issue. Avamar Server 6.1.1-87 Avamar VM Proxy 6.1.181-110 vCenter 5.1. When doing the snapshot removal, VM would be down for 120 seconds or more. Didn't have this issue with vCenter 5.0
Applied the above fix, backup now happens real fast and only lose a ping for snapshot and snapshot removal.
Thanks.
pannujat
3 Posts
0
January 23rd, 2014 14:00
But that means the backups would be very slow. I have 3000 VMs to backup, they would never finish like this? What are other alternatives.
samatheis
3 Posts
0
January 28th, 2014 17:00
We've run into this same issue. I set the flag, but we're still seeing hotadd as an option, and it was selected for the backup:
Dataset Options:
ddr=false
utilize_changed_block_list=true
[avvcbimage]transport=nbd:nbdssl
Did I miss something when entering in the option?
ionthegeek
2K Posts
0
January 28th, 2014 17:00
Was it a scheduled backup or an on-demand backup that ran with hotadd? The dataset configuration will not affect on-demand backups.
Did you set the flag for all of your VM Image datasets? If you're using the built-in datasets, keep in mind that there are two -- one for Windows and one for Linux.
ionthegeek
2K Posts
0
January 29th, 2014 10:00
Is the VM on NFS storage per the VMware KB linked at the top of this thread? If so, this is likely the same issue.
Further troubleshooting will require a look at the logs. If you're comfortable posting them (anonymized) here, please do. If not, I would recommend opening a service request.
samatheis
3 Posts
0
January 29th, 2014 10:00
I created a separate dataset with the flag set, confined to the type of VM this is (Windows.) I had tested with on-demand, but the scheduled backup this morning also had the problem.
Is it really the hotadd that's the problem? We see the guest-side issue manifest during the snapshot removal and consolidation, not during the creation or add/removal from the VCB proxy.
samatheis
3 Posts
0
January 29th, 2014 10:00
It is on a NFS datastore. Telling me I need to keep my vcbproxy on the same host as (theoretically) all my VMs is pretty sub-optimal, given the number of hosts and clusters we have involved here.
I've got a SR open on some other issues related to Avamar backups, so I'll tack this one on.