Avamar and VMware CBT Bug

there is no way in h3ll i am going to disable/enable CBT for the whole environment, i have over thousand VMs. You realize what disabling/re-enablign CBT does right ? Next backup is full, not doing that for thousand VMs.

that's the thing, you don't know what VMs are impacted unless you know what VMs were expanded from < 128G to > 128G and then you run powercli commands against those specific VMs to disable/re-enable CBT. What a mess.

Yup, you are right, for the entire environment it is not a logical fix, for failed VMs it can be used, unless the entire environment is affected, I hope they find a fix for it soon so the workaround could be avoided.

Please see the technote I’ve linked and copied down below.  Looks like disabling and re-enabling CBT is a workaround for the potentially impacted VM’s.  Let us know if you have any questions.

  http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2090639

QueryChangedDiskAreas command returns incorrect sectors after extending virtual machine vmdk file with Changed Block Tracking (CBT) enabled (2090639)

Symptoms

When you run the command QueryChangedDiskAreas("*") to return a list of allocated disk sectors, you experience these symptoms:

• Allocated portions of the virtual machine disk vmdk file are not returned.
• The list of allocated virtual machine disk sectors returned is incorrect.

Cause

  This issue occurs on a virtual machine with Changed Block Tracking (CBT) enabled, when extending a virtual disk (vmdk) file to a size strictlyabove 128 GB, due to the block tracking information being recalculated incorrectly.

Note: A virtual machine is at risk if all of these occur in sequence: 

• Changed Block Tracking is enabled.
• Virtual disk is extended to a size above 128 GB.
• An incremental backup is taken, using QueryChangedDiskAreas.
• The virtual machine is restored from that incremental backup.

Resolution

This is a known issue affecting VMware ESXi 4.x and ESXi 5.x.

Currently, there is no resolution.

To work around this issue, disable and then enable Changed Block Tracking (CBT) on the virtual machine.

For more information, see Enabling Changed Block Tracking (CBT) on virtual machines (1031873).

Additional Information

To be alerted when this article is updated, click Subscribe to Document in the Actions box. 

See Also

1. 1.        Enabling Changed Block Tracking (CBT) on virtual machines (1031873)

Request a Product Feature

To request a new product feature or to provide feedback on a VMware product, please visit the Request a Product Feature page.

Feedback

I think, if a VM fails due to change in the disks, one can disable CBT on that VM and run a backup, once the backup completed successfully, the CBT can be enabled again. I have had similar issues before where a datastore was updated on a VM that is protected on Avamar and I used the same.

Going to Policy tab, select Clients, browse to the affected VM, Select the VM, click "edit" go to VMware tab, uncheck Change block tracking.

after backup same steps and check Change block tracking.

dynamox schreef: there is no way in h3ll i am going to disable/enable CBT for the whole environment, i have over thousand VMs. You realize what disabling/re-enablign CBT does right ? Next backup is full, not doing that for thousand VMs.

How does forcing a full backup help you in the long run? Will it reset the CBT? Because otherwise after the full backup, CBT will still not detect changes to data above the 128GB.

because of the bug, your incremental backups of a disk that was expanded to > 128g could be garbage. By resetting CBT you are forcing a full backup of that VM.

Ah wait. I read your post incorrectly. I thought you were NOT going to reset CBT and just perform a full backup and solve your issue with that. That wouldn't fix it :-)

And to be exact, the backup will not make a full backup but only make a full scan !! All data that is already in the backup will not be transfered again.

To be honest, I'm a little concerned as to the lack of 'airtime' this issue is getting from all vendors.

Can someone form the Avamar PG confirm whether the "QueryChangedDiskAreas()" api call is used for CBT enabled incrementals ?

I first became aware the issue early last week. After doing some research there appeared to be only one workaround, as stated in this discussion, of doing the disable/enable cbt and full backup 'dance'.

Like Dynamox, I had concerns about applying this to my many hundreds of vm's and wearing the penalties of doing this. I also have some vm's that don't currently have CBT enabled and didn't want them changed.

I made a quick little variation of other script pieces which is a little more precise by only targetting vm's that

a) have CBT already enabled and b) have vmdk's attached that are over 120GB.

#---------------------------------------------------------------------

  $targetvms=get-vm | where {$_.ExtensionData.Config.ChangeTrackingEnabled -eq $true -AND $_.HardDisks.CapacityGB -gt 120}

$vmspec = New-Object VMware.Vim.VirtualMachineConfigSpec

$vmspec.ChangeTrackingEnabled = $false

foreach($vm in $targetvms){

$vm.ExtensionData.ReconfigVM($vmspec)

$snap=$vm | New-Snapshot -Name 'CBT enable kb2090639'

$snap | Remove-Snapshot -confirm:$false}

#---------------------------------------------------------------------

(this will also take and then remove a snapshot to stun the vm and commit the config change and runs sequentially)

There's lot's of other ways to limit the scope by changing the leading "Get-vm" by leading it from a "get-cluster" or "get-resourcepool" pipeline.

I've included more details in a blog post, which I plan on updating as/if vendors make further statements.

VMware ESXi CBT Bug – Your data may be at risk | Pragmatic IO

Hope this is useful to some of you..

I can confirm that Avamar uses the affected API call. At the moment, the only known workaround is to disable and re-enable CBT as described above.

The Engineering team is working on a remediation plan. I can't share a lot of details but the idea is to find all the potentially affected VMs and apply the workaround to a limited number each day.

Incidentally, does anyone know of a way to query the Avamar database for a list of VMs that have CBT enabled?  Our VMware team limits the Storage team's privileges in vCenter, so I'm not sure if the script check P@rity99 is using works for me or not.

Thanks!

Karl

java -jar proxycp.jar --cbtstatus

I realize this post is a little old, but I see VMware has finally released a patch for version 5.5.  Does anyone know the best way to fully remeidate this issue when using Avamar?  Apply the patch of course, but can we then just do rprnairj's suggestion:

"Going to Policy tab, select Clients, browse to the affected VM, Select the VM, click "edit" go to VMware tab, uncheck Change block tracking.

after backup same steps and check Change block tracking."

Or is there a patch for the Avamar software that will reset CBT automatically as needed?

Or do we need to do the manual reset via the process outlined in the VMware KB?

Thanks,

Bob

guys can anyone confirm that this bug was fixed in ESXi 5.5 The KB mentions this patch ..but it does not explicitly say that it addressed that specific bug.

• • VMware ESXi 5.5, Patch Release ESXi550-201501001 (2099265)

dynamox wrote: guys can anyone confirm that this bug was fixed in ESXi 5.5 The KB mentions this patch ..but it does not explicitly say that it addressed that specific bug. VMware ESXi 5.5, Patch Release ESXi550-201501001 (2099265)

Yes, confusion reigns !  there was a patch released on Jan 27 which claimed to have resolved the issue. http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2099271

However, the original KB still shows that esxi 4.x and 5.5.x are affected. VMware KB: QueryChangedDiskAreas API returns incorrect sectors after extending virtual machine VMDK file with Change…

I've reached out to our VMware TAM but haven't gotten confirmation or clarification yet.