Avamar Issue ESG127567 CBT Problems

QA testing has passed and hopefully the hotfix should be out today.

One of our engineering folks has designed below script. Use below script to find out if there are any bad backups on the GRID

THE SCRIPT HAS BEEN REMOVED, SINCE THIS IS INTENDED TO BE USED BY INTERNAL SUPPORT.

This code will find backups that have a vmdk size that is not a multiple of 512 bytes, and is likely created by a CBT backup that suffers from the issues identified by this ETA.

The script will identify these backups, and offer to reset the status-code such that these (bad) backups will no longer be candidates for CBT - thereby forcing a new, non-CBT backup for affected vms

Please note that this tool will not detect if the backups are affected by the disk re-ordering issue, and this tool could, with ~1% probability, misreport an invalid-sized backup as have a valid size.  Therefore, there are limitations associated with this tool.

Thanks

Sameer Khan

The script Sameer posted is intended to be used by support only. The support team has been specifically trained on the use of the script and its caveates and they can work with you to identify affected backups (noting the ~1% probability of a false negative mentioned by Sameer above).

If you wish to use the script, please contact support for assistance.

Hello,

is the hotfix out now?

If not, should we open a support case to find out which VMs are affected and on which to turn od CBT?

The hotfix is not out yet and I was hit with this last week. Was unable to restore a VM. Found out that all 6 weeks of backups on this server were corrupt. The only way that I found out which VM’s were impacted (by the way I have 9) was to select the restore option, browse the files since I’m using the file level backup and look at each drive letter to see if there is valid data. Each of the 9 servers that I mentioned below had blank disks either the system disks (drive C) or the data drive (drive D). I will see if I can attach an image for you.

I opened a case with EMC and was told that until this is released, to disable CBT on all VM Backups. Here’s the actual response:

Symptoms

If a VMware image is backed up via Avamar using change block tracking (CBT), there is a chance that the backups might not be able to be restored--either through the file level recovery (FLR) functionality or as a full image restore.

As an example, when attempting to mount the backed up VMware image during a file level recovery operation, the attempt might fail with the following reported error message: "Failed to mount the vmimage backup. Possibly due to corrupted and/or an incomplete backup."

Impact

If the VMware image backups are affected by this issue, then the data in the backups might not be able to be restored.

Resolution/Workaround

To prevent this issue from occurring, the recommended containment action is to disable CBT on all VMware image backups until hotfixes are available to address this issue. Note that if CBT is disabled in some environments, it might be necessary to extend the backup window and/or deploy additional proxies to ensure that all backups complete within the backup window.

EMC is currently in the process of developing a resolution for these two causes of these issues. The resolution will involve applying a hotfix to the Avamar Administrator server on the grid utility node and updating the VMware image proxy server software.

========================================================

I have NOT been able to identify any consistency in the VM environment or OS on the servers. Each of the 9 are different in one way or another.

Still waiting for the fix……

Sheila

Please keep an eye on the technical advisory as it will be updated with additional information. I've included the link below. Unfortunately I can't provide more specifics on timelines.

ETA esg127567: Avamar Client for VMware, Avamar Server: VMware image backups created with change block tracking (CBT) enabled might not be able to be restored

http://solutions.emc.com/emcsolutionview.asp?id=esg127567

A customer-installable version of the script that Sameer had posted previously will be made available soon. The updated version of the script is be able to detect all cases of this issue including the ~1% of cases that would have been missed by the previous version of the tool.

It is not likely that there is a pattern for which VMs are affected by the disk size issue and which are not because the issue is triggered by the actual content of the disk flat file(s).

If you have any other questions, please don't hesitate to post or send me a PM.

5 minutes ago I did a Live Chat with EMC support regarding this issue with CBT. We have about 300 VMs here and it would be great to find out which VMs should be disabled for CBT, hopefully not all.

Very funny that they don´t know about the CBT problem or any script to find out which VMs are affected.....

So they will transfer it to the core Avamar VMware group and I should get a callback tomorrow.....

Let´s see what happens....

The ETA has been updated. The customer installable script and the instructions on how to use it are now available as part of the Avamar Customer Procedure Generator. You can retrieve the procedure generator from PowerLink:

Home > Support > Product and Diagnostic Tools > Procedure Generators > Avamar CUSTOMER Procedure Generator

It looks like not all the PowerLink servers are in sync yet. If you get a 404 Not Found message, please try again later.

Hello,

I have tried this script and it works very well and is very easy to execute and understand.

Thanks or that!!!

The hotfix for 5.0.4.30 + 5.0.4-906 and the hotfix for 6.0.1-66 have been released. The customer procedure generator has been updated with instructions on how to install the required fixes.

The hotfixes for 5.0.3-29, 6.0.0-592 and 6.0.0-65 are still pending.

If you run into any issues, please let me know.

Thanks Dan! I'll confirm with the developers and get the doc updated appropriately.

FYI, I went thru the hotfix procedure yesterday, but found out there is a mistake in the document that the procedure generator creates. The last step, to update the Avamar Administrator GUI mcclient.jar file, says to copy mcclient.jar from the utility node into your client installation path, but doing so caused an error when launching Avamar Administrator.

You actually need to copy mcclient_unsigned.jar from the utility node, rename it to mcclient.jar, then copy it to the lib folder on your admin client. Support confirmed this for me yesterday, and said they would get the document corrected.

Otherwise the procedure went off without a hitch.

The updated procedure will be pushed out this afternoon.

The fix was pulled due to a procedural error. There are some incremental improvements coming to the fix packages that will make them easier to install.

correction.... the fix is now available again ! Let's try...

Yesterday, I tried to apply the 2 fixes, but found out that one fix (Hotfix35622-6.0.1-901.avp) was removed from the ftp servers.
Then I contacted support through live chat, and they told me the following:

>>>>>"This hotfix was released but it could not resolve the issue completely, engineering is aware of the issue and they are re-working on it"

So, now waiting for a new fix !