Brian91
2 Iron

VADP Backup Issues

Jump to solution

Hi all

We've recently upgraded from NetWorker 7.5.1 to 7.6.3 (Windows 2003 R2 Standard SP2 32-bit).  As our VCB backup proxy was also the backup server itself, we've naturally had to switch from the (fully-working) VCB backup method to the new VADP backup method.

First of all, we were on version 4.0.0 of vSphere which should have been supported, but it turns out that the minimum spec appears to be 4.0.0 Release 2 - annoying!  Fortunately, that got upgraded a week later to version 5, albeit on a new hypervisor (Windows 2008 R2 64-bit).

Some of the VADP backups work great, however we have a dozen or so failing with:

* vm_hostname:*FULL* Could not mount required number of volumes. Exiting.

* vm_hostname:*FULL* The VADP proxy and the VM client are possible clones of each other or derived from the same parent. Please try by setting the transport mode as nbd.

I've come across this http://solutions.emc.com/EMCSolutionView.asp?id=esg123567&usertype=C but our VMware admins can't see any issues with the disk configurations.

I've also added the application information value of VADP_TRANSPORT_MODE=nbd in the client resource for the particular VM, but it seems to ignore this setting as the save command output (please see below) shows:

Using 'san' as the transport mode for Hard disk 1.

The clients that are failing are mainly Windows, both 2003 and 2008 and both 32- and 64-bit, but there's also a couple of Linux as well.  I can't see any commonalities with the failing backups.  Most of the servers have up-to-date VMware Tools

Another issue we're experiencing is that one client fails with Unable to find the VM with VM name: vm_host.fqdn.com.  This is for the weekly backup with a one month retention.  However, the monthly backup with a six year retention works perfectly!  As far as I can see, both client resources are configured identically, except for the browse/retention policies.

Has anyone experienced these issues before?

Thanks

B

Output from savegrp command (from /nsr/tmp):

Initializing VADP Configuration.

Using nsr.backup.server.com as the VADP proxy server host for client vm_hostname.

Application information attribute VADP_HOST for client vm_hostname is set to "something.our.fqdn".

Application information attribute VADP_DISABLE_FLR for client vm_hostname is set to default value of "No".

Application information attribute VADP_BACKUPROOT for client vm_hostname is set to "C:\\VCB_Backups".

Application information attribute VADP_TRANSPORT_MODE for client vm_hostname is set to "san|hotadd|nbdssl|nbd".

Application information attribute VADP_MAX_RETRIES for client vm_hostname is set to default value of "0".

Application information attribute VADP_MAX_BACKOFF_TIME for client vm_hostname is set to default value of "10".

80423:nsrvadp_save: Unknown application information attribute found for client vm_hostname.

Trying to reset the client resource application information attributes.

Application information attribute VADP_HYPERVISOR for client vm_hostname is set to "hypervisor.our.fqdn".

80423:nsrvadp_save: Unknown application information attribute found for client vm_hostname.

VC\ESX Host Name        : 10.10.10.111

VC\ESX Port             : 443

VC\ESX Username         : domain\vcb_backup

Mount point             : C:\\VCB_Backups\vm_hostname

Flavor                  : fullvm

Transport Mode          : san:hotadd:nbdssl:nbd

Max retry               : 0

Max backoff time        : 10

VDDK Install Directory: C:\Program Files\Legato\nsr\plugins\VDDK.

VADP VM-TMP Directory Location: C:\\VCB_Backups\vm_hostname\vm_hostname\vm_hostname-tmp\letters\1.

Temporary Directory for VM created.

VADP TMP Directory Location: C:\\VCB_Backups\vm_hostname\vm_hostname\tmp.

Temporary Directory for VADP created.

Temporary vmMntLoc Directory for VADP created.

Performing the backup via the Virtual Center.

Application information attribute VADP_VM_NAME for client vm_hostname is set to "vm_hostname".

Trying to connect to the VM using VM name 'vm_hostname'.

Performing a FULL Image level backup.

Creating snapshot for vm_hostname.

Creating snapshot for 'vm-75126' ...

Task is -1% complete

Task is 95% complete

Task is -1% complete

SnapShot created Successfully for VM 'vm_hostname'.

Downloading VM configuration for vm_hostname.

Configuration downloaded successfully for VM 'vm_hostname'.

Following are the details of the client vm_hostname:

VirtualCenter        : 10.10.10.111

ESX Server        : physical.our.fqdn

DataCenter Name        : DataCenter Name

ConfigDatastore Name    : VNX_LUN3

Resourcepool Name    : /Cognos

Following is the disk information of the client vm_hostname:

Disk 1 Label        : Hard disk 1

Disk 1 Size        : 72.0 GB

Disk 1 Datastore name    : VNX_LUN3

Total Number of Disks    : 1

Obtained the virtual disks for the VM.

Change block tracking is disabled for VM: vm_hostname

File level Recovery from Image level backup is enabled.

MORef for the VM is obtained.

Obtained the virtual disks for the VM.

Using 'san' as the transport mode for Hard disk 1.

Starting mountvm operation.

Successfully mounted the VM as a file system.

Could not mount required number of volumes. Exiting.

The VADP proxy and the VM client are possible clones of each other or derived from the same parent. Please try by setting the transport mode as nbd.

Cleaningup VADP mount operation.

Deleting snapshot for 'snapshot-100615' ...

Task is -1% complete

Task is -1% complete

Snapshot for VM vm_hostname is deleted.

Temporary directory 'C:\\VCB_Backups\vm_hostname' for VADP deleted.

Tags (3)
0 Kudos
1 Solution

Accepted Solutions
tech88kur
2 Iron

Re: VADP Backup Issues

Jump to solution

Hi,

Updating the status of this issue..

It turned out that the disks types used were a mix of SCSI and IDE (some of them SCSI and the others IDE), this is a known limitation with VADP. Hence the backup failures.

Regards

tech88kur

0 Kudos
7 Replies
Highlighted
Bebo2k
3 Silver

Re: VADP Backup Issues

Jump to solution

Hi,

Make sure that your machines are not configured with independent persistent disks as VADP does not support the backup and recovery of these disks, Also make sure that you are not using Dynamic Disks ( You can check that from the Disk management console inside each machine )

There is space considerations and recommendations needed for the VADP mount point (VADP_BACKUPROOT), which is C:\VCB_Backups in your case, The VADP mount point cache requires space equal to at least 5-10% of the total amount of data being backed up in th e case of Windows VMs. This space is required for temporarily storing the VMDK index during the backup, and is cleaned up once the backup completes.  In the case of Linux or FLR-disabled

Windows VMs, minimal space is required as indicated in the note below.

As an example of how much space is required for a Windows VM:

If the proxy client parallelism is set to 5 so that a maximum of 5 Windows VMs are backed up concurrently, then calculate the total used disk space for the 5 largest Windows VMs in the environment. Allocate at least 10% of this total used space for the VADP_BACKUPROOT mount point.

So, if each VM in the above example has around 2 disks and each disk has 40GB used space.

• Total amount of data being backed up=40GB*2*5=400GB

• Total amount needed for mount point=400*10%=40GB

In this case, ensure that the drive specified for VADP_BACKUPROOT has at least 40GB of free space.

Note: This mount point space is only needed when performing FLR-enabled image level backups of Windows VMs. It is otherwise very minimal (in the order of a few MB per VM) when performing image level backups of Linux VMs or FLR-disabled image level backups of Windows VMs.

Thanks,

Ahmed Bahaa

Bebo2k
3 Silver

Re: VADP Backup Issues

Jump to solution

For SAN mode backups, the VADP proxy requires read access to the SAN LUNs hosting the virtual machines.

Also you have to use the Diskpart utility for SAN and hotadd transport modes , Where an RDM NTFS volume is being used for any of the VMs on the VADP proxy host, Windows will automatically attempt  to mount the volume and assign drive letters to VM disks during backup. This may lead to data corruption on the VMs.To prevent Windows from automatically assigning drive letters to the RDM NTFS, perform the following steps.

Note: Steps 1 and 2 are only applicable in the case of SAN transport mode where SAN fabric zoning is already in place such that the VADP proxy host is already displaying the SAN LUNs in Windows disk management. If this does not apply, skip to Step 3.

1. Shut down the Windows proxy.

2. Disconnect the Windows proxy from the  SAN or mask all the LUNs containing VMFS volumes or RDM for virtual machines.

3. Start the proxy and log into an account with administrator privileges.

4. Open a command prompt and run the diskpart utility by entering the following:

diskpart

The diskpart utility starts and prints its own command prompt.

5. Disable automatic drive letter assignment to newly discovered volumes by entering the following in the diskpart command prompt:

automount disable

6. Clean out entries of previously mounted  volumes in the registry by entering the following in the diskpart command prompt:

automount scrub

Waiting your updates.

Thanks,

Ahmed Bahaa

0 Kudos
Brian91
2 Iron

Re: VADP Backup Issues

Jump to solution

Thanks, Ahmed.

Space for the VADP mountpoint is not an issue, but I'll pass the disk/LUN information to our VMware admins.

Incidentally, the machines that are failing their VADP backup all had successful VCB backups prior to the NW upgrade.  Of course, it's difficult to work out what has broken things since we upgraded NetWorker and vSphere and moved to VADP at essentially the same time.

B

0 Kudos
Brian91
2 Iron

Re: VADP Backup Issues

Jump to solution

Hi all

I've managed to get the client that was failing its weekly backup backup but not its monthly backup working.  I don't remember doing anything to fix it, but it's working nonetheless...

I've also managed to get VADP to use the nbd transport method as specified in the original error.  However, the backup is still failing with the same issue:

Obtained the virtual disks for the VM.

Change block tracking is disabled for VM: vm_hostname

File level Recovery from Image level backup is enabled.

MORef for the VM is obtained.

Obtained the virtual disks for the VM.

Using 'nbd' as the transport mode for Hard disk 1.

Starting mountvm operation.

Successfully mounted the VM as a file system.

Could not mount required number of volumes. Exiting.

The VADP proxy and the VM client are possible clones of each other or derived from the same parent. Please try by setting the transport mode as nbd.

Has anyone come across this before?

Thanks

B

0 Kudos
tech88kur
2 Iron

Re: VADP Backup Issues

Jump to solution

Hi,

I am using nsr 8.0.2.6 on each machine. I am trying to configure VADP backup for a VM having Windows 2003, 32-bit OS. The proxy and storage node are the same VM machine running on Windows 2008 R2, The networker server is running on 2008 R2.

The backup is failing with "

Could not mount required number of volumes. Exiting.

The VADP proxy and the VM client are possible clones of each other or derived from the same parent. Please try by setting the transport mode as nbd "

https://emc--c.na5.visual.force.com/apex/KB_BreakFix_1?id=kA1700000000WPH is a knowledge article on this issue. I have tried it, ran the nsrvadp_save with debug mode 9 but I did not get the errors as indicated in the KB.

https://community.emc.com/message/685962 , https://community.emc.com/message/617398 also suggests some items to be checked but there is no issue, disks are not independent persistent disks, disks are formatted using NTFS, using English alphabets as the drive letters. Datastore has enough space for mount and snapshot.

Backup is not failing during snapshot rather it fails during mounting the volumes tmp.JPG

Please note the error is at the step when volumes are being mounted.

I have checked the 'recommendations and considerations for VADP backup and recovery', it talks of following w.r.t to VixMountApi,

Ensure the path specified in VixDisklib and VixMountAPI config files are enclosed in

double quotes as below:

tempDirectory="C:\Program Files\EMC NetWorker\nsr\plugins\VDDK\tmp"

These files are stored in the following location by default:

<NetWorker install folder>\nsr\plugins\VDDK\

         Note: Double quotes should be specified in the path even though the path is already

present.

I have checked and the above holds true in my case. The VADP proxy is also the proxy for other VMs and the backups for other VMs is running fine, something is wrong with this particular VM (remember this is a new configuration).

What are the things that could be checked?

Regards

tech88kur

0 Kudos
tech88kur
2 Iron

Re: VADP Backup Issues

Jump to solution

Dear All,

I gave a try to backup after rebooting the machine, hoping something would change but nothing has. Backup still fails with the same errors.

Did anybody face something similar?

Regards

tech88kur

0 Kudos
tech88kur
2 Iron

Re: VADP Backup Issues

Jump to solution

Hi,

Updating the status of this issue..

It turned out that the disks types used were a mix of SCSI and IDE (some of them SCSI and the others IDE), this is a known limitation with VADP. Hence the backup failures.

Regards

tech88kur

0 Kudos