NetWorker: Capacity Mode vProxy clone job performance problems between Data Domains

• vProxy cloning speed has dropped from GBs/sec to more conventional and realistic speeds.
• Network bandwidth is ruled out as cause of bottleneck, remaining well below threshold during cloning.
• Messages are found in the ddfs.info logs referring to one or more *-flat.vmdk files for the affected virtual machine (VM) disks containing:
  • synthesized_vbytes 0 and ending with recipe_repl FALSE
  • srepl_filecopy_recipe_validate_bf: srepl_get_replica_attr for base file and ending with error File handle is stale.
• Messages found in the clone action logs (when clone debug level is 3or higher) referring to one or more *-flat.vmdk files for the affected VM's disks containing:
  • Synthetic replication cannot be used for the file…path…/vm-vmnumber-disk-key-disknumber-flat.vmdk

vProxy uses virtual synthetics to leverage VMware's changed block tracking API, providing enormous gains for both backup and cloning operations. This requires active maintenance of internal associations of VM fileset heredity on each Data Domain involved. If problems arise while preparing to clone using virtual synthetics, NetWorker and Data Domain instead fail back to using the default replication workflow. This requires processing of the entire virtual disk files, rather than only the changed blocks, as opposed to only the changed blocks provided by the VMware API. Even if little data is ultimately sent due to deduplication, cloning job duration can increase by multiples.
 
Causes which may lead to failure of virtual synthetic tracking include:

• Different IP addresses that are resolved in NetWorker for the source or destination Data Domain devices between jobs. These must remain consistent for virtual synthetics' internal tracking
• Changing source or destination Data Domain for vProxy backups or clones
• Multiple source or destination volumes for the vProxy backups or clones, which can lead to simultaneous cloning of multiple savesets in the chain
• More than one vProxy saveset requiring cloning for a given VM, which can lead to disordered cloning of savesets in the chain
• VM disks which go for long periods without any changes (particularly, periods which exceed saveset retention periods, such as 35 days without change where retention is 30 days)
• Failure to use ChronologicalOrder Action property

As there are numerous potential causes, review the Data Domain, and NetWorker configurations and ensure:

• ifgroups are set up correctly for both source (backup) and destination (clone target) Data Domains
• NetWorker server, storage nodes, and Data Domains all have hosts file entries using the appropriate ifgroup IPs per Data Domain, and reliable DNS for clients (using NetWorker client hosts files at need).
• Single Data Domain for each backup and clone pool.

Enable the ChronologicalOrder feature for vProxy save set clone actions, which may be hidden in the UI:

• If using nsrclone command, use the -O switch to enforce this new feature when using a save set list with multiple save sets in a given client's chain
• To enable in policy, use either of the following two commands:

nsrpolicy action update clone -p policyname -w workflowname -A actionname [--chronological_order | -l] <yes|no>

• In the nsradmin prompt:

. type nsr protection policy action; policy name: policy; workflow: workflow: name: action

update Chronological order: Yes

• Once this is complete, previously mentioned restrictions on single source and destination volumes, as well as multi-instance incrementals per VM client, are lifted, and can be processed properly.

IMPORTANT: Some conditions may be possible for NetWorker and Data Domain to recover from, where save sets are cloned out of order. This causes some to be unable to use VSR for replication, but once the entire chain is restored, VSR cloning could proceed without issue. Generally, out-of-order cloning, multiple volume and multiple save set cloning can catch up and return to regular operation.
 
In the case of multiple Data Domain IP addresses having already been used or changing Data Domain source or destination entirely, the only way to return to regular and consistent VSR optimization is to force a new Full backup of the affected VMs to reset the changed block tracking and restore VSR optimization. This should be considered when the previous steps have been completed, but Data Domain and/or NetWorker logging, and cloning speeds, indicate that performance issues remain.