NetWorker: Capacity Mode vProxy clone job performance problems between Data Domains

• vProxy cloning speed has dropped from GBs/sec to more conventional and realistic speeds.
• Network bandwidth ruled out as cause of bottleneck, remaining well below threshold during cloning.
• Messages are found in the ddfs.info logs referring to one or more *-flat.vmdk files for the affected VM's disks containing:
  • synthesized_vbytes 0 and ending with recipe_repl  FALSE
  • srepl_filecopy_recipe_validate_bf: srepl_get_replica_attr for base file and ending with error File handle is stale.
• Messages found in the clone action logs (when clone debug level is 3or higher) referring to one or more *-flat.vmdk files for the affected VM's disks containing:
  • Synthetic replication cannot be used for the file ...path.../vm-vmnumber-disk-key-disknumber-flat.vmdk

vProxy uses virtual synthetics to leverage VMware's changed block tracking API, providing enormous gains for both backup and cloning operations. This requires active maintenance of internal associations of VM fileset heredity on each Data Domain involved. If problems arise while preparing to clone using virtual synthetics, NetWorker and Data Domain instead fail back to using the default replication workflow. This requires processing of the entire virtual disk files, rather than only the changed blocks, as opposed to only the changed blocks provided by the VMware API - and even if little data is ultimately sent due to deduplication, cloning job duration can increase by multiples.
 
Causes which may lead to failure of virtual synthetic tracking include:

• Different IP addresses that are resolved by NetWorker for the source or destination Data Domain devices between jobs - these must remain consistent for virtual synthetics' internal tracking
• Changing source or destination Data Domain for vProxy backups or clones
• Multiple source or destination volumes for the vProxy backups or clones, which can lead to simultaneous cloning of multiple savesets in the chain
• More than one vProxy saveset requiring cloning for a given VM, which can lead to disordered cloning of savesets in the chain
• VM disks which go for long periods without any changes (particularly, periods which exceed saveset retention periods, such as 35 days without change where retention is 30 days)
• Failure to use ChronologicalOrder Action property, if using NetWorker 19.6 and above

As there are numerous potential causes, review the Data Domain, and NetWorker configurations and ensure:

• ifgroups are setup correctly for both source (backup) and destination (clone target) Data Domains
• NetWorker server, storage nodes and Data Domains all have hosts file entries using the appropriate ifgroup IPs per Data Domain, and reliable DNS for clients (using NetWorker client hosts files at need).
• Single Data Domain for each backup and clone pool.

When using NetWorker 19.5 or below:

• Ensure that clone jobs are performed serially and in the order of backups; note this cannot be controlled when multiple clones for a given VM are in the same job list (clone Action or nsrclone saveset file list).
• Ensure a single backup and clone volume per pool, to avoid concurrent cloning when multiple volumes are available for source.

As of NetWorker 19.6, you should always enable the ChronologicalOrder feature for vProxy saveset clone actions, which may be hidden in UI:

• If using nsrclone command, use the -O switch to enforce this new feature when using a saveset list with multiple savesets in a given client's chain
• To enable in policy, use either of the following two commands:

  • nsrpolicy action update clone -p policyname -w workflowname -A actionname [--chronological_order | -l] <yes|no>

  • In nsradmin prompt:

    • . type nsr protection policy action; policy name: policy; workflow: workflow: name: action

    • update Chronological order: Yes

      
       
• Once this is complete, previously mentioned restrictions on single source and destination volumes, as well as multi-instance incrementals per VM client, are lifted, and can be processed properly.

IMPORTANT: Some conditions may be possible for NetWorker and Data Domain to recover from, where savesets are cloned out of order. This causes some to be unable to use VSR for replication, but once the entire chain is restored, VSR cloning could proceed without issue. Generally, out-of-order cloning, multi-volume and multi-saveset cloning can catch up and return to regular operation.
 
However, in the case of multiple Data Domain IP addresses having already been used or changing Data Domain source or destination entirely, the only way to return to regular and consistent VSR optimisation is to force a new Full backup of the affected VMs to reset the changed block tracking and restore VSR optimisation. This should be considered when the previous steps have been completed, but Data Domain and / or NetWorker logging, as well as cloning speeds, indicate that performance issues remain.