NetWorker vProxy: VM Restore resurrection from tape fails with "Unable to resurrect clone SSID to a Data Domain volume in a staging pool"

Unable to find a usable device for recovery of saveset SSID: Unable to find enabled Data Domain device for volume 'TAPE_DEVICE'. An attempt will be made to perform automatic clone resurrection
Attempting to automatically resurrect clone SSID with 7 days retention to DD volume in pool Data Domain Default Clone

Virtual machine recovery failed: Unable to resurrect clone SSID to a Data Domain volume in a staging pool: nsrclone finished with code 1.

The message Unable to find a usable device for recovery of saveset SSID: Unable to find enabled Data Domain device for volume 'TAPE_DEVICE' is not an error message. This message is expected because the save set specified during the restore resides on a non-ddboost device (AFTD, Tape, CloudBoost, and so forth). The VM save set recovery cannot be performed directly from a tape device and must reside on a Data Domain (DD Boost) device.

The subsequent message An attempt will be made to perform automatic clone resurrection Attempting to automatically resurrect clone SSID with 7 days retention to DD volume in pool Data Domain Default Clone explains that the restore is going to attempt to clone the save set to the Data Domain Default Clone pool.

This is expected functionality when a VM save set resides on a non DD Boost device; for example, AFTD, Tape, CloudBoost. The VM restore workflow automatically attempts to clone the tape save set to a Data Domain clone pool. This process is called "resurrection." Resurrection only occurs to DD Boost clone pools configured through the NetWorker server's local storage node; a standalone storage node cannot be used.

If a Data Domain clone pool is not available, the resurrection process attempts to use the "Data Domain Default Clone" pool. The "Data Domain Default Clone" pool is a default pool on the NetWorker server, which typically does not have any devices configured.
 

A Data Domain clone pool must be configured using the NetWorker server's local storage node for the vProxy to successfully perform resurrection. One of the following options must be performed:

• Configure a device in the default "Data Domain Default Clone" pool.
• Configure a new Data Domain clone pool on the NetWorker server to stage the resurrections.

To create a clone pool device, perform the following:

1. Log in to the NetWorker Management Console (NMC) with the NetWorker Administrator account.
2. Select the Devices tab.
3. In the left column, expand the NetWorker server tree and select Devices.
4. Right-Click the Devices window and select New Device Wizard.
5. By default the "Data Domain" radio button should be selected. Ensure that this option is selected and click Next.
6. Ensure that you have the DD Boost username and password for your Data Domain, click Next.
7. Select the Data Domain device that you would like to configure the clone volume on and enter the DD Boost Credentials, click Next.
8. Select New Folder, name the folder (e.g: Staging), check the box for the folder you created and click Next:

9. Select Backup Clone for the Pool Type pick the "Data Domain Default Clone" pool, or specify a Name for the new pool, click Next:

10. Important: Leave the NetWorker Server selected as the Storage Node and click Next. If another storage node is used, you will not see this pool listed as an available resurrection pool.

11. Optionally you can configure SNMP monitoring options. If other devices were configured on this Data Domain, these options should be automatically populated. Click Next.
12. Click Configure.
13. Go to the Server tab and click server properties.
14. In the server properties, go to the Configuration tab.
15. Ensure that the Disable RPS Cloning option is cleared:

Once the above has been performed, restart the VM resurrection restore. Select the staging pool on the Select Alternate Recovery Sources window. The data will be kept on the staging pool for 7 days after the resurrection is completed.

Workaround:

If the resurrection cloning process continues to fail, perform a manual clone of the save set to the staging pool. 

You can also get the details needed for the nsrclone command from the NetWorker server using mminfo

mminfo -avot -q vmname=vm_name -r savetime,ssid,cloneid,volume,pool

Using the mminfo output, run a nsrclone command for the ssid/cloneid to clone the save set from tape to a DD clone device configured through the NetWorker server:

nsrclone -d nsrserverhost -b staging_pool -y "7 days" -Z -S ssid/cloneid

Monitor the clone job for completion. If the clone completes successfully you can run the VM restore again and select the copy which was cloned to DD, no staging is required. The save set resides on the staging pool for 7 days as per the above command.

The cause of resurrection failure can be investigated; however, the above workaround may mitigate any urgent restore need.

=====================================================================================================

Example of cloning save set to staging pool:

nve:~ # nsrclone -d nsrserverhost -b "Data Domain Default Clone" -y "7 days" -Z -S 3662113059
180169:nsrclone: Step (1 of 16): NSRCLONE_INIT_PHASE_ONE: nsrclone has been started on host 'nve.amer.lan', with command 'nsrclone -d nsrserverhost -b Data Domain Default Clone -y 7 days -Z -S 3662113059 '.
...
181306:nsrrecopy: PID-19944 with job ID 513040 with recover stream ID 733619339: Save session is started.
72985:nsrrecopy: Source volume 'VMDRClonePool.001' ready for cloning
72986:nsrrecopy: Destination volume 'nve.amer.lan_c.dddefault.001' ready for cloning
180221:nsrrecopy: Job ID 513040: Adding new work item '3662113059/1715956924'.
202684:nsrrecopy: Job ID 513040: Completed the cloning for save set ID 3662113059 and clone ID 1715956924.
181301:nsrrecopy: PID-19944 with job ID 513040 with recover stream ID 733619339: Recover session is ending.
181307:nsrrecopy: PID-19944 with job ID 513040 with recover stream ID 733619339: Save session is ending.
181303:nsrrecopy: PID-19944 with job ID 513040 with recover stream ID 733619339: Recover session ended.
181309:nsrrecopy: PID-19944 with job ID 513040 with recover stream ID 733619339: Save session ended.
180244:nsrrecopy: Step (D of D) for PID-19944 with job ID 513040: NSRRECOPY_DATA_MOVEMENT_PHASE_D: Recover Pipe to Save succeeded.
jobid=513040 for nsrrecopy on host nve-sn-bkp.amer.lan using nve.amer.lan as server name in rps_info_t.
jobid=513040 for nsrrecopy on host nve-sn-bkp.amer.lan using nve.amer.lan as server name in rps_info_t.
180194:nsrclone: Step (12 of 16): NSRCLONE_TERMINATE_PHASE_ONE: The nsrrecopy job with job ID 513040 is exiting.
200780:nsrclone: Step (13 of 16):  NSRCLONE_TERMINATE_PHASE_TWO: Canceling the media multiplexor daemon reservation for source volume '4171793004' and requested(not guaranteed) destination volume '4021052472' with job ID 513040.
180196:nsrclone: Step (14 of 16): NSRCLONE_TERMINATE_PHASE_THREE: Clone operation is exiting with job_exit_status=0 (before): jobid=513040, args.exit_status=0.
         RPS CLONE RESULTS
         [ORIGINAL REQUESTED SAVESETS]
3662113059;
         [CLONE SUCCESS SAVESETS]
3662113059/1717445164;
180187:nsrclone: Step (15 of 16): NSRCLONE_TERMINATE_PHASE_FOUR: The nsrclone verification did not detect failures.
06/03/24 16:25:05.236175 Cloned save set list: : 3662113059

During the recovery operation, select the save set from the cloned DD device: