NetWorker: nsrclone fails to clone saveset that previously failed to clone automatically via a script when DD Retention Lock is enabled

Summary: nsrclone fails to clone saveset that previously failed to clone automatically using a script when DD Retention Lock is enabled.

This article applies to This article does not apply to This article is not tied to any specific product. Not all product versions are identified in this article.
Check out other resources

Symptoms

  • Some save sets previously failed to clone using a script or automatically.
  • 'ghost' save set may show without volume name with mminfo (see below).
  • Check that of destination filesystem shows saveset file, but NOT 'notes' file.
  • scanner command of saveset fails (see below).
  • Further attempts to clone the saveset fail with error: Err: 5004-nfs lookup failed (nfs: No such file or directory)
mminfo CLI output shows failed clone saveset with 'E' flag: 
# mminfo -q ssid=928093938 -r ssid,cloneid,ssretent,clretent,ssflags,clflags,copies
ssid         clone id retent  clretent ssflags clflg copies
928093938   1532073714 08/24/2018 08/20/2018 vrF         2
928093938   1532252170 08/24/2018 08/24/2018 vrF   E     <--- This is the incomplete clone.

However, once we add the 'volume' switch, only 1 line is reported: 
# mminfo -q ssid=928093938 -r ssid,cloneid,ssretent,clretent,ssflags,clflags,copies,volume
ssid         clone id retent  clretent ssflags clflg copies volume
928093938   1532073714 08/24/2018 08/20/2018 vrF         2 source-volume-name
                                                         ^ ---2 copies yet only 1 shows in the output.

Scanner CLI output: 
8909:scanner: using 'DD-name_destination-device-path' as the device name
8936:scanner: scanning Data Domain disk destination-volume-name on DD-name_destination-device-path
8939:scanner: Data Domain disk destination-volume-name already exists in the media index
107505:scanner: Opening the file '/nw-server-name/destination-device-path/08/04/notes/705d0dc6-00000004-5982f3fd-599ed855-190c442c-4c9896fd' failed [5004] ([57525] [140043448723232] Wed Aug  1 13:18:50 2018
        ddp_open_file() failed for File: //nw-server-name/destination-device-path/08/04/notes/705d0dc6-00000004-5982f3fd-599ed855-190c442c-4c9896fd, Err: 5004-nfs lookup failed (nfs: No such file or directory)
).
107505:scanner: Opening the file '/nw-server-name/destination-device-path/08/04/notes/705d0dc6-00000004-5982f3fd-599ed855-190c442c-4c9896fd' failed [5004] ([57525] [140043448723232] Wed Aug  1 13:18:50 2018
        ddp_open_file() failed for File: //nw-server-name/destination-device-path/08/04/notes/705d0dc6-00000004-5982f3fd-599ed855-190c442c-4c9896fd, Err: 5004-nfs lookup failed (nfs: No such file or directory)
).
8761:scanner: done with Data Domain disk destination-volume-name
nsrclone CLI output:  
80470:nsrclone: Following volumes are needed for cloning
80471:nsrclone:         source-volume-name (DD)
100303:nsrclone: Cloning savesets from storage node 'nw-server-name'
142165:nsrclone: Number of clone threads per client is set to 256. Max number of savesets per thread is set to 0. Max client threads is set to 0.
104860:nsrclone: Concurrent proxy clone started for service type DD.
Cloning save sets from the volume source-volume-name
Total client count in a volume source-volume-name is 1
77426:nsrclone: Cloning save sets from the volume source-volume-name
104861:nsrclone: Total client count in a volume source-volume-name is 1
104864:nsrclone: Number of clone threads per client set to 1. Max number of savesets per thread is set to 0. Max client threads is set to 30 for volume source-volume-name
174908:nsrclone: Saving the backup data in the pool 'destination-pool'.
175019:nsrclone: Received the media management binding information on the host 'nw-server-name'.
174910:nsrclone: Connected to the nsrmmd process on the host 'nw-server-name'.
32379:nsrclone: Cloning save set index:705d0dc6-00000004-5982f3fd-599ed855-190c442c-4c9896fd (ssid/cloneid 928093938/1532073714)
75360:nsrclone: Cloning of size 1 KB is in progress...
72985:nsrclone: Source volume 'source-volume-name' ready for cloning
75675:nsrclone: Failed to initiate the save nsrmmd to create a clone instance.
Direct File Save failed for /nw-server-name/destination-device-path/active/df304e8e-00000006-375196f2-5b5196f2-23ce16ac-34ae8c2d on volume destination-volume-name
75267:nsrclone: Failed to initiate mmd session for ssid 928093938.
75712:nsrclone: Cloning failed for save set : index:705d0dc6-00000004-5982f3fd-599ed855-190c442c-4c9896fd (ssid 928093938)
77276:nsrclone: Total size cloned so far: 0 KB.

141764:nsrclone: Unable to perform clone of client saveset. Error: proxy_clone_rs failed.

104863:nsrclone: Concurrent proxy clone completed for sevice type DD.
67790:nsrclone: Total size cloned: 0 KB.
77275:nsrclone: DD cloning failed
Proxy clone failed to clone all or few savesetsSome of the clones failed
Some of the clones failed

Cause

DD has Retention Lock enabled. This prevented the incomplete original save set from being removed, and prevents the next clone attempt from over-writing the data in the destination path.

Resolution

Workaround:

From the scanner attempt above, find the path of the saveset on the destination: 
//nw-server-name/destination-device-path/08/04/notes/705d0dc6-00000004-5982f3fd-599ed855-190c442c-4c9896fd

We need the path of the saveset itself, which will be one level above 'notes', for example: 
//nw-server-name/destination-device-path/08/04/

Then add the DD mtree path, which always starts /data/col1/mtree-name. 'mtree list' on the DD CLI shows the Mtrees configured. Typically the Mtree name matches the host NetWorker server name. Then add the long saveset name to the end.
So in this example, the full path becomes: 
/data/col1/nw-server-name/destination-device-path/08/04//705d0dc6-00000004-5982f3fd-599ed855-190c442c-4c9896fd

We must use this path to disable retention lock on the destination Data Domain system:
  • On the destination DD CLI, run: mtree retention-lock revert [path] (path from above) This asks for the sysadmin password.
  • Clone the saveset again - this should now work without error because it can now write to the destination path.
  • mminfo queries for the cloned savesets should now work correctly.
  • Repeat for any other affected savesets.

Affected Products

NetWorker

Products

NetWorker
Article Properties
Article Number: 000052916
Article Type: Solution
Last Modified: 27 Sept 2023
Version:  3
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.