NetWorker NVP vProxy: Confirm that a VM save set resides on the Data Domain file system

Summary: This KB outlines methods for confirming NetWorker VMware Protection (NVP) Virtual Machine (VM) save set data on Data Domain file systems. This procedure can be useful for more generic errors (e.g: 5004-nfs lookup failed (nfs: No such file or directory)) ...

This article applies to This article does not apply to This article is not tied to any specific product. Not all product versions are identified in this article.

Instructions

The process outlined in this KB helps identify where VMware VM backup data resides on a Data Domain file system.

1. The output can be long and exceed the SSH session buffer. Configure the SSH connection agent to log all the session output to an output file. For example, with PuTTY ensure that the following is set before initiating a session:

Enabling putty session logging

NOTE: The location must be somewhere that the user has permissions to write to without elevated credentials.

2. Connect to the Data Domain from the SSH agent and log in as sysadmin. Collect the list of Data Domain mtrees and identify which mtree is associated with the NetWorker server. 

mtree list

Example:

sysadmin@ddve01# mtree list
Name                     Pre-Comp (GiB)   Status
----------------------   --------------   ------
/data/col1/aftd_backup              0.0   RW
/data/col1/aftd_clone               0.0   RW
/data/col1/backup                   0.0   RW
/data/col1/nsr                   6064.1   RW
----------------------   --------------   ------
 D    : Deleted
 Q    : Quota Defined
 RO   : Read Only
 RW   : Read Write
 RD   : Replication Destination
 IRH  : Retention-Lock Indefinite Retention Hold Enabled
 ARL  : Automatic-Retention-Lock Enabled
 RLGE : Retention-Lock Governance Enabled
 RLGD : Retention-Lock Governance Disabled
 RLCE : Retention-Lock Compliance Enabled
 M    : Mobile
 m    : Migratable

The NetWorker server mtree name differs depending on the number of NetWorker servers connected to a Data Domain. You may see multiple different mtrees.

The mtree name can be identified in the device access information in NetWorker, for example:

NetWorker Mtree path shown in DD device properties from NMC

The /data/col1 portion of the mtree is not shown in NetWorker. 

3. From the Data Domain SSH session, run the following command, specifying the NetWorker server mtree:

filesys report generate file-location path /data/col1/NETWORKER_MTREE

Example:

sysadmin@ddve01# filesys report generate file-location path /data/col1/nsr
--------------------------------      ----------------------      -------------     -----------------
File Name                             Location(Unit Name)         Size              Placement Time
--------------------------------      ----------------------      -------------     -------------------
/data/col1/nsr/BackupDevice01/.nsr_serial       Active  316.00 B        Fri Jun  7 15:03:23 2024
/data/col1/nsr/BackupDevice01/.nsr      Active  52.00 B Fri Jun  7 15:03:24 2024
/data/col1/nsr/BackupDevice01/volhdr    Active  360.00 B        Fri Jun  7 15:03:24 2024
/data/col1/nsr/BackupDevice01/01/72/notes/638537cc-00000006-fffea368-66fea368-00025000-592bbe56 Active  896.00 B        Thu Oct  3 10:00:12 2024
...
...
...
/data/col1/nsr/FLR-nsr-vproxy03.amer.lan-8dde9c9e-e60f-4d9e-87e7-4729e92edc67/.lck-9d18000000000000     Active  92.00 B Thu Oct 10 12:42:15 2024
-----------------------------------------------------------------

Optionally, you can be more specific with the mtree path to a specific device folder, for example:

filesys report generate file-location path /data/col1/nsr/VMBackupDevice02

Or save set:

filesys report generate file-location path /data/col1/nsr/VMBackupDevice02/97/29/4bfadf0d-00000006-df07271e-6707271e-00225000-592bbe56
NOTE: If unsure about file paths, perform an unfiltered collection on the entire mtree and use the output file to search for save sets.

4. On the NetWorker server, use mminfo to query for the wanted VM save set and display the log save set ID:

mminfo -avot -q vmname=VM_NAME -r "savetime,ssid,ssid(60),ssflags,sumflags"

Example:

[root@nsr ~]# mminfo -avot -q vmname=win-client01.amer.lan -r "savetime,ssid,ssid(60),ssflags,sumflags"
  date   ssid       ssid                                                        ssflags fl
...
...
10/09/2024 3741787934 4bfadf0d-00000006-df07271e-6707271e-00225000-592bbe56      vrF    cr
NOTE: Other -r switches can be used to provide additional detail. The NetWorker command reference guide provides more information about mminfo. See, https://www.dell.com/support/home/product-support/product/networker/docs. Operating system commands (findstr, grep) can also be piped (|) onto the command to narrow down the results. In this example, both the NetWorker server and Data Domain are aware of this save set and it is recoverable.

VM save sets are stored on the Data Domain file system under the NetWorker server mtree. The files appear under a folder for the device using the "long-ssid." In this example, the save set folder should exist under the NetWorker server mtree containing the long-ssid: "/data/col1/nsr/../../../4bfadf0d-00000006-df07271e-6707271e-00225000-592bbe56." The path includes other values, so search the filesys report output for the long-ssid to find the full path.

5. Using the long-ssid from the mminfo output and comparing it to the DD filesys command output shows where the save set resides on the DD:

/data/col1/nsr/VMBackupDevice02/97/29/notes/4bfadf0d-00000006-df07271e-6707271e-00225000-592bbe56	Active	2.98 KiB	Wed Oct  9 21:15:21 2024
/data/col1/nsr/VMBackupDevice02/97/29/4bfadf0d-00000006-df07271e-6707271e-00225000-592bbe56/vm-4002-config-file-0.cfg	Active	3.79 KiB	Wed Oct  9 21:00:32 2024
/data/col1/nsr/VMBackupDevice02/97/29/4bfadf0d-00000006-df07271e-6707271e-00225000-592bbe56/vm-4002-config-file-1.cfg	Active	264.49 KiB	Wed Oct  9 21:00:33 2024
/data/col1/nsr/VMBackupDevice02/97/29/4bfadf0d-00000006-df07271e-6707271e-00225000-592bbe56/vm-4002-disk-key-2000-used-blocks.json	Active	16.06 KiB	Wed Oct  9 21:00:34 2024
/data/col1/nsr/VMBackupDevice02/97/29/4bfadf0d-00000006-df07271e-6707271e-00225000-592bbe56/vm-4002-disk-key-2000-changed-blocks.json	Active	139.09 KiB	Wed Oct  9 21:00:34 2024
/data/col1/nsr/VMBackupDevice02/97/29/4bfadf0d-00000006-df07271e-6707271e-00225000-592bbe56/vm-4002-disk-key-2000-flat.vmdk	Active	100.00 GiB	Wed Oct  9 21:14:45 2024
/data/col1/nsr/VMBackupDevice02/97/29/4bfadf0d-00000006-df07271e-6707271e-00225000-592bbe56/vm-4002-disk-key-2000.vmdk	Active	345.00 B	Wed Oct  9 21:15:03 2024
/data/col1/nsr/VMBackupDevice02/97/29/4bfadf0d-00000006-df07271e-6707271e-00225000-592bbe56/vm-4002-disk-key-2000-metadata.json	Active	322.00 B	Wed Oct  9 21:15:03 2024
/data/col1/nsr/VMBackupDevice02/97/29/4bfadf0d-00000006-df07271e-6707271e-00225000-592bbe56/BackupVmSessions.json	Active	5.47 KiB	Wed Oct  9 21:15:16 2024
/data/col1/nsr/VMBackupDevice02/97/29/4bfadf0d-00000006-df07271e-6707271e-00225000-592bbe56/BackupDescriptor.json	Active	2.50 KiB	Wed Oct  9 21:15:16 2024

This method confirms the save set’s location on the Data Domain, its VM files, and whether it is in the active or cloud tier.

Additional Information

Using sfs-dump can generate file locations on the DD, but it is more intensive on the file system, so the filesys report method is preferred. filesys report shows if the save set is in the active or cloud tier, which is crucial for VM restores since files must be in the active tier. 

To generate a sfs-dump on the Data Domain run:

filesys sfs-dump mtree /data/col1/MTREE_NAME

The long ssid collected from mminfo can be used to find the location of the save set in the sfs-dump output.

data/col1/nsr/VMBackupDevice02/97/29/notes/4bfadf0d-00000006-df07271e-6707271e-00225000-592bbe56: mtime: 1728522921752942832 fileid: 4836 size: 3048 type: 9 seg_bytes: 5140 seg_count: 4 redun_seg_count: 0 (0%) pre_lc_size: 5140 post_lc_size: 3883 (76%) mode: 02000100640 start_offset: 0 end_offset: 18446744073709551615
/data/col1/nsr/VMBackupDevice02/97/29/4bfadf0d-00000006-df07271e-6707271e-00225000-592bbe56/vm-4002-config-file-0.cfg: mtime: 1728522033340364000 fileid: 4739 size: 3885 type: 9 seg_bytes: 4953 seg_count: 3 redun_seg_count: 1 (33%) pre_lc_size: 1068 post_lc_size: 1068 (100%) mode: 02000100640 start_offset: 0 end_offset: 18446744073709551615
/data/col1/nsr/VMBackupDevice02/97/29/4bfadf0d-00000006-df07271e-6707271e-00225000-592bbe56/vm-4002-config-file-1.cfg: mtime: 1728522033872272000 fileid: 4743 size: 270840 type: 9 seg_bytes: 272552 seg_count: 26 redun_seg_count: 24 (92%) pre_lc_size: 1712 post_lc_size: 1712 (100%) mode: 02000100640 start_offset: 0 end_offset: 18446744073709551615
/data/col1/nsr/VMBackupDevice02/97/29/4bfadf0d-00000006-df07271e-6707271e-00225000-592bbe56/vm-4002-disk-key-2000-used-blocks.json: mtime: 1728522034201447000 fileid: 4746 size: 16449 type: 9 seg_bytes: 17545 seg_count: 4 redun_seg_count: 0 (0%) pre_lc_size: 17545 post_lc_size: 3299 (19%) mode: 02000100640 start_offset: 0 end_offset: 18446744073709551615
/data/col1/nsr/VMBackupDevice02/97/29/4bfadf0d-00000006-df07271e-6707271e-00225000-592bbe56/vm-4002-disk-key-2000-changed-blocks.json: mtime: 1728522034231344000 fileid: 4747 size: 142425 type: 9 seg_bytes: 143969 seg_count: 20 redun_seg_count: 0 (0%) pre_lc_size: 143969 post_lc_size: 15989 (11%) mode: 02000100640 start_offset: 0 end_offset: 18446744073709551615
/data/col1/nsr/VMBackupDevice02/97/29/4bfadf0d-00000006-df07271e-6707271e-00225000-592bbe56/vm-4002-disk-key-2000-flat.vmdk: mtime: 1728522899965171378 fileid: 4830 size: 107374182400 type: 9 seg_bytes: 107667838826 seg_count: 29139 redun_seg_count: 2510 (9%) pre_lc_size: 273295848 post_lc_size: 270278759 (99%) mode: 02006100640 start_offset: 0 end_offset: 18446744073709551615
/data/col1/nsr/VMBackupDevice02/97/29/4bfadf0d-00000006-df07271e-6707271e-00225000-592bbe56/vm-4002-disk-key-2000.vmdk: mtime: 1728522903489285000 fileid: 4831 size: 345 type: 9 seg_bytes: 1413 seg_count: 3 redun_seg_count: 1 (33%) pre_lc_size: 1068 post_lc_size: 1068 (100%) mode: 02000100640 start_offset: 0 end_offset: 18446744073709551615
/data/col1/nsr/VMBackupDevice02/97/29/4bfadf0d-00000006-df07271e-6707271e-00225000-592bbe56/vm-4002-disk-key-2000-metadata.json: mtime: 1728522903619367000 fileid: 4832 size: 322 type: 9 seg_bytes: 1390 seg_count: 3 redun_seg_count: 0 (0%) pre_lc_size: 1390 post_lc_size: 1347 (97%) mode: 02000100640 start_offset: 0 end_offset: 18446744073709551615
/data/col1/nsr/VMBackupDevice02/97/29/4bfadf0d-00000006-df07271e-6707271e-00225000-592bbe56/BackupVmSessions.json: mtime: 1728522917375230000 fileid: 4833 size: 5602 type: 9 seg_bytes: 6670 seg_count: 3 redun_seg_count: 0 (0%) pre_lc_size: 6670 post_lc_size: 3853 (58%) mode: 02000100640 start_offset: 0 end_offset: 18446744073709551615
/data/col1/nsr/VMBackupDevice02/97/29/4bfadf0d-00000006-df07271e-6707271e-00225000-592bbe56/BackupDescriptor.json: mtime: 1728522916952761000 fileid: 4834 size: 2556 type: 9 seg_bytes: 3624 seg_count: 3 redun_seg_count: 0 (0%) pre_lc_size: 3624 post_lc_size: 2629 (73%) mode: 02000100640 start_offset: 0 end_offset: 18446744073709551615

=======

NetWorker: How To Collect ddfs.info log from Data Domain

Affected Products

NetWorker

Products

Data Domain, Data Domain Boost, Data Domain Boost – File System, NetWorker
Article Properties
Article Number: 000158023
Article Type: How To
Last Modified: 18 Mar 2025
Version:  14
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.