NetWorker: NDMP Troubleshooting Guide

1. NetWorker server name.
2. NetWorker version, and build number.
3. NetWorker server host OS type and version.

See:

• NetWorker: Methods for Identifying NetWorker Software version
• NetWorker: How to Use the NSRGet NetWorker Data Collection Tool

1. NDMP client name
2. NAS filer OS type, and version
3.  Is the configuration Direct-NDMP (backing up to SAN-attached NDMP tape drives directly from the NAS) or NDMP-DSA (backing up over a TCP/IP connection to a NetWorker server or remote storage node device)?
   1. If NDMP-DSA configuration is the backup device on the NetWorker server or on a NetWorker remote storage node?
   2. If NDMP-DSA configuration what is the backup device type (AFTD, DD Boost, Tape)?

Before troubleshooting any issue, check the NetWorker Compatibility Guide for any support conflicts: E-Lab Interoperability Navigator 2.0-HOME
Review the NetWorker version-specific Network Data Management Protocol User Guide: Support for NetWorker | Manuals & Documents 

1. The most useful NDMP backup related errors are found in the backup action log. You can get the backup action log either from the NMC user interface (in the Monitoring window, double-click the failed save set and click "Get Full Log"). Alternately collect the action logs for the workflow folder under /nsr/logs/policy/[policy_name]/[workflow_name]/*.* on the NetWorker server (collect all files and subfolders below the workflow named folder).
2. To identify the relevant error for a particular failure, look for the first errors logged by 'nsrndmp_save' or 'nsrdsa_save' in the log. Most frequently, the first error is the relevant error and others logged later in the output are symptoms of the failure and not necessarily directly related. For example, the error below is typically logged with most failures, it is logged last, and almost always it is unrelated to the real failure and root cause of the problem:

88394:nsrndmp_save: Couldn't open NDMP file history directory C:\Program Files\EMC NetWorker\nsr\tmp\FileIndex3288118553, The system cannot find the file specified. (Win32 error 0x2)

3. Review the backup action log output for any error messages, and compare with any known issues in Knowledge Base.

For recovery-related issues:

1. NDMP recoveries are only logged in the output of the interface used for any particular restore. That includes the NMC Recovery, the NetWorker Windows User (winworkr.exe), and the 'recover' CLI,
2. The NDMP recovery output is similar in format to that of the backup action log. The processes logging the error in the NDMP recovery output are 'nsrndmp_recover' and 'nsrdsa_recover'. As with the backup logs, look for the first errors logged by 'nsrndmp_recover' or 'nsrdsa_recover' and compare with any known issues in the Knowledge Base.

Step 4:  Additional log and configuration information gathering:

1. NDMP backup and recovery failures are often caused by client resource or NDMP device configuration issues. Whenever possible, collect the 'nsrdb' resource database from the NetWorker server for review.

• Linux: /nsr/res/nsrdb
• Windows (Default): C:\Program Files\EMC NetWorker\nsr\res\nsrdb
• NetWorker: How to Use the NSRGet NetWorker Data Collection Tool

2. NDMP backups can also be run from a terminal or command window on the NetWorker server by running the 'nsrndmp_save' command manually (with or without debug). The output of the command with -D5 debug added can be directed to a file for review using the example command syntax below:

nsrndmp_save -D5 -T dump -c [ndmp_client] -g [group_name] -l [level] [/save_set_name] > ndmpSaveOut.txt 2>&1

3. NDMP recoveries can also be done manually from a command window on the NetWorker Server by running the 'nsrndmp_recover' command (with or without debug) as in the example that follows:

nsrndmp_recover -D5 -s [nw_server] -c [ndmp_client] -S [ssid] -v off -m [ndmp_client]::[/restore_target_path] [/backup_source_path] > ndmpRecoverOut.txt 2>&1

4. Use the 'ndmpsup' utility to collect NDMP-specific version and support information from the NAS filer. The utility also serves as a troubleshooting tool because it requires connectivity to the NDMP host and authenticates using the NDMP username and password configured on the NAS. If 'ndmpsup' runs successfully, this confirms that NetWorker can resolve the NDMP client name and connect to it successfully and that the NDMP user credentials are valid. It also confirms that the NDMP service is enabled and running on the NAS. Run 'ndmpsup' from the NetWorker server and collect the utility output using the syntax below:

ndmpsup -c [ndmp_client] -o ndmpsup_out.txt

5. NAS filer logs following an NDMP backup or restore failure to be collected vary by NAS filer type. The most common NAS filers are Celerra/VNX, Isilon, and NetApp. Collect the NAS logs from these filers as follows:

************************************************************************************************
Recovering 1 file from /data3/data3/restore-VAR/data3/users/george/testNotEmpty/ into /data4/george/restore
Total estimated disk space needed for recover is 2 KB.
Requesting 1 file(s), this may take a while...
52949:recover: Performing recover with directories included
03/30/16 23:01:06.573191 NDMP Service Debug: The process id for NDMP service is 0x475c2d40
42795:nsrndmp_recover: Performing recover from Non-NDMP type of device
03/30/16 23:01:06.838828 NDMP Service Debug: The process id for NDMP service is 0x475c2d40
85183:nsrndmp_recover: DSA is listening for an NDMP data connection on: 10.8.67.221, port = 9026
42689:nsrndmp_recover: Performing DAR Recovery..
86724:nsrdsa_recover: DSA listening at: host 'oscar', IP address '10.8.67.221', port '9026'.
91651:nsrdsa_recover: Successfully established AFTD DFA session for recovering save-set ID '3237787234'.
42938:nsrdsa_recover: Performing Direct File Access Restore
42940:nsrdsa_recover: Reading Data...
42619:nsrndmp_recover: NDMP Service Error: Permission denied to restore on Root fs

42617:nsrndmp_recover: NDMP Service Log: server_archive: emctar vol 1, 1 files, 262144 bytes read, 0 bytes written

42738:nsrndmp_recover: Data server halted: Error during the restore.
42856:nsrndmp_recover: NDMP data server has an internal error.
42871:nsrndmp_recover: Error during File NDMP Extraction.
nsrdsa_recover : Aborted
42840:nsrndmp_recover: NDMP recover failed.
42880:nsrndmp_recover: Error during NDMP recover
16279:recover: NDMP retrieval: child failed with status of 1
************************************************************************************************