NetWorker: How to Troubleshoot Service Response Issues in NetWorker using the nsrwatch utility

The NetWorker server installation package provides a command-line utility known as nsrwatch. The nsrwatch command is used to monitor and operate NetWorker servers. It can also be used to troubleshoot NetWorker processes that are unresponsive or consuming system resources unexpectedly. Using the nsrwatch utility is not the first step in troubleshooting service response issues in NetWorker. It should only be used once the more likely causes of a service or performance-related issues have been ruled out:
 

Check First That:

• All elements of the NetWorker datazone are supported.
  • See the compatibility guides on elab navigator: https://elabnavigator.dell.com/eln/modernHomeAutomatedTiles?page=NetWorker
  • See the NetWorker version-specific Performance and Optimization Guide, available on the Dell support site: https://www.dell.com/support/product-details/product/networker/overview 
  • In environments where remote NetWorker storage nodes are deployed. The NetWorker version of the storage node must match the NetWorker version of the NetWorker server.
• There is no disk space issue on the NetWorker server.
  • Linux:

df -Th

• • Windows: Check disk space usage from Windows File Explorer or Disk Management.
• The resources available on the NetWorker server (CPU, RAM, and so on) are not behind exhausted during peak times.
  • Linux:

top

• • Windows: Task Manager or Perfmon
  • NetWorker: Troubleshooting High Memory or CPU usage by Process 
• There are no significant errors in the operating system logs
  • Linux: /var/log/messages
  • Windows: Event Viewer (Application and System Event logs)
• The issue is with all standard NetWorker operations and is not more specific, for example:  Backups to one Storage Node, only Cloning operations are affected, and so forth.
• Core dump generation has been enabled at the OS level on the NetWorker server, and there are no processes core dumping.
  • NetWorker Troubleshooting Guide: Process Crashes and Core Dumps
• NetWorker Performance and Optimization Planning:

• • There are enough target volumes available for the NetWorker backups.
  • There are enough open file descriptors defined on a linux NetWorker server. See: NetWorker kernel parameter requirements
  • Performance optimization settings and parameters have been applied.
  • The NetWorker server meets the minimum requirements for a datazone of this size. See: System Components.
  • For a large datazone, the NMC server is on a separate machine from the NetWorker server. See: Memory requirements for the NetWorker server and NetWorker Management Console

• The port ranges have been set correctly as per the Security Configuration Guide, https://www.dell.com/support/product-details/product/networker/docs

nsrwatch Command-Line Options:

Windows OS: How to Use nsrwatch to troubleshoot an apparent NetWorker service or process issue:

1. Install Debugging Tools for Windows on the NetWorker server
2. Ensure that the cdb.exe is in the Windows PATH variable . A new command prompt window must be opened in order for this change to take effect. No reboot is required.
3. The symbol files (pdb files) for your NetWorker version must be available. Dell NetWorker Support can supply these.
4. Run nsrwatch and leave running waiting for the service or process condition.

For Example: E:\Symbols being the path where the pdb files are deployed.

C:\>nsrwatch -p nsrd -i 10 -t 10 -k 10 -S E:\Symbols > E:\Logs\nsrwatch.nsrd  2>&1
C:\>nsrwatch -p nsrexecd -i 10 -t 10 -k 10 -S E:\Symbols > E:\Logs\nsrwatch.nsrexecd 2>&1
C:\>nsrwatch -p nsrmmdbd -i 10 -t 10 -k 10 -S E:\Symbols > E:\Logs\nsrwatch.nsrmmdbd  2>&1
C:\>nsrwatch -p nsrjobd -i 10 -t 10 -k 10 -S E:\Symbols > E:\Logs\nsrwatch.nsrjobd  2>&1

Logs to be collected:

• Rendered daemon.raw
  • NetWorker: How to use nsr_render_log to render .raw log files
• daemon.raw
• nsrwatch output for the relevant daemons (as outlined above).
• System and Application event logs

Linux OS: How to Use nsrwatch to troubleshoot an apparent NetWorker service or process issue:

1. Install non-stripped binaries in place of the standard binaries for the process which is of interest: Usually nsrd, nsrjobd, nsrmmdbd, and nsrexecd. These can be provided by Dell NetWorker support.
2. This requires that NetWorker first be shut down. They must have the correct permissions.
3. Run nsrwatch and leave running waiting for the service or process issue.

nsrwatch -p nsrd -i 30 -t 30 -k 30 > nsrd_out
nsrwatch -p nsrexecd -i 30 -t 30 -k 30 > nsrexecd_out
nsrwatch -p nsrjobd -i 30 -t 30 -k 30 > nsrjobd_out
nsrwatch -p nsrmmdbd -i 30 -t 30 -k 30 > nsrmmdbd_out

Logs to be collected:

• Rendered daemon.raw
  • NetWorker: How to use nsr_render_log to render .raw log files
• daemon.raw
• nsrwatch output for the relevant daemons.
• OS messages file (/var/log/messages)
  • Ensure to check the /var/log/messages file and ensure that the log covers the full nsrwatch window. If there is a lot of activity on the server, the messages log can roll over. Other /var/log/messages-date or /var/log/messages# exist if the messages log file has rolled over. Carefully review the timestamp from when these files were created. If they also cover the nsrwatch session, grab these additional messages logs as well.

• NetWorker: How To Determine Server Load Using nsrwatch
• NetWorker Processes and Ports
• NetWorker: How to Debug Backup Operations
• NetWorker: Log Files and Locations
• NetWorker Troubleshooting Guide: Process Crashes and Core Dumps
• NetWorker: Troubleshooting High Memory or CPU usage by Process