NetWorker: The NetWorker server intermittently becomes unresponsive

NetWorker server is unresponsive intermittently.
The times and durations of unresponsiveness appear to be random and can vary in time.

There are clients and other NSR resources that are trying to be resolved through DNS by nsrd because there is no entry in the hosts file.

These names are not configured in the DNS, and nsrd gets stuck until the lookup call has timed out.

daemon.raw log analysis:
 

• Linux: /nsr/logs/daemon.raw
• Windows (Default): C:\Program Files\EMC NetWorker\nsr\logs\daemon.raw
• NetWorker: How to use nsr_render_log to render .raw log files

nsrd shows some logging at 11:02.

0 11/12/25 11:02:34 nsrd NSR notice 11/12/25 11:02:34.209951 nsrd-D1 Forward DNS lookup failed for client01: Temporary failure in name resolution
0 11/12/25 11:02:34 nsrd NSR notice 11/12/25 11:02:34.209970 nsrd-D9 Adding cache entry for ip_address: "NULL" Host: "client01"

But does not respond back until:

0 11/12/25 11:03:13 nsrworkflow NSR notice 11/12/25 11:03:13.850538 nsrworkflow-D3 prc_policy_progress_changed 0
0 11/12/25 11:03:14 nsrd NSR notice 11/12/25 11:03:14.242852 nsrd-D7 lg_inet_getaddrinfo(): EXIT rc=-3 output cannonname = null output addr = retval error
0 11/12/25 11:03:14 nsrd NSR notice 11/12/25 11:03:14.242963 nsrd-D1 Forward DNS lookup failed for client02: Temporary failure in name resolution
0 11/12/25 11:03:14 nsrd NSR notice 11/12/25 11:03:14.242999 nsrd-D9 Adding cache entry for ip_address: "NULL" Host: "client02"

In both occasions, the nsrd came back after few minutes to log this message: Forward DNS lookup failed for short name client

If not-stripped binaries are placed on the server, nsrwatch stack traces can be collected:
Snippet from one of the many repeated stacks traces when nsrd stopped responding:

#7 0x00007f47fc52f09d in getaddrinfo () from /lib64/libc.so.6
#8 0x0000564ef498547e in lg_inet_getaddrinfo (host=0x7fffafbf2310 "client02", service=0x0, hints=0x7fffafbf2ce0, res=0x7fffafbf2cd8) at /disks/nwbase/nas/nw/19.13/liblocal/lg_inet.c:450
#9 0x0000564ef49860d3 in getaddrinfo_and_log (hostname=0x7fffafbf2310 "client02", service=0x0, hints=0x7fffafbf2ce0, res=0x7fffafbf2cd8) at /disks/nwbase/nas/nw/19.13/liblocal/lg_inet.c:676
#10 0x0000564ef4987a14 in lg_getaddrinfo (hostname=0x7fffafbf3386 "client02", service=0x0, hints=0x7fffafbf2ce0, res=0x7fffafbf2cd8) at /disks/nwbase/nas/nw/19.13/liblocal/lg_inet.c:1262
#11 0x0000564ef497e1c2 in dns_lookup_host (host=0x7fffafbf3386 "client02", addrs=0x7fffafbf3178, names=0x7fffafbf3170) at /disks/nwbase/nas/nw/19.13/liblocal/is_mynam.c:1519
#12 0x0000564ef497d025 in dns_host_find_or_add_byname (host=0x7fffafbf3386 "client02", now=1762946321, mutex=0x564ef6d6c0b0) at /disks/nwbase/nas/nw/19.13/liblocal/is_mynam.c:694
#13 0x0000564ef497cb88 in dns_get_primaryhost (host=0x7fffafbf3386 "client02", phost=0x7fffafbf31f0 "", len=256) at /disks/nwbase/nas/nw/19.13/liblocal/is_mynam.c:508
#14 0x0000564ef497cd97 in is_validhostname (host=0x7fffafbf3386 "client02") at /disks/nwbase/nas/nw/19.13/liblocal/is_mynam.c:588

The process attempts to lg_inet_getaddrinfo on a short name --> hangs --> then nsrd comes back with "failed to do forward DNS lookup"

When the host whose DNS lookup fails is a Storage Node, this issue can appear more frequently.

Either update NetWorker server's hosts file or amend the DNS configuration.

• Linux: /etc/hosts
• Windows: C:\Windows\System32\drivers\etc\hosts

hangs are expected to be gone once the hosts file includes definitions for the client's shortname, or DNS correctly set and configuration has been cleared from nonexisting host names.

Areas of NetWorker configuration where client names are checked and may cause nsrd to stop responding:
NSR client REMOTE ACCESS. This field contains a list of user@hosts, for which all specified hosts are resolved through DNS.
NSR client ALIASES: each alias introduced in this aliases field is subject to be looked up on DNS.
NSR USERGROUP: any user@host specified in the user list will be looked up too.

Example:

• The NetWorker server has an NSR client with these aliases: FQDN, shortname.
• hosts does not contain any information for the client (or is missing FQDN or shortname).
• DNS works on FQDN, but not on shortname.

NetWorker tries to resolve the shortname and fails. Either remove shortname from aliases, or add the entry in the hosts file.

Similar issues can be observed:

• NetWorker: nsrd takes a long time to start "checking resource types in the RAP database"