NetWorker: Troubleshooting 'Inactivity Timeout' Errors
Summary: Backups fail with Inactivity Timeout errors.
Symptoms
Symptoms
- Error:
Inactivity timeout - Backups failing due to inactivity timeout error
- Backups aborted due to inactivity timeout error
- Error:
(client):(save set) has been inactive for (number) minutes - Error:
SYSTEM STATE:\ aborted due to inactivity - Error:
nsrexec: clientname has been abandoned by nsrexec - Error:
nsrexecd: Recvd signal to kill process group - aborted due to inactivity - Error:
nsrexec: Attempting a kill on remote save - Backups fail with Inactivity Timeout errors
- Error:
inactivity timeoutduring incremental backup - Error:
No full backups of this save set were found in the media database; performing a full backup - Incremental backups fail
- Saves stop responding
Cause
NetWorker provides the 'Inactivity Timeout' attributes in several locations:
- NetWorker server job inactivity timeout:
- Action inactivity timeout:
The NetWorker Administration Guide provides additional information regarding "Inactivity Timeout" settings: NetWorker Documentation
Resolution
In general, an inactivity timeout may be triggered by many reasons.
Inactivity timeout errors often occur when the client scans a large file system. If few files changed, the next save stream may be delayed significantly. This condition can be resolved by increasing the value of the 'Inactivity Timeout' attribute for the NetWorker backup action that is reporting the error. Set the value to 0 to disable the timeout. Then observe the save set duration and set the inactivity timeout to that number of minutes or more.
If the timeout is not due to the reason listed above, then confirm the following items:
- Check NetWorker client has not been turned off and the network cable is attached.
- Retries of backups always fail.
- All name resolutions between the NetWorker client and backup server are successful: NetWorker: Name Resolution Troubleshooting Best Practices
- Verify all known aliases for the NetWorker client are entered in the alias attribute of the NetWorker client resource.
- All network cards and switches have the same Duplex, Speed, MTU, and other settings.
- Parallelism settings are properly defined. See the NetWorker Performance Optimization and Planning Guide: NetWorker Documentation
- Verify the adjustable TCP/IP parameter. See the NetWorker Performance Optimization and Planning Guide: NetWorker Documentation
Firewalls:
If a firewall separates the NetWorker server and client, open enough ports within the NetWorker connection range to allow proper communication. The NetWorker server may reach the client, but the client might fail to open new ports to respond. NetWorker Processes and Ports
NOTE: Consult with your network or security team to monitor firewall traffic during the action that is failing with timeout errors, and monitor for any communication issues.
NetWorker Configuration:
- This issue was seen due to corruption of files on the disk. Verify the checksum or date of the files to ensure their integrity. Remove, move, or fix the corrupted file or use NetWorker directive to skip it.
- Corruption of the NetWorker resource file, especially the backup, or clone action resource, causes inactivity timeout errors on occasion. Try deleting the failing backup or clone action's resource and re-creating it. You might also try recovering the NetWorker resource file from a time before the errors started. This typically requires NetWorker support engagement to perform NetWorker Server Disaster Recovery (NSRDR)
- Corruption of NetWorker Client File Indexes or Media database is known to result in inactivity timeout errors. Run
nsrck -L6to attempt to repair indexes or runnsrck -L7to recover indexes. For the media database, corruption might be fixed by recovering an older media database. This typically requires NetWorker support engagement to perform NetWorker Server Disaster Recovery (NSRDR).
- Restart NetWorker daemons to clear caches that may be out of date.
nsr_shutdown systemctl start networker OR /etc/init.d/networker start
net stop nsrexecd /y For NetWorker servers: net start nsrd For NetWorker Management Console Servers: net start nsrexecd net start gstd For NetWorker client systems: net start nsrexecd
- Reboot the server and client which clears memory from any fragments of dead code and reinitialize all applications and the OS.