nsrck SYSTEM warning Cannot lock file index for client '<Client Name>' (Too many open files)

Summary: NetWorker Servers running version 19.4.0.2 and 19.5.0.0 may encounter an issue where both automated and manually run NSRCK commands will fail with "too many open files" errors.

This article applies to This article does not apply to This article is not tied to any specific product. Not all product versions are identified in this article.
Check out other resources

Symptoms

NetWorker Servers running version 19.4.0.2 and 19.5.0.0 may encounter an issue where both automated and manually run NSRCK commands fail with "too many open files" errors.
  • NSRCK execution as part of the Server Protection Policy will fail without presenting an error or failure to the UI, errors are logged in the Policy log and Daemon.raw files:
    • Policy Log
      • 96467 1624042989 2 1 24 15160 10448 0 Server.Domain.com nsrck SYSTEM warning 43 Cannot lock file index for client '%s' (%s) 2 12 20 Client.Domain.com 24 19 Too many open files
    • Daemon.raw
      • 71193 1622838475 0 0 0 5876 5980 0 Server.Domain.com nsrd NSR info 9 %s %s: %s 3 11 10 2935:Index 11 12 19362:Notice 49 70 195146 32 Checking index for client '%s'.  1 0 20 Client.Domain.com
      • 96467 1622838492 2 1 24 6444 7952 0 Server.Domain.com nsrck SYSTEM warning 43 Cannot lock file index for client '%s' (%s) 2 12 20 Client.Domain.com 24 19 Too many open files
  • Running NSRCK via the command line will return the following error:
    • nsrck SYSTEM warning Cannot lock file index for client '<Client Name>' (Too many open files)
  • NetWorker Client File Indexes (CFI) will continue to grow and consume storage which could cause the system to become unstable if all storage is consumed on the host drive
  • Backup and Restore jobs may intermittently fail if running at the same time as NSRCK

Cause

This issue is caused by a code defect introduced in the 19.4.0.2 Cumulative Hotfix (CHF) & 19.5.0.0 Initial Release.

Resolution

Product (Operating System or Operating Environment)
 		 NetWorker Server 19.4.0.3
NetWorker Server 19.5.0.1
Action Type
 		 Apply Hotfix
Action Needed
 		 Apply appropriate Hotfix based on impacted version currently installed
Who Can Complete the Action
 		 NetWorker Administrator
Issue Addressed in this OS, OE or Software
 		  
Resolution Detail Resolution:
NetWorker servers running 19.4.0.2 must apply the 19.4.0.3 Cumulative Hotfix (CHF) to resolve this problem.

NetWorker servers running 19.5.0.0 must apply the 19.5.0.1 Cumulative Hotfix to resolve this problem.

Interim Workaround:
To work around this issue, increase the number of available File Descriptors at the Operating System level.  For Linux systems:
 
  1. To ensure a persistent setting edit the /etc/sysctl.conf file:

# vi /etc/sysctl.conf
 
  1. Append a configuration directive as follows:

fs.file-max = <number>

where <number> is a decimal number that defines the max number of open files.  32768 is the recommended setting for large NetWorker deployments and is a good starting point.
 
  1. Save and close the file. You can either reboot the machine or enter the following command to force all processes to get the new value:

# sysctl -p

(preference is for a reboot).

Note: To work around the issue administrators may need to define values higher than what is recommended above, every system will be different.  Please consult with the Operating System documentation for safe operating limits when increasing these values.
Solution Considerations It is highly recommended that the Hotifx be applied as soon as possible.

It is critical that the Operating System File Descriptor values be restored to values within the range defined in the Performance and Optimization Guide after the Hotfix has been applied.

Products

NetWorker
Article Properties
Article Number: 000188258
Article Type: Solution
Last Modified: 23 Aug 2022
Version:  6
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.