Isilon: NetWorker: NDMP backup error: "Too many open files"

Summary: isilon 8.2 NetWorker NDMP backup error: "Too many open files", which can be fixed by increasing open file limit of isilon isi_ndmp_d

This article applies to This article does not apply to This article is not tied to any specific product. Not all product versions are identified in this article.
Check out other resources

Symptoms

  • Networker 19.4 
  • Isilon 8.2
  • NDMP backup has error: "Too many open files".

Cause

Isilon 8.2 has  open file limit in Isilon  isi_ndmp_d.

Resolution

"Too many open files" is  a known issue. It's recommended increase the open file limit of isi_ndmp_d.
  1. Edit /etc/mcp/sys/services/ndmpd file and increase max_open_files
<action>/bin/sleep 30</action>
<action>/usr/bin/limits -B -P $(pgrep isi_ndmp_d) -n 32768</action>
  1. Edit the ndmpd on any node :
cd /etc/mcp/sys/services
cp ndmpd /ifs/data/Isilon_Support
vi /ifs/data/Isilon_Support/ndmpd
  1. In the <actionlist name="start"> section , add the following lines:
<action>/bin/sleep 30</action>
<action>/usr/bin/limits -B -P $(pgrep isi_ndmp_d) -n 32768</action>

Example: 
<?xml version="1.0"?>
<service name="ndmpd" enable="0" display="1" options="require-quorum,kill-on-sigquorum,require-post-ifs">
      <isi-meta-tag id="ndmp_service">
        <mod-attribs>enable</mod-attribs>
      </isi-meta-tag>
      <description>Network Data Management Protocol Daemon</description>
      <process name="isi_ndmp_d" pidfile="/var/run/isi_ndmp_d.pid"
               startaction="start" stopaction="stop"/>
      <actionlist name="start">
        <action>/usr/bin/isi_ndmp_d</action>
        <action>/bin/sleep 30</action>
        <action>/usr/bin/limits -B -P $(pgrep isi_ndmp_d) -n 32768</action>
      </actionlist>
      <actionlist name="stop">
        <action>/usr/bin/killall isi_ndmp_d</action>
      </actionlist>
</service>
  1. Save the file (may have to use ! as the file is read only)
  2. Then distribute the file to all of the nodes:
isi_for_array 'cp /ifs/data/Isilon_Support/ndmpd /etc/mcp/sys/services'
  1. Restart NDMP service:
isi services ndmpd disable
isi services ndmpd enable
  1. Check openfiles limit for isi_ndmp_d PIDs:
isi_for_array 'procstat -l $(pgrep isi_ndmp_d)|grep openfiles' 
dklimki-mr1512e-4: 26547 isi_ndmp_d       openfiles              16384            16384            
dklimki-mr1512e-5: 77179 isi_ndmp_d       openfiles              16384            16384            
dklimki-mr1512e-3: 74909 isi_ndmp_d       openfiles              16384            16384            
dklimki-mr1512e-1: 27389 isi_ndmp_d       openfiles              16384            16384            
dklimki-mr1512e-2:  3725 isi_ndmp_d       openfiles              16384            16384  
  1. Check again after 30 sec:
isi_for_array 'procstat -l $(pgrep isi_ndmp_d)|grep openfiles' 
dklimki-mr1512e-3: 74909 isi_ndmp_d       openfiles              32768            32768
dklimki-mr1512e-1: 27389 isi_ndmp_d       openfiles              32768            32768
dklimki-mr1512e-2:  3725 isi_ndmp_d       openfiles              32768            32768
dklimki-mr1512e-4: 26547 isi_ndmp_d       openfiles              32768            32768
dklimki-mr1512e-5: 77179 isi_ndmp_d       openfiles              32768            32768


Article Properties
Article Number: 000191876
Article Type: Solution
Last Modified: 31 Aug 2022
Version:  2
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.