Troubleshooting failed NFS mounts

Procedure

1. Open an SSH connection on any node in the cluster and log on using the "root" account.
2. Run the following command to confirm the NFS service is enabled:
   
   isi services nfs
   
   If the service is enabled, output similar to the following will display:
   
   Service 'nfs' is enabled.
    

   NOTE
   If the NFS service is disabled, run the following command to enable it:
   
   isi services nfs enable
   
   If the NFS service was disabled, enabling it might resolve your issue.

3. To determine the ports used by NFS daemons, run the following command:
   
   rpcinfo -p
   
   The output should contain entries for all of the NFS daemons:
   • portmapper
   • nlockmgr
   • mountd
   • nfs
     
      

   IMPORTANT!
   Record the port numbers for each of the daemons. You will use the port numbers in subsequent steps. 

    

   Output similar to the following is displayed:

          program  vers  proto  port service
   100000    4   tcp    111  portmapper
   100000    3   tcp    111  portmapper
   100000    2   tcp    111  portmapper
   100000    4   udp    111  portmapper
   100000    3   udp    111  portmapper
   100000    2   udp    111  portmapper
   100000    4     7    111  portmapper
   100000    3     7    111  portmapper
   100000    2     7    111  portmapper
   134420856     1   udp    985
   134420856     1   tcp    684
   100021    0   udp    609  nlockmgr
   100021    1   udp    609  nlockmgr
   100021    3   udp    609  nlockmgr
   100021    4   udp    609  nlockmgr
   100021    0   tcp    834  nlockmgr
   100021    1   tcp    834  nlockmgr
   100021    3   tcp    834  nlockmgr
   100021    4   tcp    834  nlockmgr
   100005    1   udp    852  mountd
   100005    3   udp    852  mountd
   100005    1   tcp    676  mountd
   100005    3   tcp    676  mountd
   100003    2   udp   2049  nfs
   100003    3   udp   2049  nfs
   100003    2   tcp   2049  nfs
   100003    3   tcp   2049  nfs
   100024    1   udp    614  status
   100024    1   tcp    693  status
    

   If the rpcinfo command does not return any information for NFS, perform the following procedure.

   1. Run the following command:
      
      killall -9 nfsd
      
       
   2. Wait 20 seconds and then run the following command to confirm whether NFS is now bound to port 2049:
      
      rpcinfo -p
      
       
   3. If the rpcinfo command returns information for NFS, try mounting the cluster again.
    

   NOTE
   For the remaining steps, <IP_address> is the IP address of the cluster node which you are testing.
   
   If a step fails, address that issue and then retest the NFS mount before proceeding to the next step. For example, if you cannot ping a node, address the node ping issue and then restest the mount. If the mount still fails, proceed to the next step. If the mount succeeds, then no further steps are necessary.

4. Attempt to ping the node from the client computer, using the hostname of the node, then the IP address of the node:
   
   ping
   
   ping
   
    
5. Test connectivity to the portmapper daemon by making a telnet connection to the portmapper port:
   
   telnet
   
   For example, assuming port 111 in the rpcinfo output above:
   
   telnet 123.456.789.12 111
   
   Output similar to the following will display if connectivity fails:
   
   telnet: Unable to connect to remote host
6. Test connectivity to the mountd daemon by making a telnet connection to the TCP mountd port:
   
   telnet
   
   For example, assuming port 676 in the rpcinfo output above:
   
   telnet 123.456.789.12 676
7. Test connectivity to the nlockmgr daemon by making a telnet connection to the TCP nlockmgr port:
   
   telnet
   
   For example, assuming port 834 in the rpcinfo output above:
   
   telnet 123.456.789.12 834
8. Test connectivity to the nfs daemon by making a telnet connection to the TCP nfs port:
   
   telnet
   
   For example, assuming port 2049 in the rpcinfo output above:
   
   telnet 123.456.789.12 2049
9. On a Linux/Unix client computer, attempt to mount /ifs/ (assuming that there is an export entry for /ifs/ which allows access by the client computer; if not, create one):
   
   mount :/ifs /mnt
   
    
10. To to create an /ifs/ export, open the Isilon web administration interface and do one of the following:
    1. In OneFS 7.0.x and later, click Protocols > UNIX Sharing (NFS) > NFS Export > Add an export. See the OneFS online documentation for the procedure, Add an NFS export.
    2. In OneFS 6.5.x and earlier, click File Sharing > NFS > Add Export. See the OneFS online documentation for the procedure, Add an NFS export.
11. Repeat steps 4-9, above, while performing a packet capture of all traffic between the Linux/Unix client computer and the cluster node. The packet capture can be used by Isilon Technical Support for further troubleshooting, if needed. Below is an example packet capture (tcpdump) command run on the cluster node, using the em0 network interface. Note the "-s0" tcpdump argument, which indicates to not truncate any packet data.
    
    tcpdump -ni em0 -s0 -w tcpdump.OUT host
12. If you complete all of the steps without discovering the cause of the mount failures, contact Isilon Technical Support for assistance and send them the packet capture.