PowerScale: nfs.log shows messages: "rendezvous_request(450) Connection limit hit count: 404, max 404"

Summary: After upgrade from OneFS version v8.2.x to v9.x, NFS clients may experience intermittent issues with connectivity to mount/files or intermittently unable to list files or access their exports. ...

This article applies to This article does not apply to This article is not tied to any specific product. Not all product versions are identified in this article.
Check out other resources

Symptoms

When running "showmount -e" against NFS server, there is error, e.g:

isiloncluster-1# showmount -e localhost
RPC: Unable to receive
showmount: can't do exports rpc
isiloncluster-1#

Check the node /var/log/nfs.log file, there are errors:

...
2024-09-04T15:11:20.982330-06:00 <30.4> isioncluster-1(id4) nfs[5201]: [nfs] rendezvous_request(453) Connection limit hit count: 404, max: 404
2024-09-04T15:11:21.030873-06:00 <30.4> isioncluster-1(id4) nfs[5201]: [nfs] rendezvous_request(453) Connection limit hit count: 404, max: 404
2024-09-04T15:11:22.004448-06:00 <30.4> isioncluster-1(id4) nfs[5201]: [nfs] rendezvous_request(453) Connection limit hit count: 404, max: 404
...

The clients exhibited a problem with connectivity or network issue even right after restarting the NFS service no change occurred.
These messages started to log right after the upgrade was completed.

Cause

During nfs server initializing it logs the Physical Memory and VMEM Limit in the /var/log/nfs.log, e,g: 

...
2024-09-04T15:00:33.123909-06:00 <30.5> isioncluster-1(id4) nfs[4410]: [nfs] NfsInitialize(612) Initializing...
2024-09-04T15:00:33.142086-06:00 <30.5> isioncluster-1(id4) nfs[4410]: [nfs] Physical Memory: 16967553024
2024-09-04T15:00:33.142617-06:00 <30.5> isioncluster-1(id4) nfs[4410]: [nfs] VMEM Limit: 4241887232
2024-09-04T15:00:33.800550-06:00 <30.4> isioncluster-1(id4) nfs[4410]: [nfs] NfsTuneNodeConfig(2783) RPC Max Connections: 404
2024-09-04T15:00:33.800612-06:00 <30.4> isioncluster-1(id4) nfs[4410]: [nfs] NfsTuneNodeConfig(2811) SchedulerLimitNfsKernelSizeMB: 63, SchedulerLimitNfsUserSizeMB 16
2024-09-04T15:00:33.966847-06:00 <30.5> isioncluster-1(id4) nfs[4410]: [nfs] NfsInitialize(798) Limits bytesin: 4294967296, bytesout: 4294967296, workitems: 0,kernelbytes: 848377651, userbytes: 848377446, connections: 0, openfiles: 0
2024-09-04T15:00:34.953730-06:00 <30.4> isioncluster-1(id4) nfs[4410]: [nfs] Config backend for nfs_scheduler_config is now synchronized
2024-09-04T15:00:35.328750-06:00 <30.4> isioncluster-1 syslogd: last message repeated 1 times
2024-09-04T15:00:43.140745-06:00 <30.5> isioncluster-1(id4) nfs[4410]: [nfs] NfsInitialize(1493) Initialization complete: status: 0x0 (STATUS_SUCCESS)
2024-09-04T15:00:47.633530-06:00 <30.4> isioncluster-1(id4) nfs[4410]: [nfs] Config backend for nfs_config_zone_1 is now synchronized
2024-09-04T15:00:47.853666-06:00 <30.4> isioncluster-1(id4) nfs[4410]: [nfs] Config backend for nfs_config is now synchronized
2024-09-04T15:00:47.905511-06:00 <30.5> isioncluster-1(id4) nfs[4410]: [nfs] Server now ready to process requests
...

The Node Physical Memory: 16967553024, VMEM Limit: 4241887232
The RPC Max Connections is 404, and this is relevant to the default registry.Services.lwio.Parameters.Drivers.nfs.RpcPercentVMemMBConnections is 10. 

Resolution

Remedy/Workaround:

# isi_gconfig registry.Services.lwio.Parameters.Drivers.nfs.RpcPercentVMemMBConnections=0

After changing the value OneFS_NFS must be restarted:

# isi_for_array -sX '/usr/likewise/bin/lwsm restart onefs_nfs; sleep 30'

Or disable/enable NFS:

# isi services nfs disable
# isi services nfs enable

 

NOTE: Existing nfs connection will be interrupted for a short time.

 

Resolution: 

Increase the A200 node physical memory to 64G.

Article Properties
Article Number: 000228332
Article Type: Solution
Last Modified: 20 شوال 1447
Version:  2
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.