Powerscale nfs.log shows messages: “ [nfs] rendezvous_request(450) Connection limit hit count: 404, max 404”

When running showmount -E against NFS server, there is error, e.g:

isiloncluster-1# showmount -e localhost
RPC: Unable to receive
showmount: can't do exports rpc
isiloncluster-1#

Check the node /var/log/nfs.log file, there are errors:

...
2024-09-04T15:11:20.982330-06:00 <30.4> isioncluster-1(id4) nfs[5201]: [nfs] rendezvous_request(453) Connection limit hit count: 404, max: 404
2024-09-04T15:11:21.030873-06:00 <30.4> isioncluster-1(id4) nfs[5201]: [nfs] rendezvous_request(453) Connection limit hit count: 404, max: 404
2024-09-04T15:11:22.004448-06:00 <30.4> isioncluster-1(id4) nfs[5201]: [nfs] rendezvous_request(453) Connection limit hit count: 404, max: 404
...

The clients exhibited a problem with connectivity or network issue. Even right after restarting the NFS service no change occurred.
These messages started to log right after the upgrade was completed.

During nfs server initializing it will log the Physical Memory and VMEM Limit in the /var/log/nfs.log, e,g: 

...
2024-09-04T15:00:33.123909-06:00 <30.5> isioncluster-1(id4) nfs[4410]: [nfs] NfsInitialize(612) Initializing...
2024-09-04T15:00:33.142086-06:00 <30.5> isioncluster-1(id4) nfs[4410]: [nfs] Physical Memory: 16967553024
2024-09-04T15:00:33.142617-06:00 <30.5> isioncluster-1(id4) nfs[4410]: [nfs] VMEM Limit: 4241887232
2024-09-04T15:00:33.800550-06:00 <30.4> isioncluster-1(id4) nfs[4410]: [nfs] NfsTuneNodeConfig(2783) RPC Max Connections: 404
2024-09-04T15:00:33.800612-06:00 <30.4> isioncluster-1(id4) nfs[4410]: [nfs] NfsTuneNodeConfig(2811) SchedulerLimitNfsKernelSizeMB: 63, SchedulerLimitNfsUserSizeMB 16
2024-09-04T15:00:33.966847-06:00 <30.5> isioncluster-1(id4) nfs[4410]: [nfs] NfsInitialize(798) Limits bytesin: 4294967296, bytesout: 4294967296, workitems: 0,kernelbytes: 848377651, userbytes: 848377446, connections: 0, openfiles: 0
2024-09-04T15:00:34.953730-06:00 <30.4> isioncluster-1(id4) nfs[4410]: [nfs] Config backend for nfs_scheduler_config is now synchronized
2024-09-04T15:00:35.328750-06:00 <30.4> isioncluster-1 syslogd: last message repeated 1 times
2024-09-04T15:00:43.140745-06:00 <30.5> isioncluster-1(id4) nfs[4410]: [nfs] NfsInitialize(1493) Initialization complete: status: 0x0 (STATUS_SUCCESS)
2024-09-04T15:00:47.633530-06:00 <30.4> isioncluster-1(id4) nfs[4410]: [nfs] Config backend for nfs_config_zone_1 is now synchronized
2024-09-04T15:00:47.853666-06:00 <30.4> isioncluster-1(id4) nfs[4410]: [nfs] Config backend for nfs_config is now synchronized
2024-09-04T15:00:47.905511-06:00 <30.5> isioncluster-1(id4) nfs[4410]: [nfs] Server now ready to process requests
<snip>

So Node Physical Memory: 16967553024, VMEM Limit: 4241887232.
The RPC Max Connections is 404, and this is relevant to the default registry.Services.lwio.Parameters.Drivers.nfs.RpcPercentVMemMBConnections is 10. 

Remedy/Workaround:

# isi_gconfig registry.Services.lwio.Parameters.Drivers.nfs.RpcPercentVMemMBConnections=0

After changing the value OneFS_NFS needs to be restarted:

# isi_for_array -sX '/usr/likewise/bin/lwsm restart onefs_nfs; sleep 30'

Or disable/enable NFS:

# isi services nfs disable
# isi services nfs enable

Please note existing nfs connection will be interrupted for a short time.

Resolution: 

Increase the A200 node physical memory to 64G.