FluidFS Series: Due to SMB high resource usage new sessions are not allowed
Summary: The events may report that due to Server Message Block (SMB) high resource usage, new sessions are not allowed.
Symptoms
Example:
2021 Dec 3 08:04:42.746961 (08:04:42) node1 INFO eventDispatcher[18467]: Event ( SMBMemThreshold4 ) Category ( Client Access ) Text ( Due to SMB high resource usage new sessions not allowed. Description: Due to SMB high resource usage new sessions are not allowed on NAS controller 0. )
The issue is likely to be encountered initially when new client connections to the appliance are denied, resulting in a loss of access.
Existing client connections continue to function properly, while others fail when attempting to reconnect to the problematic node. In this situation, the appliance attempts to load balance connections by failing over to the other node.
Unfortunately, this error is only classified as informational, so it is unlikely that administrators see the alert before clients experience connection problems.
Cause
The appliance has a session limit, if this limit is reached new sessions are denied.
Example:
2021 Dec 3 08:03:15.011115 (08:03:15) node0 NOTICE lwio: [22628] ALWAYS:0x7f77f41cc700:SrvElementsAllowNewSessions():lwio/server/srv/elements/libmain.c:631:2919018928: Sessions were limited because of sessions limitation (current = 24000, max = 24000)
Sessions are not to be confused with connections. The appliance has a maximum connection count of 10,000 if it is a 24GB appliance and up to 30,000 if it is a 48GB appliance. More connections occur with the increasing number of appliances added to a cluster.
The difference between connection and session in this context is that a connection represents the physical layer communication channel.
Session sees a state of information exchange. A single connection may have multiple sessions.
Sometimes, there are scenarios where connection counts are low, but session counts reach the maximum. In these instances, there could be a client software, application, or script running against the appliance, creating additional sessions.
For more information about the FluidFS appliance limitations, see the FluidFS Version 6 Support Matrix.
Resolution
There are multiple ways to tackle this issue. The quickest resolution would be rebooting the afflicted node.
This action disconnects all clients from this node and forces a failover. Also, any unused or idle sessions are also disconnected.
It is possible to manage and monitor these connections Dell Storage Manager (DSM) and the Microsoft Management Console (MMC).
This review can help pinpoint suspected clients with high session counts for future reference. For more information about the MMC and FluidFS, see FluidFS Series: How to use the MMC snap-in to manage FluidFS SMB shares
If clients are identified as the root cause of all the session counts, it is possible to manually disconnect only those clients. This approach leaves all other existing connections intact.
Additional Information
FluidFS CLI for managing sessions:
CLI/client-access/activity/active-sessions> list list-idle-sessions list-sessions-with-many-open-files view-SMB-session view-NFS-session view-FTP-session logoff-NFS-session-by-id logoff-SMB-session-by-id logoff-FTP-session-by-id logoff-NFS-sessions logoff-SMB-sessions logoff-FTP-sessions