Isilon Intermittent slow performance seen while accessing files for the first time over SMB
Summary: This article covers a possible cause of intermittent slow performance while accessing files for the first time over SMB due to high latency in LSASS - pac_to_ntoken operation.
Symptoms
To confirm if the cluster is facing the issue documented in this KB, run the following statistics command to check the latency for LSASS operations:
# isi statistics protocol list --protocols=lsass_in,lsass_out --nodes=all --sort=timeavg --limit=10 -d
Ops In Out TimeAvg TimeStdDev Node Proto Class Op
12.3 0.0 0.0 3163314.5 1890297.9 4 lsass_in other lsa:pac_to_ntoken
7.3 0.0 0.0 5656841.0 3072360.8 12 lsass_in other lsa:pac_to_ntoken
7.0 0.0 0.0 9009370.0 2337733.0 1 lsass_in other lsa:pac_to_ntoken
5.2 0.0 0.0 2064725.1 1497097.5 2 lsass_in other lsa:pac_to_ntoken
4.8 0.0 0.0 8029455.0 3292905.8 3 lsass_in other lsa:pac_to_ntoken
4.7 0.0 0.0 1394934.1 592686.2 11 lsass_in other lsa:pac_to_ntoken
4.2 0.0 0.0 21618318.0 15180645.0 6 lsass_in other lsa:pac_to_ntoken
4.0 0.0 0.0 7413947.5 1518084.4 5 lsass_in other lsa:pac_to_ntoken
3.1 0.0 0.0 1251842.8 438924.1 26 lsass_in other lsa:pac_to_ntoken
3.0 0.0 0.0 23923752.0 15054548.0 13 lsass_in other lsa:pac_to_ntoken
We notice that the TimeAvg for pac_to_ntoken LSASS operation is very high. [In the above output, TimeAvg is in microseconds]
Also, on checking the SMB protocol statistics, we notice high latency times for Session Setup:
# isi statistics protocol list --protocols=smb1,smb2 --nodes=all --sort=timeavg --limit=10 -d
Ops In Out TimeAvg TimeStdDev Node Proto Class Op
30.0 76.4k 7.2k 1529625.2 2925519.5 12 smb2 session_state session_setup
18.3 30.7k 4.0k 1958311.6 2178009.5 2 smb2 session_state session_setup
16.4 46.1k 3.9k 2710180.8 2546982.5 4 smb2 session_state session_setup
14.1 21.7k 4.1k 8245491.5 13119604.0 3 smb2 session_state session_setup
8.0 21.0k 2.0k 450049.3 519492.5 27 smb2 session_state session_setup
7.7 27.6k 1.9k 8541166.0 3304804.8 1 smb2 session_state session_setup
7.0 18.9k 1.7k 939299.2 792166.9 11 smb2 session_state session_setup
5.6 10.5k 1.2k 36783816.0 13622883.0 13 smb2 session_state session_setup
5.3 14.8k 1.1k 6898343.0 2006835.1 5 smb2 session_state session_setup
3.3 10.9k 843.8 2356597.2 1650669.1 26 smb2 session_state session_setup
2.2 5.9k 490.3 28361250.0 12338387.0 6 smb2 session_state session_setup
Cause
If this issue occurred, users might have experienced slowness when accessing files.
Resolution
DELL EMC Engineering has already investigated this problem. A permanent fix is available. Contact the DELL EMC Customer Support Center and quote this article number for more information.
If the cluster is running OneFS 8.0.0.4, apply the patch (supplied by DELL EMC Support).
During installation or removal of this patch, the following services will be stopped and restarted automatically:
* Likewise
* NFS
* HDFS
Dell EMC Isilon recommends that you install the patch during an off-hours maintenance window to minimize the disruption of service to clients.