Ask the Expert: SMB Protocol on an Isilon Cluster

Mark, you could also do the same with wireshark.  It can roll as many logs as you require and you can roll them on size or time.

Rocketman,

Connections are idle or active.

An active connection means that at the time you checked the stat, the client had sent a SMB request that the node had not responded to yet.

Idle connections mean that the TCP session is active but there was no SMB request sent during the time the stat was collected.

If you see "stale" connections, it really means they are idle and there is an active tcp session associated with it that is being kept alive or has not timed out yet due to inactivity.  As soon as TCP times out, the associated smb session will be cleaned up.

You could compare your smb sessions to the raw netstat output (also you have to remember that these counters are going to be a per node basis):

For example when I look node 1 of my cluster, I see that I have two smb sessions that have been idle for a long time:

isi-ess-east-1# isi smb session list 

Session [1]

        Computer                :192.168.1.10

        Username                :ISI-ESS-EAST\sli

        Client type             :DOS LM 2.0

        Number of opens         :5

        Active time             :54932

        Idle time               : 0

        Guest login             :no

        Encryption              :no

Session [2]

        Computer                :192.168.1.11

        Username                :ISI-ESS-EAST\pete

        Client type             :DOS LM 2.0

        Number of opens         : 0

        Active time             :1096

        Idle time               : 0

        Guest login             :no

        Encryption              :no

When I run netstat -na:

isi-ess-east-1# netstat -na |grep .445

tcp4       0      0 10.111.176.100.445     192.168.1.11.64493     ESTABLISHED

tcp4       0      0 10.111.176.100.445     192.168.1.10.10875   ESTABLISHED

tcp4       0      0 *.445                  *.*                        LISTEN

tcp46      0      0 *.445                  *.*                        LISTEN

Both of those TCP connections are in an established state meaning that the clients are keeping the tcp connection alive.

When I run netstat on my client it too shows the connection established:

C:\Users\pete>netstat -na

Active Connections

  Proto  Local Address          Foreign Address        State

In the end though, if you would like to terminate just those sessions, you can do so via:

For 6.5:

isi smb session delete --computer-name=

For 7.X

isi smb sessions delete --computer-name=

dynamox, how much slower are you seeing browsing for Macs, and are you dealing with large and wide directory structure?  Typically that's not an ideal workflow for SMB.  NFS handles it better due to the readdirplus calls it can make, but NFS comes with its own set of challenges on the Mac (like, AppleDouble's creation of dotbar files).

While I have heard of some folks modifying /etc/nsmb.conf on Macs to disable change notification and alternate data streams to try improve performance, past experience hasn't shown much (if any) improvement gleaned by making those changes.  And, making those changes to the client require that all clients get the change.

As the document describes, retrieving metadata faster from the Isilon is the best way to get the Finder to display objects more quickly.  Reducing network latency may not be possible, which is why the document does call out the 5-7x improvement in speed when using SSDs.

At this point, it's far to early for me to say if something like the SMB2 support in OS X 10.9 is going to make much of a difference, although it is something I'm starting to test with for an update to the Mac guide.

Yeah, you should no longer disable change notify as it breaks Windows Vista clients and beyond.  Per KB:

https://support.emc.com/kb/91441

When the Change Notify setting is set to None, the feature is disabled for the share. Clients will not be notified of any changes that are made in the current directory or sub-directories. If this is the case, Windows clients might experience the following: Windows 7 clients cannot manually refresh the directory. This is a limitation of how directories are cached locally, and of not performing a findfirst of the directory during the refresh. However, when a Windows 7 client makes a change in a directory, that change and all other changes made by other clients will be detected without the need to manually refresh. Windows XP clients can manually refresh the directory by pressing F5 from within Windows Explorer, or right-clicking and selecting Refresh. When a Windows XP client makes a change in a directory, the client detects only the changes that it makes. Changes made by other clients are not detected until a manual refresh is done.

Bernie,

from your experience quota size has no impact on how fast Finder displays directory content, it's all about how many objects (directories/files) are there ?

I'll let Pete speak to the Windows experience of changing the change notification settings (I vaguely seem to recall that SMB2 requires change notification).  My experience with the Mac has shown that changing change notification from All to norecurse solves a strange behavior in the Finder where it'll occasionally flip from one subdirectory all the way back to the root of the share (without user intervention).  We have this documented in KB 89045.  But, again, I haven't seen much performance difference with it set to all or norecurse.

Disabling change notification may be a problem in some workflows where users rely on quick updates to Finder views.  Because there's no F5 in the Finder to refresh a Finder window, change notification sends an OS X fsevent up to the Finder to refresh its view.  That can be useful in cases where two graphics editors are watching the same directory, and one needs to know when the other has updated a file, or added a color label.

Quota size, no.  In my experience, several factors come into play for directory listings on a Mac:

• Number of objects in a directory (If you're using AppleDouble, that may double the number of objects in the directory; the Finder hides the dotbar files from view, but it still has to enumerate them, and read them to apply resource fork data.)
• The view being used in the Finder (Column view can be initially as fast any other view, but if Finder needs to start refreshing each column, it'll hold up displaying lower levels in the tree until the upper levels have finished their refresh.).
• Network latency (round-trip time)
• The time it takes the Isilon cluster to pull metadata from disk.

The worst possible situation for SMB on the Mac is in WAN environments.  I recall seeing an environment where the Mac was in Los Angeles and the Isilon cluster was in London.  The client, even on the customer's private network, was getting 70ms round trip response times, which is pretty good RTT for such a long link.  But, with the hundreds or thousands of SMB calls it had to send over that link, the Finder just ended up spinning for minutes.  Even NFS can only help so much here.  The Finder's quite picky about its metadata (resource forks), compared to Windows.  Those resource forks are essentially additional file data that the Mac has to go enumerate and read before displaying it to the user.

In my experience with the Mac, my goal is to get metadata retrieval latencies down to minimums.  WIth the inability of SMB1 to batch together those metadata calls, SSDs make the biggest difference, as you cut out large amounts of seek latency.  But, you're still at the mercy of network latency, as I mentioned above.

Thank you everyone for participating in this Ask the Expert event. Many thanks to Pete for hosting the discussion! He has posted a summary of some of the key topics here: https://community.emc.com/docs/DOC-26337.

Please feel free to continue the conversation (note that it will not be formally moderated by Pete after today) or start a new thread in the Isilon Support Forum to dicuss other topics.

Best,

Stephanie

Hi Peter,

Is there a way to find out which share is connected to what file server(s)

Also, the document that you have mentioned, Active Directory Discovery and Failover for OneFS, 

is not available anymore. I can't find it in support as well. Could you please share it with us if you have it with you.

Thanks

Damal

Peter, what is the maximum TCP timeout value for the smb idle connections to get dropped.

Is there a limit for number of idle sessions to exist

Hi Experts,

Regarding the latency issues with SMB2, The client connected on 100 Mbps and clueter connected on 10 GB net work to switch.

When we perform a simple copy test we see the read from cluster is 6 MB/Second and write to cluster is 12 MB/Second. so, which is 48 Mbps read and 96 Mbps write speed.

So the above performance is as expected? as the customer has very low speed network ...!!

Thanks, Paddy

Hello Peter

My question is this :

How to copy the smb shares from main isilon to the dr isilon ?

isl-1# isi smb shares list

Share Name Path                           

--------------------------------------------

cpacs-arc1 /ifs/data/test

cpacs-arc2 /ifs/data/ test2

cpacs-arc3 /ifs/data/test3

ifs$ /ifs                           

miki /ifs/miki                      

miki1 /ifs/miki1                     

og /ifs/data/og                   

--------------------------------------------

Total: 7

isl-1#

oreng,

There is not a current native means to duplicate those shares.  You will likely need to use a tool like ShareDupe but its calls are made with API calls similar to MMC.  OneFS 7.1.1. will be the initial release of OneFS to allow MMC management so the use of this tool would require the latest OneFS version.