Improving OSX SMB I/O speed.

This is unfortunate, but what seems to be the norm for Apple when it comes to "enterprise" support. They've never put a lot of effort into the network protocols, even NFS.

I know we are doing some internal testing at Isilon so I would hope can find out what's going on here and try to resolve it. What I would be interested in is seeing a Mac to Mac SMB2 test. Apple is deprecating AFP so if there is speed to be found with any tweaking you'd think they'd at least have those options set between two Macs.

I can do a Mac>Mac transfer test tomorrow to see if that somehow performs better.

NFS is decent in terms of speed, but does not play well at all with Finder, which leaves it as a fairly unusable option as well.

I just ran into this Arstechnica article about SMB performance, but it's over wireless so not that relevant.

Mavericks fully fixes 802.11ac transfer speeds in OS X | Ars Technica

Also looks like there are issues with the SMB2 implementation... no surprise...

http://cammodude.blogspot.com/2013/10/os-x-109-mavericks-workaround-for-smb.html

we tried this in our environment and it had absolutely no impact in our environment. Not just for connectivity to Isilon but to VNX as well.

Many have found it extremely useful to disable TCP delayed acks on the Mac side.  This is a tried and true method to significantly increase SMB throughput for Mac clients.

sudo sysctl -w net.inet.tcp.delayed_ack=0

is your friend :-)  If you want to save this across reboots, you (Mac admins) know what to do.

Indeed, and that's why I said many environments, not all.  If your workload isn't throughput-constrained, and/or your network is already heavily congested, you may not see any improvement, since the negative interaction between Nagle and delayed acks (200 ms delay) isn't a bottleneck. 

But I can cite cases where the throughput went from 30 MB/sec to well over 300 MB/sec with just this change for a given stream from a given Mac (10 GE network)

Keep in mind that OS X's SMB client stack is FreeBSD's smbfs.

http://www.apple.com/opensource/

I have done some basic traces during testing on OS X 10.9 (13A603) Mavericks and I can confirm that it is using SMB2.

Here's what I am seeing over a single 1GbE connection from a Mac Pro running 10.9 to a 1GbE interface (standard MTU 1500) on a 5 node X200 cluster running OneFS 7.0.2.4:

Here's what I am seeing over a ATTO FastFrame 10GbE NIC in the same Mac Pro to a 10GbE interface with jumbo frames enabled on the same 5 node X200 cluster:

Both tests were run without any special tuning performed to Mac OS 10.9.

What are the Win 7 and 10.9 clients that are being used in your test?  Can you provide some detail on the hardware configuration and network topology for the test?  Do you have an open Support Request for this issue?

Robert,

that looks almost NFS-ish to me...

Can you post the relevant lines (for that client) from

isi statistics client --long

so we can see how these impressive numbers are achieved

(i.e. in terms of ops, transfer sizes and latencies)? 

Thanks

-- Peter

According to the EMC/Isilon product compatibility resources, OS X 10.9 Mavericks isn't officially supported at this time.

Can someone show me information proving otherwise? I can't deploy 10.9 into production until Isilon/EMC signs off.

I'm surprised that EMC employees like Peglarr are discussing a non-supported OS without a disclaimer.

Peter_Sero wrote: Robert, that looks almost NFS-ish to me... Can you post the relevant lines (for that client) from isi statistics client --long so we can see how these impressive numbers are achieved (i.e. in terms of ops, transfer sizes and latencies)? Thanks -- Peter

Peter, here is the output for the 1GbE connection:

X200-3# isi statistics client --long --remote_addrs=10.25.9.58

       TimeStamp     NumOps        Ops      InMax      InMin         In      InAvg     OutMax     OutMin        Out     OutAvg    TimeMax    TimeMin    TimeAvg       Node      Proto            Class     UserId   UserName        LocalAddr        LocalName       RemoteAddr       RemoteName

               s                   N/s          B          B        B/s          B          B          B        B/s          B         us         us         us                                                                                                                               

    1383242030.2       1.2K      239.2       524K       287K       120M       503K         84         84        20K       84.0      10466       1763     4862.3          1       smb2            write          *    UNKNOWN       10.25.9.88       10.25.9.88       10.25.9.58       10.25.9.58

Here is the output for the 10GbE connection:

Disclaimer:  Mac OS 10.9.0 is a new release from Apple and not fully tested for enterprise deployments.  Do not try this at home!

Great, so the data gets written in SMB(2) packets of 503KB in both cases!

(That's independent from the network MTU, like 1500B or 9000B =Jumbo.)

Consequently, the Ops rates are 240/s and 410/s resp., resulting

in the observed throughputs.

With the shown latencies (4.9ms and 2.8ms) the throughputs cannot go

much higher, as Ops x latency (= "channel utilization")

is already about 1 (means 100%) in both cases.

These scenarios might serve as a reference for other setups,

where findings can differ in several ways:

1) If the packet size is small, say 60KB or less, then the negotiation

between server and client would need to be carefully observed...

For example, it is usually negatively affected by SMB packet signing.

2) If much higher latencies are seen, one should

look at the server situation first (e.g. CPU load, disk IOPS).

3) If the "channel utilization" (Ops x average latency)

is much lower than 1.0 (or 100%), then the Isilon cannot to much

about it: The client is somehow unable to keep the server "busy".

Increasing network memory buffer sizes might help, for example.

Further observations are welcome!

-- Peter

Robert McNeal wrote: What are the Win 7 and 10.9 clients that are being used in your test?  Can you provide some detail on the hardware configuration and network topology for the test?  Do you have an open Support Request for this issue?

I'd love to see a support case for this, so we can dig in a bit more.  I agree with everything that everybody has said here.  I would suggest setting net.inet.tcp.delayed_ack to 2 instead of 0 as it seems to perform just as well as 0 (in my testing) and it's what OS X server sets by default.

Testing my 10.9 client against a 6.5.5 cluster, I see SMB2 writes going to the cluster that are only 61440 bytes, but on 7.0.2.4, I see 512KB writes.  I suspect the same is true for the reads, but I haven't tested that (yet).  What I see in a quick packet capture is that the Max Transaction, Max Read, and Max Write Size on 6.5.5 is reported at 64KB in the initial negotiate protocol response.  On 7.0.2, it's 1MB.  So, the newer version supports much larger writes, or would appear to.

I'm curious if we respond similarly to a Windows 7 client that connects to 6.5.5.

Please private message me if you do have a support case open.  This would cause a fairly large difference in performance for SMB2 Macs connecting to OneFS 6.5.x.

BernieC wrote: I'm curious if we respond similarly to a Windows 7 client that connects to 6.5.5.

Even though the server responds that the max read and write size is 64KB, Windows 7 batches multiple reads and writes together in order to get higher throughput.  I saw 10 reads all sent to the server at the same time on my client; your mileage may vary.  I saw 2 writes usually sent within quick succession from my Win7 VM client (admittedly, it's not really the fastest client, so something on faster hardware might behave differently/better).

The Mac, however, appears to behave much like it has in the past.  It reads and writes only one chunk of data at a time, and it's not even really going up to the maximum read/write size the server supports when contacting OneFS 6.5.5 or 7.0.2.  This behavior is going to limit the amount of data it can get from the cluster, while Windows 7's SMB stack really shines.

It looks like there may be some hope, though.  Like I saw, and Robert demonstrated, when the Mac contacts an Isilon OneFS 7.x server, the server supports SMB 2.1's "large MTU" feature which puts the maximum read and write sizes much higher (1MB).  At least for the write test I did, the Mac was writing at 512KB at a time, a nice 8x increase over OneFS 6.5.5.

So, for the highest SMB2 performance on the Mac, I'd recommend going to the latest OneFS 7.0.2 maintenance release.  OneFS 7.1 will also soon be an option.

Hey guys, just ran into this article on the net regarding SMB2 performance.

They mention a tuning parameter: max protocol = smb2 in your smb.conf

Looks like it's a tuning parameter for the SERVER, not the client, but wanted to throw it out there incase it can help anyone.

http://blog.backcovesoftware.com/post/66125194865/benchmarking-samba-2-on-os-x-10-9-mavericks