Ask The Expert, OneFS and the Mac: Command+K to Awesome!

[MRWA]

Welcome to this EMC Support Community Ask the Expert conversation. This is an opportunity to learn about and discuss the way Macs and OneFS interoperate.

Topics of discussion we could have in this thread may include:

• SMB2 in OS X Mavericks
• NFS vs SMB, which is better for my Mac users?
• User identity with OneFS and Mac users
• Multi-protocol and multi-platform data access with OneFS
• Maintaining high metadata performance with OS X and OneFS

This discussion begins on 11/11/2013 and concludes on 11/22/2013 Get ready by bookmarking this page or signing up to receive email notifications.

Your host is Bernie Case!

Name:	Bernie Case
Occupation:	Tech Support Engineer IV
Biography:	My original career path was taking me toward audio production and engineering, but I got mixed up in a "bad" crowd of IT folks during the dot-com boom of the late 1990s and that's where I've focused my talents ever since. I use my knowledge and experience with networking protocols, media and entertainment workflows, Mac OS X, *nix, Windows, and system administration to help solve interesting, complex, and sometimes urgent problems.
URL:	http://www.emc.com
Expertise:	Multiple flavors of Mac OS, including pre-OS X through OS X 10.9, networking protocols (OSI layer 2 and up), system administration.
Company:	EMC Isilon
City:	Seattle
State:	WA
Country:	USA
Display Name:	Bernie Case
Web Site:	http://www.emc.com

[MRWA]

This Ask the Expert thread is now open!

[arichard1]

Hello a litte bit of topic but

Is there a plan to push Isilon as a premary VMware storage in the next onefs release? We are currently use onefs 7.1.on S200, any improuvemt soon?

last question, should we put SSD in our S200 of VMware only storage (A lot of VM conected but not a lot of files)?

[BernieC2]

Hi Alain!

Thanks for asking your questions.  I'd recommend, in the future, to start a new thread so that we can properly answer your questions there.

I'm not completely aware of any new plans to reposition OneFS for VMware environments, although there were a number of performance improvements implemented in 7.1 that give us higher IOPs which may help.  It's possible someone else can give more details.

Regarding SSDs on your S200s.  If your entire workflow is nothing but VM, then there likely would not be a large benefit from SSD.  But, if you stored home directories or other highly latency-sensitive metadata on your nodes, SSDs would help quite a lot.

Thanks for the questions,
Bernie

[BernieC2]

So let's start with something that I'm quite excited about: SMB2 support in OS X.

Apple, in OS X Lion, moved away from Samba, instead implementing its own SMB client and server called smbx.  SMB2 support was something that had been rumored to be coming in Lion (10.7), but the SMB client in Lion and Mountain Lion (10.8) was sadly absent.  Funny enough, however, the SMB2 server is present, and can be confirmed by running /usr/sbin/smbd --help, which explains that that SMB2 support is experimental in those releases, and even Mavericks (10.9) where it is enabled by default.

Thankfully though, the SMB2 client in Mavericks is not experimental, and has been enabled by default in Mavericks.  What this means for the Mac users is that they can get much higher throughput now when connecting to OneFS 7.x clusters due to the SMB 2.1 support included in those releases.  We expect that performance for Macs with OneFS 6.5.x will be largely similar to SMB1 because that version of OneFS supports a smaller (64KB) maximum read and write size.  OneFS 7.x supports up to 1MB read and write sizes, which greatly boosts throughput on 10Gbit networks.  For more information, see this previous post on ECN.

Apple indicated in its Mavericks Core Technologies Overview that SMB2 is automatically used to share files between two Macs running OS X Mavericks.  That has not been my experience locally; AFP appears to still be used between my Macs.  But, SMB2 will be used automatically against Isilon clusters that support it (6.5.x and later).

While the compatibility guides do not reflect support for OS X Mavericks, it's something I'm working with our documentation group to update.  We do support OS X Mavericks in OneFS 6.5 and later, but we prefer customers use OneFS 7.0.x and later for maximum performance.

There is an issue I have seen with the SMB2 support in Mavericks.  Getting a list of SMB shares may fail when the Isilon cluster is running OneFS 6.5.x, particularly in environments with a large number of shares.  Connecting directly to the share, by its sharename (i.e. smb://server/share), will still work.  If you are having this issue in your environment please open up a support case and ask that your issue be escalated to Isilon engineering.

I look forward to future questions and comments.  Thanks!

[broreg]

Hi Bernie!

Thanks for taking the time out for answering our questions on all things OSX!

I know you did some work with OSX and NFS 4 with kerberos/AD. Can you share with us some of your findings and oberservations.

Hints,Tips, pitfalls etc.

Look forward to the thread!

[BernieC2]

Hi broreg,

Thanks for the note and thanks for reading!  OS X, NFSv4, and Kerberos are all very interesting topics.  I've been working on an update to the Mac Best Practices Guide (see the OneFS 6.5 version here) which takes into account all of these topics.  The updated guide is being reviewed by our Info Development team and should be published this quarter.  But, getting back to your question, here's what I've learned...

• NFSv4:  While the protocol is supported by OS X and I was, indeed, able to mount Isilon clusters with NFSv4, the idmapper component gave me a few headaches.  The idmapper on the Mac has to map the user@dns_domain UTF-8 strings sent from the server to known entities within OS X's own Directory Services framework.  Other OSes like Linux use a separate daemon (idmapd) to map those strings to UIDs and GIDs.  OS X has to do the same, but the idmapper does not seem to work correctly, yet, on OS X 10.8, or 10.9.  This forum discussion on Apple's discussion forums goes into good detail about what others have seen and links to a couple other sites where others have seen the same behavior.
  
  Because of these problems with the NFSv4 idmapper in OS X, I'm recommending customers shy away from NFSv4 and OS X for the time being.  As soon as these problems are taken care of, or if new information comes out about how to work around them, I'll share details in the best practices guide.
• Kerberized NFS: There is significant interest in this.  Kerberized NFS can allow environments to forgo UID and GID coherency and allow for consistent authentication into multiple authentication environments.  While the updated guide that will get published soon won't have many details on this, I am working on a KB which will detail some sample configurations for Kerberized NFSv3 from the Mac against OneFS when the backing authentication environment is Open Directory or Active Directory.
  
  Kerberized NFSv3 against Active Directory is a little tricky, however, because the Mac can only use DES encryption against 2003 and 2008 domains (AD doesn't use des3-cbc-sha1, which is the Mac's preferred encryption type).  That requires some changes to those domains, and to user accounts, to enable that encryption type, which some may be hesitant to do, due to DES's known shortcomings.  We have been investigating the feasibility of adding AES support to our NFS server in the future, however, so there's hope that we can avoid DES entirely.  Stay tuned for more info soon, and watch support.emc.com for the KB!

As it is currently, if customers would like to use NFS in an AD authentication environment while, at the same time, maintaining the ability to authenticate and authorize users by their AD credentials, they can do one of two things:

• Configure the NFS export to map all incoming requests to a specific AD user and/or group.  This is the same thing as NFS user squash.
• Use Services for Unix within AD to populate Mac users' AD accounts/groups with UIDs and GIDs, and then configure RFC 2307 support on the Active Directory provider in OneFS to pull those details out of AD.

NFS user squash is obviously the easier option, but does not allow for a high level of permissions granularity.  It may, however, work well in environments that only have a few Mac users in an otherwise larger AD domain.

Services for Unix implementation can be an involved process, particularly if no UIDs and GIDs have been added to accounts in AD.  Isilon professional services can assist customers with getting UIDs and GIDs for AD users and groups from the Mac and importing them into AD (details on the Mac side of this process can be found in this KB).  The Mac, by default, automatically generates UIDs and GIDs for AD accounts based on the GUID in an AD account (meaning, the UIDs and GIDs match for the same AD account, no matter which Mac you've logged into).  This enables us to get a full list of UIDs and GIDs from the Mac, import them into AD accounts, and then configure OneFS to get UIDs and GIDs directly from AD.  This allows OneFS to map UIDs and GIDs it retrieves from AD directly to Windows SIDs, which makes multi-protocol (NFS for Macs, SMB for Windows) environments possible, and manageable.

Again, thanks for the question, and please let me know if you have any others!

Thanks,
Bernie

[BernieC2]

NFS vs SMB, which is better for my Mac users?

Like a lot of things with NAS, the answer is often, "It depends..."  I find that, often, the answer comes when you answer a few other questions:

• What sorts of applications are you using, and with what types of workflows?
• What are your requirements for performance, and particularly metadata performance?
• What are your requirements for authentication and authorization?
• Will Windows clients be accessing the same data as the Macs?

The Mac Best Practices guide goes into these topics in the Workflow Consideration section, but I want to share some anecdotes that precipitated the creation of those questions.

Media & Entertainment environments typically like to see higher performance for their users, particularly in interactive or near-interactive video editing or video post-production workflows.  NFSv3 has been the better choice here, given that it provides high throughput on higher throughput networks like 10GbE.  SMB2 may, however, become more feasible in these environments when used with OneFS 7.x, given the performance improvements in that protocol.  I will still recommend NFSv3 in environments that make heavy use of wide directories (e.g. environments that process Cineon frames) because NFSv3 can retrieve metadata from Isilon much faster due to enhancements in its protocol.

SMB can be beneficial in media & entertainment workflows where Windows clients are also accessing the cluster.  Using SMB allows Mac-specific resource fork data to get stored within an Alternate Data Stream, which the Windows clients will not see.  That's better for some environments that don't want to have to deal with all of the dotbar (._) files that the Mac will create on an NFS export.  But, the Mac retrieves file and resource fork metadata via SMB much more slowly than it does via NFS, which can be frustrating for users in environments where thousands of items are stored within just one directory.  It often helps, with SMB, to architect directory structure that limits the total number of objects stored in one directory.  And, for both protocols, limit the use of the Finder's column view option!  (The Finder deserves its own post, so I'll save more details for then!)

SMB is a good choice for environments that require AD authentication.  Given that it will natively attempt Kerberos authentication on AD domain-joined Macs, it can be easier to get it authenticated to the cluster.  But, the Mac's SMB client will not show file system ACLs when connected to SMB shares (a note on this: OS X 10.5 does show POSIX-style ACLs, but 10.6 and later do not).

NFS authentication and authorization can be a little more challenging depending on the environment, as I mentioned earlier in this thread.  Care must be taken with NFS to make sure that the POSIX style permissions that are seen on the Mac are going to give the logged-in user access to an object.  Programs like Final Cut Pro rely on these permissions to determine access (a forthcoming KB will share more detail about this).  Finder, on the other hand, will often try the operation on the object even if the permissions do not seem to give it much access.

I think this is all I have for now.  Cheers!

[BernieC2]

So let's talk about this guy...

The Finder, for those not too familiar with the Mac, is the graphical file manager that has been with the Mac since the beginning (24 January 1984, and that date has special significance even now as I'll explain below).  The Finder has some interesting behaviors when it's used against network storage, such as:

• When loading a large list of items from a network server, Finder will not display those items to the user until the full list has been retrieved, including resource fork data.  This can be particularly painful for users on WAN connections who use SMB given that displaying a list of files requires getting the initial list, checking for resource fork data, and then reading that resource fork data.  That's not to say NFS on WAN is immune to the same problems, though, but for NFSv3 users, the Mac Best Practices guide recommends using a mount option called READDIRPLUS to speed up metadata retrieval by cutting down on network calls.  SMB has no analog to READDIRPLUS.  More info here, and in the Mac Best Practices Guide.
• Finder throws some really interesting, and confounding, error codes.  -36 is one you may have seen.  It is an I/O error (You can see the full list of of error codes in a file called MacErrors.h, which ships standard with Xcode and its SDKs.).  We've also got an article about error -36 in the knowledgebase.
• When mounting SMB shares and NFS exports in the Finder by SmartConnect name, we recommend users not click on the server name listed in the Finder sidebar under "SHARED".  Why?  Because clicking on that name forces the Mac to open a new connection to the SmartConnect name, which forces a new DNS connection, and will end up mounting the same network server a second time on the Mac.  Instead, browse to the mountpoint under "DEVICES" > "Mac's Name".  That may require users to enable the display of the Mac under the "DEVICES" section in the sidebar in the Finder preferences.
• When the Mac copies a folder to an SMB share, it sets it's creation date to 24 January 1984 12:00am US Pacific Time.  This is a special, hard-coded, date in the Finder and renders the folder unusable through the Finder, and grayed out.  After the copy is done, the Mac will set the creation date to its real creation date, which opens the folder up to normal use.  The Mac does this, in particular, to avoid users modifying folder contents in the Finder while they're being copied, but if a copy fails, the folder never gets its correct creation date set.  This can be resolved by using the "SetFile" command from the command line on the Mac.
• OS X 10.6.x through OS X 10.8.3 had a Finder bug which affected NFS users.  If you were viewing a folder on an NFS export and someone else happened to rename something in that folder, you'd lose access to that object until you remounted the export, because Finder wouldn't refresh its list of that folder automatically.  Apple fixed this bug in OS X 10.8.4, thankfully (I myself was hit by this on more than one occasion!).  More details in this KB.

We have a number of other KB articles about the Finder available here.  Thanks for reading, and have a great weekend everybody!

[BernieC2]

There's one other big thing about the Finder I didn't mention, and I think it's fairly important: try to avoid the use of Column View.  For those not familiar with column view, here's what it looks like in 10.9:

What this view shows you is a directory structure that's a few levels deep.  My example here is very, very simple, and I have used TinkerTool to enable the display of the path up in the title bar of the window (and a Finder option to show it at the bottom, as well).  Each column denotes a different level of the overall directory tree.  No doubt, column view is a popular view, and shows a lot of useful information.  That information comes at a price, however.

Remember in my previous post where I said that Finder won't display items to a user until it's retrieved the list of items and resource fork data for the folder?  Imagine how this plays out with Column view, where not one, but multiple levels of a directory tree are on display for the user.  Finder has to keep multiple directories refreshed and updated constantly, which takes more network calls between the Mac and the server.  I have seen some cases, with column view, where someone was trying to load just a small directory containing just a few items, but had to wait minutes for that directory to load.  When I was able to see what was happening on the network, I found that the Finder was doing directory listings and resource fork data retrieval for directories that were several levels higher in the tree.  Until Finder had completed its work there was it able to display the directory.  Not a great experience for the user, unfortunately.

I typically recommend list view, but I do so with a couple of extra recommendations:

• Disable "Calculate all sizes" - calculating all sizes for directories forces the Finder to do treewalks in that directory to determine how much space on disk the directory consumes.  This takes a long time and is, again, quite network intensive.
• Disable "Show icon preview" - Getting a preview of an icon entails going out to the server and reading in data from that file, which involves network calls that would be better served by the user's applications, rather than the Finder.

There are, of course, some Finder replacements out there which greatly alleviate shortcomings in the Finder's retrieval of file and directory data.  Pathfinder is one such option, and may work better for you.  But, for those stuck to using Finder, like me, these simple tips should hopefully improve your Finder experience against Isilon, or any NAS for that matter.

Thanks,
Bernie