Ask The Expert, OneFS and the Mac: Command+K to Awesome!

This Ask the Expert thread is now open!

Hello a litte bit of topic but

Is there a plan to push Isilon as a premary VMware storage in the next onefs release? We are currently use onefs 7.1.on S200, any improuvemt soon?

last question, should we put SSD in our S200 of VMware only storage (A lot of VM conected but not a lot of files)?

Hi Alain!

Thanks for asking your questions.  I'd recommend, in the future, to start a new thread so that we can properly answer your questions there.

I'm not completely aware of any new plans to reposition OneFS for VMware environments, although there were a number of performance improvements implemented in 7.1 that give us higher IOPs which may help.  It's possible someone else can give more details.

Regarding SSDs on your S200s.  If your entire workflow is nothing but VM, then there likely would not be a large benefit from SSD.  But, if you stored home directories or other highly latency-sensitive metadata on your nodes, SSDs would help quite a lot.

Thanks for the questions,
Bernie

So let's start with something that I'm quite excited about: SMB2 support in OS X.

Apple, in OS X Lion, moved away from Samba, instead implementing its own SMB client and server called smbx.  SMB2 support was something that had been rumored to be coming in Lion (10.7), but the SMB client in Lion and Mountain Lion (10.8) was sadly absent.  Funny enough, however, the SMB2 server is present, and can be confirmed by running /usr/sbin/smbd --help, which explains that that SMB2 support is experimental in those releases, and even Mavericks (10.9) where it is enabled by default.

Thankfully though, the SMB2 client in Mavericks is not experimental, and has been enabled by default in Mavericks.  What this means for the Mac users is that they can get much higher throughput now when connecting to OneFS 7.x clusters due to the SMB 2.1 support included in those releases.  We expect that performance for Macs with OneFS 6.5.x will be largely similar to SMB1 because that version of OneFS supports a smaller (64KB) maximum read and write size.  OneFS 7.x supports up to 1MB read and write sizes, which greatly boosts throughput on 10Gbit networks.  For more information, see this previous post on ECN.

Apple indicated in its Mavericks Core Technologies Overview that SMB2 is automatically used to share files between two Macs running OS X Mavericks.  That has not been my experience locally; AFP appears to still be used between my Macs.  But, SMB2 will be used automatically against Isilon clusters that support it (6.5.x and later).

While the compatibility guides do not reflect support for OS X Mavericks, it's something I'm working with our documentation group to update.  We do support OS X Mavericks in OneFS 6.5 and later, but we prefer customers use OneFS 7.0.x and later for maximum performance.

There is an issue I have seen with the SMB2 support in Mavericks.  Getting a list of SMB shares may fail when the Isilon cluster is running OneFS 6.5.x, particularly in environments with a large number of shares.  Connecting directly to the share, by its sharename (i.e. smb://server/share), will still work.  If you are having this issue in your environment please open up a support case and ask that your issue be escalated to Isilon engineering.

I look forward to future questions and comments.  Thanks!

Hi Bernie!

Thanks for taking the time out for answering our questions on all things OSX!

I know you did some work with OSX and NFS 4 with kerberos/AD. Can you share with us some of your findings and oberservations.

Hints,Tips, pitfalls etc.

Look forward to the thread!

Hi broreg,

Thanks for the note and thanks for reading!  OS X, NFSv4, and Kerberos are all very interesting topics.  I've been working on an update to the Mac Best Practices Guide (see the OneFS 6.5 version here) which takes into account all of these topics.  The updated guide is being reviewed by our Info Development team and should be published this quarter.  But, getting back to your question, here's what I've learned...

• NFSv4:  While the protocol is supported by OS X and I was, indeed, able to mount Isilon clusters with NFSv4, the idmapper component gave me a few headaches.  The idmapper on the Mac has to map the user@dns_domain UTF-8 strings sent from the server to known entities within OS X's own Directory Services framework.  Other OSes like Linux use a separate daemon (idmapd) to map those strings to UIDs and GIDs.  OS X has to do the same, but the idmapper does not seem to work correctly, yet, on OS X 10.8, or 10.9.  This forum discussion on Apple's discussion forums goes into good detail about what others have seen and links to a couple other sites where others have seen the same behavior.
  
  Because of these problems with the NFSv4 idmapper in OS X, I'm recommending customers shy away from NFSv4 and OS X for the time being.  As soon as these problems are taken care of, or if new information comes out about how to work around them, I'll share details in the best practices guide.
• Kerberized NFS: There is significant interest in this.  Kerberized NFS can allow environments to forgo UID and GID coherency and allow for consistent authentication into multiple authentication environments.  While the updated guide that will get published soon won't have many details on this, I am working on a KB which will detail some sample configurations for Kerberized NFSv3 from the Mac against OneFS when the backing authentication environment is Open Directory or Active Directory.
  
  Kerberized NFSv3 against Active Directory is a little tricky, however, because the Mac can only use DES encryption against 2003 and 2008 domains (AD doesn't use des3-cbc-sha1, which is the Mac's preferred encryption type).  That requires some changes to those domains, and to user accounts, to enable that encryption type, which some may be hesitant to do, due to DES's known shortcomings.  We have been investigating the feasibility of adding AES support to our NFS server in the future, however, so there's hope that we can avoid DES entirely.  Stay tuned for more info soon, and watch support.emc.com for the KB!

As it is currently, if customers would like to use NFS in an AD authentication environment while, at the same time, maintaining the ability to authenticate and authorize users by their AD credentials, they can do one of two things:

• Configure the NFS export to map all incoming requests to a specific AD user and/or group.  This is the same thing as NFS user squash.
• Use Services for Unix within AD to populate Mac users' AD accounts/groups with UIDs and GIDs, and then configure RFC 2307 support on the Active Directory provider in OneFS to pull those details out of AD.

NFS user squash is obviously the easier option, but does not allow for a high level of permissions granularity.  It may, however, work well in environments that only have a few Mac users in an otherwise larger AD domain.

Services for Unix implementation can be an involved process, particularly if no UIDs and GIDs have been added to accounts in AD.  Isilon professional services can assist customers with getting UIDs and GIDs for AD users and groups from the Mac and importing them into AD (details on the Mac side of this process can be found in this KB).  The Mac, by default, automatically generates UIDs and GIDs for AD accounts based on the GUID in an AD account (meaning, the UIDs and GIDs match for the same AD account, no matter which Mac you've logged into).  This enables us to get a full list of UIDs and GIDs from the Mac, import them into AD accounts, and then configure OneFS to get UIDs and GIDs directly from AD.  This allows OneFS to map UIDs and GIDs it retrieves from AD directly to Windows SIDs, which makes multi-protocol (NFS for Macs, SMB for Windows) environments possible, and manageable.

Again, thanks for the question, and please let me know if you have any others!

Thanks,
Bernie

NFS vs SMB, which is better for my Mac users?

Like a lot of things with NAS, the answer is often, "It depends..."  I find that, often, the answer comes when you answer a few other questions:

• What sorts of applications are you using, and with what types of workflows?
• What are your requirements for performance, and particularly metadata performance?
• What are your requirements for authentication and authorization?
• Will Windows clients be accessing the same data as the Macs?

The Mac Best Practices guide goes into these topics in the Workflow Consideration section, but I want to share some anecdotes that precipitated the creation of those questions.

Media & Entertainment environments typically like to see higher performance for their users, particularly in interactive or near-interactive video editing or video post-production workflows.  NFSv3 has been the better choice here, given that it provides high throughput on higher throughput networks like 10GbE.  SMB2 may, however, become more feasible in these environments when used with OneFS 7.x, given the performance improvements in that protocol.  I will still recommend NFSv3 in environments that make heavy use of wide directories (e.g. environments that process Cineon frames) because NFSv3 can retrieve metadata from Isilon much faster due to enhancements in its protocol.

SMB can be beneficial in media & entertainment workflows where Windows clients are also accessing the cluster.  Using SMB allows Mac-specific resource fork data to get stored within an Alternate Data Stream, which the Windows clients will not see.  That's better for some environments that don't want to have to deal with all of the dotbar (._) files that the Mac will create on an NFS export.  But, the Mac retrieves file and resource fork metadata via SMB much more slowly than it does via NFS, which can be frustrating for users in environments where thousands of items are stored within just one directory.  It often helps, with SMB, to architect directory structure that limits the total number of objects stored in one directory.  And, for both protocols, limit the use of the Finder's column view option!  (The Finder deserves its own post, so I'll save more details for then!)

SMB is a good choice for environments that require AD authentication.  Given that it will natively attempt Kerberos authentication on AD domain-joined Macs, it can be easier to get it authenticated to the cluster.  But, the Mac's SMB client will not show file system ACLs when connected to SMB shares (a note on this: OS X 10.5 does show POSIX-style ACLs, but 10.6 and later do not).

NFS authentication and authorization can be a little more challenging depending on the environment, as I mentioned earlier in this thread.  Care must be taken with NFS to make sure that the POSIX style permissions that are seen on the Mac are going to give the logged-in user access to an object.  Programs like Final Cut Pro rely on these permissions to determine access (a forthcoming KB will share more detail about this).  Finder, on the other hand, will often try the operation on the object even if the permissions do not seem to give it much access.

I think this is all I have for now.  Cheers!

So let's talk about this guy...

The Finder, for those not too familiar with the Mac, is the graphical file manager that has been with the Mac since the beginning (24 January 1984, and that date has special significance even now as I'll explain below).  The Finder has some interesting behaviors when it's used against network storage, such as:

• When loading a large list of items from a network server, Finder will not display those items to the user until the full list has been retrieved, including resource fork data.  This can be particularly painful for users on WAN connections who use SMB given that displaying a list of files requires getting the initial list, checking for resource fork data, and then reading that resource fork data.  That's not to say NFS on WAN is immune to the same problems, though, but for NFSv3 users, the Mac Best Practices guide recommends using a mount option called READDIRPLUS to speed up metadata retrieval by cutting down on network calls.  SMB has no analog to READDIRPLUS.  More info here, and in the Mac Best Practices Guide.
• Finder throws some really interesting, and confounding, error codes.  -36 is one you may have seen.  It is an I/O error (You can see the full list of of error codes in a file called MacErrors.h, which ships standard with Xcode and its SDKs.).  We've also got an article about error -36 in the knowledgebase.
• When mounting SMB shares and NFS exports in the Finder by SmartConnect name, we recommend users not click on the server name listed in the Finder sidebar under "SHARED".  Why?  Because clicking on that name forces the Mac to open a new connection to the SmartConnect name, which forces a new DNS connection, and will end up mounting the same network server a second time on the Mac.  Instead, browse to the mountpoint under "DEVICES" > "Mac's Name".  That may require users to enable the display of the Mac under the "DEVICES" section in the sidebar in the Finder preferences.
• When the Mac copies a folder to an SMB share, it sets it's creation date to 24 January 1984 12:00am US Pacific Time.  This is a special, hard-coded, date in the Finder and renders the folder unusable through the Finder, and grayed out.  After the copy is done, the Mac will set the creation date to its real creation date, which opens the folder up to normal use.  The Mac does this, in particular, to avoid users modifying folder contents in the Finder while they're being copied, but if a copy fails, the folder never gets its correct creation date set.  This can be resolved by using the "SetFile" command from the command line on the Mac.
• OS X 10.6.x through OS X 10.8.3 had a Finder bug which affected NFS users.  If you were viewing a folder on an NFS export and someone else happened to rename something in that folder, you'd lose access to that object until you remounted the export, because Finder wouldn't refresh its list of that folder automatically.  Apple fixed this bug in OS X 10.8.4, thankfully (I myself was hit by this on more than one occasion!).  More details in this KB.

We have a number of other KB articles about the Finder available here.  Thanks for reading, and have a great weekend everybody!

There's one other big thing about the Finder I didn't mention, and I think it's fairly important: try to avoid the use of Column View.  For those not familiar with column view, here's what it looks like in 10.9:

What this view shows you is a directory structure that's a few levels deep.  My example here is very, very simple, and I have used TinkerTool to enable the display of the path up in the title bar of the window (and a Finder option to show it at the bottom, as well).  Each column denotes a different level of the overall directory tree.  No doubt, column view is a popular view, and shows a lot of useful information.  That information comes at a price, however.

Remember in my previous post where I said that Finder won't display items to a user until it's retrieved the list of items and resource fork data for the folder?  Imagine how this plays out with Column view, where not one, but multiple levels of a directory tree are on display for the user.  Finder has to keep multiple directories refreshed and updated constantly, which takes more network calls between the Mac and the server.  I have seen some cases, with column view, where someone was trying to load just a small directory containing just a few items, but had to wait minutes for that directory to load.  When I was able to see what was happening on the network, I found that the Finder was doing directory listings and resource fork data retrieval for directories that were several levels higher in the tree.  Until Finder had completed its work there was it able to display the directory.  Not a great experience for the user, unfortunately.

I typically recommend list view, but I do so with a couple of extra recommendations:

• Disable "Calculate all sizes" - calculating all sizes for directories forces the Finder to do treewalks in that directory to determine how much space on disk the directory consumes.  This takes a long time and is, again, quite network intensive.
• Disable "Show icon preview" - Getting a preview of an icon entails going out to the server and reading in data from that file, which involves network calls that would be better served by the user's applications, rather than the Finder.

There are, of course, some Finder replacements out there which greatly alleviate shortcomings in the Finder's retrieval of file and directory data.  Pathfinder is one such option, and may work better for you.  But, for those stuck to using Finder, like me, these simple tips should hopefully improve your Finder experience against Isilon, or any NAS for that matter.

Thanks,
Bernie

Hello Bernie,

thanks for the great series of write ups in this thread!

Two more things one might be curious about (I am):

Modern OSX applications have replaced the "File->Save" function by an implicit version history per document, but only if the file system supports it. What, if at all possible, does it take to have versioned documents on an Isilon share (SMB or NFS)?

Another thing is Spotlight indexing. While it's probably a good thing to have indexing of network volumes DISabled in general, one might want to activate it in certain scenarios. I have seen OS X clients indexing NFS home folders in previous versions---and where the NFS server was an also running OS X. Is there any magic known about how the indexing of mounted network volumes could be controlled for OneFS 7.x? (The Isilon & Mac paper only talks about OneFS 6.5 and suggests using iSCSI...)

Cheers

-- Peter

Hi Peter,

Thanks for the comments.  Let me respond to them below.

Modern OSX applications have replaced the "File->Save" function by an implicit version history per document, but only if the file system supports it. What, if at all possible, does it take to have versioned documents on an Isilon share (SMB or NFS)?

Document versioning in OS X uses a special directory at the root of a volume in addition to a SQLite database in order to track revisions.  It also takes a developer who implements a specific framework in order to support this within the application.  (An overview is available in the excellent Ars Technica review of 10.7 available here.)  What it also appears to need is a filesystem that supports document versioning.  It looks like OS X just added an API, but didn't make any actual modifications to the underlying filesystem in order to support this, from what I can tell.  But, what it would need is some way to fool OS X into believing that a NAS device was eligible for document versions, and right now applications like TextEdit complain about the filesystem not supporting versioning when you make modifications to a document, save it to NAS, and then close the window.

I imagine any changes made to support document versioning may need to be made both in the framework in OS X and on the NAS device in order to understand how to handle specific calls to put data into that special directory.  Interesting request, and it might be something Apple would consider in the future.  By the way, I also tested this via AFP between two Macs... it's not even supported in that sort of environment.  It would seem it's only designed to work with direct-attached storage.

Another thing is Spotlight indexing. While it's probably a good thing to have indexing of network volumes DISabled in general, one might want to activate it in certain scenarios. I have seen OS X clients indexing NFS home folders in previous versions---and where the NFS server was an also running OS X. Is there any magic known about how the indexing of mounted network volumes could be controlled for OneFS 7.x? (The Isilon & Mac paper only talks about OneFS 6.5 and suggests using iSCSI...)

The reason iSCSI was mentioned in the best practices guide was somewhat similar to why document versioning only works with direct-attached storage - it boils down to where the Spotlight indices are built (.Spotlight-V100 directory on the root of a direct-attached volume).  But, it also boils down to the protocol being used between client and server.  AFP supports the notion of network Spotlight searches, which uses a special AFP call (FPSpotlightRPC, which just happens to be undocumented) in order to search through the server's Spotlight Index over the network.  Only the AFP protocol supports this call, even though Apple's own documentation for file serving in the 10.6 timeframe says that network spotlight is supported over SMB shares.  (I've tested network spotlight between a Mac SMB client and Mac SMB server; it didn't work.  It did work with AFP, however).  In the end, network Spotlight is only supported with AFP, and iSCSI was mentioned as a way to reshare a portion of Isilon filesystem using iSCSI and AFP in order to provide network Spotlight for clients.  Due to the layers of protocols and devices involved (AFP > Mac server > iSCSI > NAS), high performance would not be a guarantee.  And, you'd be limited to 16TB HFS+ volumes on iSCSI LUNs.  That defeats many of the benefits of scale out NAS.

I can certainly understand the interest in this topic!  Running a command like mdls (from a Mac) against a file gives a wealth of Spotlight information that the Mac keeps for it.  But, that same information is not created or Indexed on Isilon, and there's no way with NFS or SMB to share those Spotlight details over those protocols (the protocols just don't support the special Spotlight RPCs).  One thing that could help here is if we somehow created a metadata index for the filesystem structure to speed up name-based searches, but generating that index would be a very time consuming process without something like the FSEvents system constantly keeping tabs on directories.  That would be a significant undertaking.  Instead, one thing we recommend are digital asset management (DAM) solutions that create their own indices based on Isilon storage and then share those out to clients directly.  These sorts of solutions are very useful for media & entertainment workflows, in particular, as they offer a layer of abstraction between the client and the underlying storage, and also allow for workflow processes to be enforced.

Thanks for the questions and comments!

--Bernie

Thanks Bernie for sharing these insights!

It's kind of amazing what Apple can achieve with their pre-UNIX filesystem...

This Ask the Expert event has now concluded, but the conversations continue here in the Isilon Support Forum. Thanks to Bernie Case for all his advice and insight!

Thank you to all those who participated in this event!

It's kind of amazing what Apple can achieve with their pre-UNIX filesystem...

Agreed!  One such big additions to their file system was added with 10.5, when Time Machine was introduced.  Using directory hard links in the file system is what makes Time Machine possible and very efficient.

And believe me, I have spent quite a lot of time trying to get Time Machine working via NFS or SMB.  10.5 and 10.6 could be tricked into doing backups via those protocols, but later versions really only support "unsupported" servers so long as they're running AFP :-(

Well, thanks for the ask the expert thread.  It was fun!

A couple notes...

1) We ran into problems with the finder having huge stalls when we mounted a directory with about 3,000 files.  I do not remember if it was NFS or SMB, I kind of think I tried both.  sorting things into smaller directories, under 1,000 each, made display about 100x faster. 

2) Be very wary of running time machine against a non-apple-supported AFP server.  netatalk has nominal support, but I and others I've talked to have had reliability problems with time machine and netatalk.