NFSv4 and CIFS in a mixed environment.

We are putting into production mixed cifs, nfs3 and kerberized nfsv4 environment in very near future. Hopefully startting next week.

Accespolicy is mixed. We use AD's kerberos and really would like to use AD's ldap as well but right now the groups don't work. Uids are comng from AD's ldap but groups come from files.

NFSv4 comes to production little by little so we can educate users enough to work with that. Cifs and nfsv4 are similar enough that they won't cause that much problems in the cifs side. Actually the celerra is handling ACLs nicely.

Some of our users (~5000) have separate homes for cifs and nfs so there's lot of migrating to do. It's gonna be painful and need a lot of work but we think it's the only way to go in the end.

This was one of our main reasons to buy celerra.

--
Jussi

I'm currently in the process of something almost exactly the same as this. I'm curious how it's been going for you?

I'm a Unix admin so I'm a little biased, but I've not been super impressed with the mixed environment options and their behavior. So far I've found the Unix bit mode mapping in MIXED mode to be untenable (0707 for just about everything). MIXEDCOMPAT seems a little bit better but it still gets interpreted incorrectly once you twiddle something in the Security tab on the Windows side. I don't understand what's so hard about this. Samba has done it pretty well for ages.

I would really like to have Kerberized NFSv4 (sec=krb5) on the clients but there's a couple things currently standing in the way of me even testing that. The Celerra was configured originally to get it into the AD domain. We have separately (but with consistent data) an MIT KRB5 domain for our Linux clients. I've yet to find a document (though haven't been looking long) that says how exactly to configure the Celerra to allow for krb5 NFSv4 mounts from clients, whether I can have MIT and AD KRB5 play along side each other on the Celerra, or if I have to setup a cross-realm trust.

My main reason for wanting Kerberized NFSv4 is to get around our 16 group limit. Currently the emcgetsd utility dies rather ungracefully if you have 16 groups, not to mention the other hassles that more than 16 groups entails.

Thanks,
Brian

This is pretty much what we've seen too. We've been playing with OpenSolaris with a view to looking at ZFS and it's mode mapping system seems more sane and possibly even better than Samba as it support native NFSv4 ACLs at the filesystem level.

Even when using NFSv4/CIFS only onto the Celerra the end user confusion caused by the POSIX bits is causing big problems. There's also the ACE ordering issue but there is a Celerra config setting to workaround that forces the windows ordering for all ACL entries.

I will admit that I think some of the problem here is that native linux clients and tools are generally assuming that any basic ACL support they may have is based on the posix draft standard which is different again to the NFSv4/CIFS model.

For now we're just not using multiprotocol access.

A related issue is the non support for AD paging in LDAP which truncates group membership details. At least member/uniqueMember semantics are supported now, if only to about 1500 members per group.

I can now see why many Universities that require cross platform filesystems have opted for OpenAFS or similar. It may be non-native to all the platforms, but at least it's consistent between them.

Agreed on all counts.  Ah, to use ZFS and Samba ...

We were eventually able to work out some of the 707 modes.  Part of it was due to the order in which you set the group owner and then ACL.  I forget which had to come first, but if you didn't do it that way it treated the group ACL as an "other" ACL.  In MIXED mode it appears that any sort of non-(user|group)-owner ACL gets mapped into the other bits rather than using POSIX extended attributes or ACLs.

We've also since ditched MIXED mode and went with NATIVE so that each side can have their own bits.  You can expose the Unix mode bits to CIFS clients as extra ACEs using the acl.extacl param.

The final nail in the coffin was the executable bit.  Apparently Windows wants everything down to .txt files to be executable for "normal" operations while most Unix admins think that's crazy.

Now the issues are:

1) emcgetsd, emcsetsd doesn't work when your user has more than 16 groups, so NFS has no ability to see what the CIFS ACLs are set to.  This fails client side!

2) No ACLs for NFS in NATIVE mode.  Neither POSIX ACLs nor NFS4 ACLs.

The second can only kind of be worked around by using ntcredential to have the server ignore the RPC SYS_AUTH groups and do it's own lookup.  The trouble is that it only works for RWX requests on the file's content, not it's attributes, so chown/chmod/chgrp all fail.  I have wrappers to use sg to get around the trouble with chown/chgrp, but chmod is proving to be more difficult.

BTW, for Linux NFS4 ACLs I've using these tools:

http://www.citi.umich.edu/projects/nfsv4/linux/

Cheers,

Brian

Whith ZFS you don't have to use Samba, there's a new native CIFS server in OpenSolaris that get very close to doing the right thing (tm) as it stores one set of ACLs based on the NFSv4 model as native ZFS attributes. AFAIK Samba doesn't do as good a job of this.

We've pretty much gone CIFS only now with most of our shared areas. Those that need NFS are being split off into cheaper more flexible storage systems.

ZFS is nice, but Solaris still has a brain-dead 32 groups per user limited hard wired into the kernel. You can avoid this in part with NFSv4 and CIFS as those lookups check the ACLs but its still broken from a local user login point of view. A real pain when you're trying to offer a UNIX compute service or similar.