Daryn,

that sounds pretty good, you will need two subnets with one pool per subnet to give you that separation just like you mentioned. I use CIFS/NFS pools for management, i know some folks like to dedicate separate subnets ..your call/requirements.

Good paper to view https://support.emc.com/docu58740_Isilon-External-Network-Connectivity-Guide---Routing,-Network-Topologies,-and-Best-Pra…

I guess one of the confusions I get, as I continue to read the different papers and posts, and playing around with the simulator, one is with multiple subnets you end up with multiple default gateways, and one takes priority, so I was wondering how that was handled in the multiple subnet scenario or if that's where static routes come in.

The other was, I read in one paper where it says a OneFS cluster can only have one SSIP, but then when you create multiple subnets in the OneFS interface, each subnet has a place to specify an SSIP, which would mean you could/would have an SSIP for each subnet, so that's confusing me as well.

I'm using the simulator (with OneFS 8 on it) so this is where I'm getting  a little experience with the interface and concepts. 

there is only one default gateway, regardless of how many pools/subnets you have. If you have multiple subnets, the one configured with gateway priority 1 is your default gateway. Let's imagine your scenario:

subnet 192.168.1.0

mask: 255.255.255.0

gateway 192.168.1.254, priority 1

SSIP 192.168.1.1

pool name CIFS

Pool  ip allocation static , 192.168.1.2-253

Zone name: cifs.mycompany.com

in your AD DNS/Bind/Infoblox  you create NS Record cifs.mycompnay.com that points to SSIP 192.168.1.1

If clients tries to connect to cifs.mycompany.com Isilon will give them an IP from 192.168.1.2-253 range

subnet 192.168.2.0

mask: 255.255.255.0

gateway 192.168.2.254, priority 2

SSIP 192.168.2.1

pool name NFS

Pool  ip allocation dynamic, 192.168.2.2-253

Zone name: nfs.mycompany.com

in your AD DNS/Bind/Infoblox  you create NS Record nfs.mycompnay.com that points to SSIP 192.168.2.1

If clients tries to connect to nfs.mycompany.com Isilon will give them an IP from 192.168.2.2-253 range

So in this configuration your default gateway on the cluster is 192.168.1.254, because it has priority 1.  For routing you can rely on static routes or better yet enable source based routing.

Does it help ?

Thanks dynamo, that's a great illustration of what we're looking at, thanks for spending the time typing that one up.

When I had the simulator playing with one subnet things were simple and when I went to play with multiple, a lot a questions started popping up in regards to gateways, routing and multiple SSIP's and what not.  Thanks for clearing that up.

One thing in our environment is we have most of our NFS clients with an interface in a subnet dedicated to NFS only so I'm thinking of grabbing a set of those IP's for the NFS IP pool and I think I wouldn't need to bother with a gateway at all for NFS and just use the CIFS subnet as the default gateway, since it serves desktops all over campus.

Thanks again!

Daryn wrote: One thing in our environment is we have most of our NFS clients with an interface in a subnet dedicated to NFS only so I'm thinking of grabbing a set of those IP's for the NFS IP pool and I think I wouldn't need to bother with a gateway at all for NFS and just use the CIFS subnet as the default gateway, since it serves desktops all over campus. Thanks again!

how big is that subnet ? Think future Isilon expansion, for NFS IP pool, you want to have each Isilon node with a couple of IP address, so that when failover  happens they get spread out to multiple nodes, for example:

node1: 192.168.2.2, 192.168.2.3, 192.168.2.4

node2: 192.168.2.5, 192.168.2.6, 192.168.2.7

node3: 192.168.2.8, 192.168.2.9, 192.168.2.10

node4: 192.168.2.11, 192.168.2.12, 192.168.2.13

now let's imagine node1 going offline with hardware issues. Its IP addresses will be moved to other nodes in the cluster, something like this:

node2: 192.168.2.5, 192.168.2.6, 192.168.2.7, 192.168.2.2

node3: 192.168.2.8, 192.168.2.9, 192.168.2.10, 192.168.2.3

node4: 192.168.2.11, 192.168.2.12, 192.168.2.13, 192.168.2.4

you see how nicely and evenly they got spread out to other nodes in the cluster. If you only had one IP address per node, all that workload would move to just one node.  It might not be an issue in your environment because workload is not very high  but something for your to consider. 

Thanks, our current NFS subnet, upon further examination is pretty full so I'll need to see if the Linux team won't mind moving to a /23 net for that traffic since looks like I'll need more IP's than I think I would have.

How do you handle management traffic and replication, do you use a dedicated static pool for mgmt. or just say the cifs subnet and how about replication, I read in one of the docs it is best to use a dedicated subnet for that traffic, which does make sense.

Also, do you do any segmentation at the network port level, dedicating ports to specific traffic or just keep everything on the 10g pipes?

Thanks again for all the tips, really helping me out come up with a network plan.

I manage my cluster though the same IP as my cifs clients,  i don't have requirements to keep that isolated to a "special" network.  I mean by default apache and ssh listen on all IPs so you would need to modify apache and ssh settings to only allow connections from special subnets. It's doable but not something i need to do.

I use 10G interfaces for user traffic and 1G interfaces for replication traffic only. I do have a dedicated subnet for replication traffic, you have to use static IP pool for replication. My datacenters are 15 miles apart and we have dark fiber connecting them together so i actually have a layer 2 network that spans both datacenters so my replication subnet is the same on both Isilon clusters so i am not even going through any routers.  Not that it would be a big deal, i only replicate ~600G worth of new/changed files every night.

Our DR site is a few hundred miles away, we're in a possible hurricane zone, with different IP spaces that don't span sites, so I'll have to figure that one out, though would think it's just a matter of having an IP pool on either end and then using static routes to get back and forth.

One thing, well, one other thing that seems most confusing after reading thru a lot of papers and posts is the physical setup of the interfaces in the box.  Being that we would be using multiple subnets, all of which are separate vlans on the network, would we use LACP for the 10g ports on all the nodes in the cluster, but would we have to use vlan tagging since each of the subnets are their own vlans?

So that option on the subnets setup page for vlan tagging we'd check it and put the vlan number in there for the subnet, or is this really something we would need to collaborate with our network group?  We have Brocade VDX switches that do the vLAGs which I think do a little more than you're normal LACP as far as aggregating goes.  I just wasn't sure at what point you have to use vlan tagging.

Thanks again Dynamox, my whiteboard is almost full now!

Daryn wrote: Our DR site is a few hundred miles away, we're in a possible hurricane zone, with different IP spaces that don't span sites, so I'll have to figure that one out, though would think it's just a matter of having an IP pool on either end and then using static routes to get back and forth.

exactly, nothing special there as long as they can communicate.

Daryn wrote: One thing, well, one other thing that seems most confusing after reading thru a lot of papers and posts is the physical setup of the interfaces in the box.  Being that we would be using multiple subnets, all of which are separate vlans on the network, would we use LACP for the 10g ports on all the nodes in the cluster, but would we have to use vlan tagging since each of the subnets are their own vlans? So that option on the subnets setup page for vlan tagging we'd check it and put the vlan number in there for the subnet, or is this really something we would need to collaborate with our network group?  We have Brocade VDX switches that do the vLAGs which I think do a little more than you're normal LACP as far as aggregating goes.  I just wasn't sure at what point you have to use vlan tagging.

I am in a university environment with tons of subnets, i can relate .   It would be easier for me to show you what i did in my shop.  I assume you are in the US, east/west coast ?