Isilon x200 (3 nodes) install question

last install that i did , we used EMC internal network for IB subnets, that "should" guarantee to never overlap with your internal environment.

ib0 - 128.221.253.0

ib1 - 128.221.252.0

lo0 - 128.221.254.0

Isilon Best Practices are to use the following Ranges:

int-a: 128.221.252.1-254 /24

int-b: 128.221.253.1-254 /24

failover: 128.221.254.1-254 /24

Our Professional Services Organization does not even ask customers what to use any longer, we just use these.

Why?

1st

For those of you with VNX or Celerra experience, these 3 ranges should look very familiar, they are the same ranges used internally on a Unified Celerra or VNX.  EMC owns these IP blocks, so they will never be used on any production system on the public internet, ever.  In fact if you check the ARIN registry you'll see that EMC owns the 128.0.0.0/8 range.

2nd part of the question is usually:

Why give a 3 node cluster 254 addresses?

Because changing back-end IP ranges requires cluster-wide reboots, which you never want to have a need to do for a reason as mundane as this.  The largest clusters currently supported are 144 nodes, so 254 addresses more than meets those needs.

3rd

This is a DDR or QDR Infiniband Network, it will never be connected to a customer's routable network, and as such the IPs used on it are discrete. and have no effect on the customer's front-end networking environment.

Hope this helps clarify the topic.

Chris Klosterman

EMC Isilon Senior Solution Architect

email: chris.klosterman@emc.com

twitter: @croaking

I have 8 running clusters and all of them have the same subnet ranges for IB.  I have them registered in our internal address space so nobody else can use them and I know they'll never need to route outside of the cluster.

Thanks for answer.

I am wondering why use failover option when int-a, int-b, failover are in different subnets?

What is goal to failover on another non-routed network? In another words, what for isilon use failover option?

If customer use 2 IB switches in cluster but does not configured failover, what will happen if one of switches will fail?

If the customer does not configure failover then the nodes will not failover if the switch it is using goes down. The results will vary based on which node is active on which switch as well as the switch that fails. With the above discussion of private non-routable subnets, this should not be a problem as it should take nothing from the customer's network address space and they do not need to manage this once it is initially configured.

In fact to be perfectly clear, EMC Professional Services uses 3 ranges as our standard during new cluster installations.

They are:

128.221.252.1-.254/24 (int-a)

128.221.253.1-.254/24 (int-b)

128.221.253.1-.254/24 (failover)

To those of you with VNX or Celerra experience these may look oddly familiar.  Celerra has long used these addresses for its own internal networks.  This is because EMC owns the 128.0.0.0/8 Class A address, and therefore we are able to guarantee that these 3 class-C networks will never be used on the open internet.  Also the infiniband networks are non-routeable IBoIP (Infiniband over IP), and thus a customer's external network would never touch the IB network.  Also dedicating 254 addresses ensures that you can grow the cluster as large as you would wish, (current maximum cluster size is 144 nodes), and never have to allocate more back-end IPs.

Hope this helps explain it to the curious.

Chris Klosterman

Senior SA

EMC Isilon O&E Team

chris.klosterman@emc.com

twitter: @croaking

Thank you all for explanation and help!!!

Best regards