Best process to change subnets?

Question

Hi

I am relatively new to Isilon admin so please bear with me.

We are a radio/tv station with a number of live processes reading files live from isilion, as well as a number of virtual servers which have their host disks on isilon as well.

We have recently run into an IP shortage as our network was configured some years ago, and a proliferation of devices has caused us a shortfall. We have configured our router to accept a larger range (class c to b), but in order for anything in the second half of the range (we have extended from 192.168.2.1-254, up to 192.168.3.254) we need to drop a bit on the subnet on isilon.

Now, all our servers are in the .2 range, so even without changing Isilon they can still talk to it fine, but all of our DHCP machines will need to move into the .3 range.

I have read about suspending nodes and so forth, but most of the connections come from machines that have pretty much permanent connections to the array (and are all on node 1). The radio and TV playout systems are VERY inflexible in regards to network outages and so are the VM guests.

Suspending the nodes will just stop new connections but not cause the existing connections to fail over.

I also read that dropping a node should just cause an auto failover to the other nodes.

So the question I have is, what are the repercussions of changing subnets on the nodes, and more importantly, the smart connect IP?

Peter_Sero · Answer

So are you planning to change any actual IPs in use on the Isilon,

or just the netmask resp. prefix size?

Btw, are you feeling comfortable with all connections sitting on node 1?

You may want to check wether SmartConnect has been set up correctly,

so you get reasonable load balancing; even more important,

will node-failovers do no harm to the running playout services?

Cheers

-- Peter

geoffgummer · Answer

Hi Peter

All we want to do is change the subnet mask from 255.255.255.0 to 255.255.254.0.

Smartconnect had not been set up correctly, but it is now, and I do have connections on all the other nodes, however the core devices have permanent isilon mounts and havent switched over - I imagine they will just stay connected until such time as they are rebooted or we force them to change some how.

In respect to the failovers and playout, well, this is the question, I have no idea. I was hoping someone would know.

Is there a way to gracefully fail a node in order to force connections to move to another node? I havent been able to find a way. And if we do fail a node, is the failover seamless, or will it drop the connection? We have TV playout playing video files of up to an hour in length (4GB or so) live off Isilon. Ideally I'd like to be able to tell it ok, you can keep playing this one, but after that you need to get your file from a different node...

Peter_Sero · Answer

> All we want to do is change the subnet mask from 255.255.255.0 to 255.255.254.0.

OneFS allows this to be done on the fly.

Interestingly, testing it on a somewhat stressed virtual Isilon node running OneFS 8.0.1

I found it to cause NFS service stalls of few tenths of a second up to one second.

> In respect to the failovers and playout, well, this is the question, I have no idea. I was hoping someone would know.

Migrating an IP address to a different physical interface (on a different node)

needs to be tracked by the ARP protocol which has certain timeouts

that can cause stalls from several seconds to minutes. That's basic

networking functionality and not specific to Isilon; the Internet has

good articles on ARP caching, and you'll find the arping tool for flushing the cache.

(We are using NFS here, so I never bothered wether the Isilon

takes care of flushing the ARP cache during failovers.)

In the end you will need to find out how your environment behaves,

I'd recommend setting up a separate test subnet and test clients to experiment,

using your Isilon, or better, a test Isilon. NFS v3 connections will

stall but not fail or disconnect, SMB2 connection will disconnect

and usually re-connect, while ongoing SMB2 transfers will fail.

It depends on your playout systems how they can manage

stalls or disconnects. I would assume they buffer at least a couple of

seconds of a video stream and can handle certain transient outages.

It's also noteworthy that the newest versions of OneFS and SMB support HA.

> - I imagine they will just stay connected until such time as they are rebooted or we force them to change some how.

Have there been set up multiple IP addresses per node, as recommended?

If so, this command will redistribute the addresses across the cluster (OneFS 7.x syntax):

isi networks --sc-rebalance-all

(Expect the aforementioned ARP cache timeouts here).

hth

-- Peter

geoffgummer · Answer

Hi Peter

Thanks for your help so far.

We arent using NFS, only samba - its a largely windows environment here, and NFS is only used for backups from our linux servers.

I've done the subnet change, that was painless thanks for that.

Fortunately we are not having to migrate any IP's only the subnet mask needs to be changed (I made this decision because of ARP and a few other obvious reasons - like having to renumber 100 servers as well)

The rebalance command has not seemed to make much of a difference. We do not have multiple ip addresses per node (EMC did not ask for them), we have one lagg per node with one IP per lagg. I still have about 100 connections on node1 and only 7-8 on the other each (each).

Peter_Sero · Answer

Glad to hear the subnet change has worked well.

As for the re-balancing, it cannot work with only one IP address per node.

Isilon

Best process to change subnets?

Was this post helpful?