Draining IP connections to an external interface

Question

I know there is a way to drain connections to a node in preparation for a rolling upgrade, but am interested in seeing if there is a graceful way to drain connections to a front end interface.

Hello, I am rather new to Isilon and am in a situation where we need to change IP addressing from the current 192.168.x.x scheme to a 10.x.x.x scheme. Part of the challenge is that we need to introduce new shares for application access on the new network, without breaking - or at least controlling the breakage for - existing applications that write directly to the Isilon.

Here is the scenario:

- 4x NL-Nodes

- Each NL node has both ext-1 and ext-2 connected in a single IP pool. These are individual, not aggregated connections.

- This is a hospital with very limited maintenance windows

- Currently existing data will eventually be re-structured and re-shared utilizing new Smartconnect zones

- The applications access SMB shares

Here is the question:

Is there a way to "drain off" connections to one external interface per node?

For example, we would like to stop accepting new connections on ext-2, allow it be become idle through attrition and then move it out of the current IP pool. It would then be placed in the new IP pool. Then we would fix the mappings of the applications to point to the new smartconnect zone (reflecting the new IP range). Once that is complete, the ext-1 interfaces would theoretically be idle and we could then move those interfaces to the new IP pool.

Is this a responsible approach, or is there a recommended White Paper to illustrate how to go about this with minimal interruption to the business?

Any guidance is greatly appreciated. Thank you.

dynamox · Answer

suspend one node a time ? isi networks modify pool --name=subnet_3542_vlan:admin_3542_pool --sc-suspend-node=1

crklosterman · Answer

DHC,

How many nodes are in the cluster? If it's 4 or more (why 4? because then you always have at least 1 node for redundancy, I know someone will ask this question), the simplest approach from my perspective would be:

1. Build the new subnet and pool on the cluster, add no interfaces to the pool yet.

2. Pick a couple of nodes and suspend them from their current pool.

isi networks modify pool subnet0:pool0 --sc-suspend-node=3

isi networks modify pool subnet0:pool0 --sc-suspend-node=4

3. Now watch as the connections to those nodes trickle off. When all connections to those nodes are gone, remove the interfaces from the relevant nodes

isi networks modify pool subnet0:pool0 --remove-ifaces 3:10gige-1

isi networks modify pool subnet0:pool0 --remove-ifaces 3:10gige-2

isi networks modify pool subnet0:pool0 --remove-ifaces 4:10gige-1

isi networks modify pool subnet0:pool0 --remove-ifaces 4:10gige-2

4. Now add these same interfaces to the new subnet/pool and move over the physical connections, or change the VLANS on the switch(es) in question.

isi networks modify pool subnet1:pool1 --add-ifaces 3:10gige-1

isi networks modify pool subnet1:pool1 --add-ifaces 3:10gige-2

isi networks modify pool subnet1:pool1 --add-ifaces 4:10gige-1

isi networks modify pool subnet1:pool1 --add-ifaces 4:10gige-2

5. So now you have assuming again 4 nodes, 2 up and running on each subnet. The question now becomes how to move the load over. If it were my cluster, I would pick a change window, and remove the smartconnect zone name from subnet0:pool0, and then add that same name to subnet1:pool1, like this:

isi networks modify pool subnet0:pool0 --zone=

isi networks modify pool subnet1:pool1 --zone=pool1

Also you'll need to change at this point the NS delegation from delegating to the A record for subnet0's SSIP to instead delegating to the A record for subnet1's SSIP.

This should not interrupt any current ongoing IO, just new requests while you make the changes.

6. Now watch the client IO trickle off of nodes 1 and 2. When there is nothing left, or when you're tired of waiting, and the number of connections is at a minimum, remove the interfaces for nodes 1 and 2 from subnet0:pool0, add them to subnet1:pool1, and then swapover the cables, or again, update the VLANs on the switchports.

7. Done.

I know that there is more to it than this, but these are the highlights, and order of operations hope this helps,

~Chris Klosterman

Senior Solution Architect

EMC Isilon Offer & Enablement Team

chris.klosterman@emc.com

twitter: @croaking

dynamox · Answer

--sc-suspend-node

Suspends SmartConnect DNS query responses for the specified node. While

suspended, SmartConnect does not return IP addresses for this node, but allows

active clients to remain connected.

dynamox · Answer

my NS records point to CNAME. If you are going to change CNAME make sure to work with your DNS team to lower TTL before the migration (depending on default TTL time) and then changing it back after the migration.

DHCSTI · Answer

Thank you very much for your detailed response, it is very helpful. The way I read it though, we will want to do this either before we introduce any new applications and smartconnect zones, or well after.

In any case, there will be significant coordinate of effort to accomplish the IP change.

Thanks!

Isilon

Draining IP connections to an external interface

Was this post helpful?