change isilon cluster default gateway

"The" default gateway can be set (modified) with

route change default

But beware, "the" default gateway is used when

any packets are being sent to an address OUTSIDE of the configured subnets.

If any of your client hosts are NOT directly located within your 

Internal-P/S (which appear not  to be the cluster's interconnects)

nor External subnets,

then any packets sent to those clients will be routed through the

new default gateway,

and therefore will leave the cluster through the "Management" subnet.

Is this really what you are aiming at?

(Probably you might want to choose the "External" subnet's gateway as default?)

Peter

Thanks Peter, the route change default gave an error :

#route change default 10.250.25.254

route: writing to routing socket: Address already in use

change net default: gateway 10.250.25.254: gateway uses the same route

But I tried changing the priorities of the gateways for the subnets (management to priority 1, all other to 2)

#isi networks list subnets

Name                        Subnet      Gateway:Prio      SC Service Pools

--------------- ------------------ ------------------ --------------- -----

Internal-P          10.250.14.0/24    10.250.14.254:2  10.250.14.239    1

Internal-S          10.250.16.0/24    10.250.16.254:2  10.250.16.239    1

External          157.150.187.0/24  157.150.187.254:2 157.150.187.239    1

Management          10.250.25.0/24    10.250.25.254:1  10.250.25.239    2

and that changed the cluster default gateway to what I wanted :

# netstat -rn

Routing tables

Internet:

Destination        Gateway            Flags    Refs      Use  Netif Expire

default            10.250.25.254      UG          0   991445  vlan3

I'm still confused how the subnet gateways and their priorities affect routing - I could not find any information about that.

By default each subnet will have a route for it's gateway and all traffic in that subnet will transverse that network...  Otherwise each node has a default gateway..  gateway priority is really just a ranking of 1 (highest) to 10 (lowest)  which specify's which subnet will be the default on a node with more than one subnet.

So if a node has just one subnet it will be the default.  If a node as two, one with priority 5 and one with 8) the default with be the subnet with priority 5.

Mark,

i also could not understand the purpose of default gateway for each subnet, did not make any sense to me because a node can only have one default gateway for the whole node. So what you are saying is that gateway priority controls which gateway becomes the default gateway ?

isilon1-2# isi networks list subnets
Name                        Subnet       Gateway:Prio      SC Service Pools 
--------------- ------------------ ------------------ --------------- ----- 
admin_3428_vlan    10.221.144.0/24     10.221.144.2:1   10.221.144.10     2 
dmz_3039_vlan      10.228.144.0/24     10.228.144.2:2   10.228.144.10     2

so if i were to change gateway priority for my dmz subnet my default gateway would change to 10.228.144.2 ?

Right now we have to set static routes to force traffic to certain subnets to go through specific interfaces.

Thanks

That is was got me confused, too.

But Mark is right, and the Isilon online help makes it also clear:

Gateway priority: The priority of the subnet's gateway in relation to

other subnets' gateways for nodes that are assigned to more than one

subnet. Only one default gateway can be configured on each Isilon

node, but each subnet can have its own gateway. When a node belongs to

more than one subnet, this option allows you to define the preferable

default gateway. A value of 1 is the highest priority, with 10 being

the lowest priority.

The Unix way of "route add/change default...." clearly cannot work at the same time.

Peter

Correct,  if you changed your dmz subnet gateway priority to 10 (or anything lower than the other) it would become the default subnet.

Isilon documentation prior to 7.0 doesn't explain this concept very well,  I'm told 7.0 is better but I haven't checked it out yet.

You can use the FreeBSD method of changing a default route,  but it is specific to a node and (I believe) not supported by EMC/Isilon.  It is also not persistent through a reboot or other gateway/smartconnect changes..

Mark,

can you please explain what do you mean by "By default each subnet will have a route for it's gateway and all traffic in that subnet will transverse that network... "

Sure.  What I mean is that if you have two SmartConnect Zones/Subnets like you described:

nyst0087-2# isi networks list subnets

Name                        Subnet       Gateway:Prio      SC Service Pools

--------------- ------------------ ------------------ --------------- -----

Internal-P          10.250.14.0/24    10.250.14.254:1   10.250.14.239     1

Internal-S          10.250.16.0/24    10.250.16.254:1   10.250.16.239     1

External          157.150.187.0/24  157.150.187.254:1 157.150.187.239     1

Management          10.250.25.0/24    10.250.25.254:1   10.250.25.239     2

All traffic with destination in 10.250.14.0/24 will go out the interface's which make up Internal-P, as seen in the route bolded in the example below:

#netstat -rn

Routing tables

Internet:

Destination        Gateway            Flags    Refs      Use  Netif Expire

default            10.250.14.254      UG          0 470738724  vlan1

10.250.14.0/24     link#9             UC          0        0  vlan1

10.250.14.72       00:50:56:9f:70:aa  UHLW        0    26916  vlan1   1185

10.250.14.73       00:50:56:9f:70:b0  UHLW        0     2691  vlan1   1162

but i don't understand why you need to specify a gateway, everything on that subnet will connect directly, it does not need to go to any routers. If it's not on that subnet it will hit your default gateway.

You don't at all,  my example was a bad one because I was in a hurry.

dynamox wrote: but i don't understand why you need to specify a gateway, everything on that subnet will connect directly, it does not need to go to any routers. If it's not on that subnet it will hit your default gateway.

Mark May wrote: Correct,  if you changed your dmz subnet gateway priority to 10 (or anything lower than the other) it would become the default subnet.

I've never had enough subnets to use any priority as high as 10, but I'm fairly certain it would likely be ignored with a priority that high.

I could be wrong.  However, I will typically define a priority of 1 for the gateway that I want all else to be sent via (except of course for anything with a static route or on the same subnet of any defined within OneFS).  Then all else will be 2 (or greater).

The only time this is problematic is if a node doesn't have an interface in the priority 1 gateway subnet and several in the priority2 gateways - you never know which one will end up as default.

Very good point.

My cluster has 5 subnets on them.  Most nodes only have two subnets connected.  One of the subnets is completely isolated, so it can only access traffic in the same subnet.  This caused me two problems.  On one node it was selected as the default gateway - so that node was dead in the water so to speak.  The other was users connecting across the isolated subnet were unable to access via CIFS due to lack of access to a domain controller on a node with just that subnet.