Highlighted
2 Bronze

How do I just use 10gig and not 1gig on my Isilon cluster?

Jump to solution

If you don't have any 1gig infrastructure near your Isilon cluster and only have 10gig, how do you setup just the 10gig network and put it into subnet0?

0 Kudos
1 Solution

Accepted Solutions
Highlighted
2 Bronze

Re: How do I just use 10gig and not 1gig on my Isilon cluster?

Jump to solution

By default subnet0 is setup for the 1gig interfaces on most Isilon clusters.  If you do not wish to have the 1gig network active at all, you can delete the pool and subnet where the 10gig interfaces are, then remove the 1gig interfaces from subnet0 and add in your 10gig SFP+ interfaces.  This needs to be done from the command line, and is best done from the serial console, so it doesn't involve the network.

First delete the pool and subnet containing your 10gig interfaces

    # isi networks delete pool --name=somesubnet:somepool

    # isi networks delete subnet --name=somesubnet

Then just delete the pool in your gigE subnet (do this from console).

    # isi networks delete pool --name=subnet0:pool0

Then, modify the subnet so its on the right network:

    # isi networks modify subnet --name=subnet0 --gateway=10.14.250.1

Now, create a new pool in that subnet:

   # isi networks create pool --name=subnet0:10gig --ranges=10.14.250.101-10.14.250.110 --ifaces=1-7:10gige-1

Verify that your commands executed successfully and accomplished what you desired:

   # isi networks list ifaces –v

  # isi_for_array –s “ifconfig cxgb1 | grep inet”

Now your 10 gig interfaces are in subnet0 and your 1gige interfaces are not being used.  If you desire to use them at a later time, you can create a ne subnet and pool and add them in, using similar commands.

View solution in original post

4 Replies
Highlighted
2 Bronze

Re: How do I just use 10gig and not 1gig on my Isilon cluster?

Jump to solution

By default subnet0 is setup for the 1gig interfaces on most Isilon clusters.  If you do not wish to have the 1gig network active at all, you can delete the pool and subnet where the 10gig interfaces are, then remove the 1gig interfaces from subnet0 and add in your 10gig SFP+ interfaces.  This needs to be done from the command line, and is best done from the serial console, so it doesn't involve the network.

First delete the pool and subnet containing your 10gig interfaces

    # isi networks delete pool --name=somesubnet:somepool

    # isi networks delete subnet --name=somesubnet

Then just delete the pool in your gigE subnet (do this from console).

    # isi networks delete pool --name=subnet0:pool0

Then, modify the subnet so its on the right network:

    # isi networks modify subnet --name=subnet0 --gateway=10.14.250.1

Now, create a new pool in that subnet:

   # isi networks create pool --name=subnet0:10gig --ranges=10.14.250.101-10.14.250.110 --ifaces=1-7:10gige-1

Verify that your commands executed successfully and accomplished what you desired:

   # isi networks list ifaces –v

  # isi_for_array –s “ifconfig cxgb1 | grep inet”

Now your 10 gig interfaces are in subnet0 and your 1gige interfaces are not being used.  If you desire to use them at a later time, you can create a ne subnet and pool and add them in, using similar commands.

View solution in original post

Highlighted
2 Bronze

Re: How do I just use 10gig and not 1gig on my Isilon cluster?

Jump to solution

Hello Davin -

Thanks for the sharing the information.  Certainly this is helpful; although I have slightly different problem.

We bought  THREE X200 Nodes with 1GbE NIC (total 4 x 1GbE ports). Later we ordered and 10GbE Isilon NIC for all three nodes. After installing and connecting them to the network; I was expecting the 10gig-1, 10gig-2 interfaces will show up on all the nodes.  unfortunately 10GbE NIC/Ports are not detected. 

Am I missing anything? Do i need to run any command to detect 10GbE Ports on the nodes or do i need any driver?

Note: 10GbE is NIC is official part.

Interface       Status      Membership                Addresses

--------------- ----------- ------------------------- ------------------------------

1:ext-1         up          subnet0:pool0             xxxx

1:ext-2         up

1:ext-3         up

1:ext-4         up

2:ext-1         up          subnet0:pool0             xxxx

2:ext-2         up

2:ext-3         up

2:ext-4         up

3:ext-1         up          subnet0:pool0             xxxx

3:ext-2         up

3:ext-3         up

3:ext-4         up

xxx-1#  isi_for_array -s "ifconfig cxgb1 | grep inet"

xxx-1 exited with status 1

xxx-2 exited with status 1

xxx-3 exited with status 1

xxx-1# ifconfig

xxx-1# ifconfig

cxgb0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000

        options=405bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,LRO,VLAN_HWTSO>

        ether 00:07:43:0b:48:a6

        media: Ethernet 10Gbase-SR <full-duplex>

        status: active

cxgb1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000

        options=405bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,LRO,VLAN_HWTSO>

        ether 00:07:43:0b:48:a7

        media: Ethernet 10Gbase-SR <full-duplex>

        status: active

em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500

        options=19b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4>

        ether 68:05:ca:28:6d:c8

        media: Ethernet autoselect (1000baseTX <full-duplex>)

        status: active

em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500

        options=19b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4>

        ether 68:05:ca:28:6d:c9

        media: Ethernet autoselect (1000baseTX <full-duplex>)

        status: active

em2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500

        options=19b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4>

        ether 00:25:90:ec:53:d0

        inet 10.xx.xx.xx netmask 0xfffffc00 broadcast 10.xx.xxx.255 zone 1

        inet 10.xx.xx.xy netmask 0xfffffc00 broadcast 10.xx.xxx.255 zone 1

        media: Ethernet autoselect (1000baseTX <full-duplex>)

        status: active

em3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500

        options=19b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4>

        ether 00:25:90:ec:53:d1

        media: Ethernet autoselect (1000baseTX <full-duplex>)

        status: active

lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384

        inet6 ::1 prefixlen 128 zone 1

        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x7 zone 1

        inet 127.0.0.1 netmask 0xff000000 zone 1

        inet 192.168.30.21 netmask 0xffffff00 zone 1

ib0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 2004

        lladdr 0.15.1b.0.10.bd.f6.ae

        inet 192.168.20.21 netmask 0xffffff00 broadcast 192.168.20.255 zone 1

        media: Infiniband autoselect

        status: active

ib1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 2004

        lladdr 0.15.1b.0.10.bd.f6.af

        inet 192.168.10.21 netmask 0xffffff00 broadcast 192.168.10.255 zone 1

        media: Infiniband autoselect

        status: active

aetitan-1#

aetitan-1#

Thanks,

~Bish

bishwajit.kumar@emc.com

0 Kudos
Highlighted
2 Bronze

Re: How do I just use 10gig and not 1gig on my Isilon cluster?

Jump to solution

Bish,

You cannot just replace the 1Gig cards with 10gig cards and have them show up.  You also must update the node so that it knows it's new configuration. this is done by a custom script script, written by engineering for each specific node.  This needs to be done for any hardware changes that are done to CTO nodes.  Contact your local Sales Ops support person, give them the serial numbers of the nodes and tell them that you need scripts generated for each node to update them to reflect the new configuration. 

Devin

Highlighted
Not applicable

Re: How do I just use 10gig and not 1gig on my Isilon cluster?

Jump to solution

This is obviously not recommended to do on your own and I do not take responsibility for you breaking your cluster BUT...

here is how you do it.

isi_inventory_tool --display --itemType=history

isi_inventory_tool --display --itemType=receipt

     get the receipt file name from that output.

isi_inventory_tool --display --itemType=receipt > /ifs/data/Isilon_Support/tmp/[filename].xml

     edit the .xml

     For example your receipt may show:

     <sbb part="613-0001-01" desc="SBB, 1GbE Base-T Dual-port PCIe" qty="1"/>

     Replace that line with

     <sbb part="613-0002-01" desc="SBB, 10GbE, Dual-port, SFP+, without Optics" qty="1"/>

     Save the xml as a new file and import.

isi_inventory_tool --importFile --fileType=receipt --filePath=/ifs/data/Isilon_Support/tmp/[newfilename].xml

isi_inventory_tool --reInit

Reboot node.

isi_inventory_tool --configCheck

As a note this will only work if the output of your

"isi_inventory_tool --display --itemType=softavl"  contains that part number, in this case "613-0002-01".

Profit.

0 Kudos