PowerScale: Understanding Network Interface cards with DE-based nodes

Gen6 nodes have two major categories:

• DE-Based Nodes:
  • A200
  • A2000
  • H400
• EP-Based Nodes:
  • H500
  • H600
  • H5600
  • F800
  • F810

DE-based nodes have an onboard Intel 10G card used for frontend connectivity. The network chipset is embedded in the CPU. This embedded network chipset is connected to a pelican adapter which provides the physical connection to the network. DE-based nodes cannot have the onboard network card removed. Instead, it is the pelican adapter (physical connecting board) that is removed for a new PCIe card. After the pelican adapter is replaced on a DE-based node, the onboard card is still seen. Also, the ports for the new PCIe card are seen, totaling seven network interfaces.
 

• 1x management port ([bge] 1 GbE management interface)

• 2x frontend ports (the new network card)

• 2x backend ports (labeled as ISIINTERNAL)

• 2x "ix" ports that are part of the embedded network chipset.

This is a side effect of DE-based nodes and is expected behavior. The command "ifconfig" displays the following interfaces (the output varies depending on the hardware installed).
 

ix0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
       options=e407bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
       ether 00:60:48:4e:24:d0
       inet netmask 0xffffff00 broadcast  zone 1
       nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
       media: Ethernet autoselect
       status: no carrier
ix1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
       options=e407bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
       ether 00:60:48:4e:24:d1
       nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
       media: Ethernet autoselect
       status: no carrier
bge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
       options=8009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LINKSTATE>
       ether 00:60:16:88:e3:a6
       inet 10.10.10.10 netmask 0xffffff00 broadcast 10.10.10.255 zone 1
       nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
       media: Ethernet autoselect (1000baseT <full-duplex,master>)
       status: active
lo0: flags=1008049<UP,LOOPBACK,RUNNING,MULTICAST,ISIINTERNAL> metric 0 mtu 16384
       options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
       inet6 ::1 prefixlen 128 zone 1
       inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4 zone 1
       inet 127.0.0.1 netmask 0xff000000 zone 1
       inet 128.221.254.1 netmask 0xffffff00 zone 1
       inet 127.42.0.1 netmask 0xffff0000 zone 1
       nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
mlxen0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
       options=d07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,LINKSTATE>
       ether ec:0d:9a:26:bb:40
       nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
       media: Ethernet autoselect (10Gbase-CX4 <full-duplex,rxpause,txpause>)
       status: active
mlxen1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
       options=d07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,LINKSTATE>
       ether ec:0d:9a:26:bb:41
       nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
       media: Ethernet autoselect
       status: no carrier
ib0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,ISIINTERNAL> metric 0 mtu 2044
       options=80018<VLAN_MTU,VLAN_HWTAGGING,LINKSTATE>
       lladdr 0.0.0.48.fe.80.0.0.0.0.0.0.ec.d.9a.3.0.b1.73.31
       inet 128.219.252.1 netmask 0xffffff00 broadcast 128.221.252.255 zone 1
       nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
       media: Infiniband autoselect
       status: active
ib1: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,ISIINTERNAL> metric 0 mtu 2044
       options=80018<VLAN_MTU,VLAN_HWTAGGING,LINKSTATE>
       lladdr 0.0.0.49.fe.80.0.0.0.0.0.0.ec.d.9a.3.0.b1.73.32
       inet 128.220.253.1 netmask 0xffffff00 broadcast 128.221.253.255 zone 1
       nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
       media: Infiniband autoselect
       status: active

The above output indicates that there are ix interfaces, mlxen interfaces, and InfiniBand interfaces. There are many different variations of network configurations to which some may include more ix interfaces and mlxen interfaces. Be aware of the different variations of hardware configuration when considering these issues. Whereas EP-based nodes CAN have their network card replaced with the same hardware.