VMWare - ScaleIO v1.32 - using wrong vnic for data traffic

Question

Greetings all, Trying to figure out why ScaleIO (v1.32) appears to be using the wrong vnic in VMWare when connecting to my SDS volumes. Background: ---------------- We have a  minimal lab setup with 3 ScaleIO MDM/SDS servers (baremetal, CentOS 6.6, local drives) presenting storage to our VMWare server.  Each SDS server has a 1G NIC for management (172.16.1.21/24, 172.16.1.22/24, 172.16.1.23/24) and a 10G NIC for data (172.16.50.21/24, 172.16.50.22/24, 172.16.50.23/24). On the hypervisor side, we have a server running ESXi 5.5U2 (latest patches) with a single 1G NIC for management (vswitch0 - 172.16.1.10/24) and a 10G NIC for data (vswitch1 - 172.16.50.10/24).  This server has a single 100G LUN configured and attached using the 10G NIC IPs from the SDSs.  The ESXi server can read/write to the volume without issue. Given this is a single server install (and in the lab), we don't have  a vCenter server running (thus, we are not using the ScaleIO plugin for vCenter). Problem ------------ It seems the ScaleIO data traffic is going over the management NIC even though we specified the 10G network when configuring the kernel module.  This can be verified by the performance graph on the VMware client (Performance-->Advanced-->Network) when writing data to the LUN.  Here is the command we use to configure the scaleio module parameter on the hypervisor: esxcli system module parameters set -m scini -p ' IoctlIniGuidStr= IoctlMdmIPStr=172.16.50.21,172.16.50.22,172.16.50.23' Observations ----------------- If we query the SDCs on the MDM, we see the Hypervisor attached to the 10G NIC side as expected. If we query the module on the hypervisory ('esxcli system module parameters list -m scini'), the IoctlMdmIPStr variable has the correct MDM IPs. If we move the management interface to the 10G NICs, the performance goes up significantly. Is there a reason the ScaleIO data traffic seems to be tied to the management IP address?

keller51 · Accepted Answer

I *knew* as soon as I submitted the topic I would find the answer 30secs later!

The problem is tied to the way the IPs are configured on the ScaleIO servers. For all 3 SDS servers, I had both IP addresses (172.16.1.x/24 and 172.16.50.x/24) configured for SDC and SDS communication.

To fix:

--------

Open the "EMC ScaleIO" dashboard, navigate to backend, find the servers in question, right-click --> Configure IP Addresses. For the management IP Address, do NOT select "SDC" communication. For the data network IP Address, make sure to select both SDC and SDS. After making these changes, I verified all data communication was going over the 10G NICs (using "nmon" on the CentOS servers to look at network stats). When this proper NIC configuration is selected on the SDS nodes, you will see little-to-no traffic over the management interface when reading/writing data to the SDS volumes.

PowerFlex

Was this post helpful?