Celerra 3 Port Trunk Group being Saturated

Question

I have a Celerra NS122 configured with a 3 port Trunk Group (trk0 = cge0, cge1, cge2) using LACP. This trunk group handles all of my iSCSI and CIFS traffic. I have cge3 as a standalone port handling replication traffic to our DR Celerra.

I currently have 6 ESXi Hosts writing to an FC LUN and an ATA LUN on the Celerra without issue. I recently began copying data over to a new ATA LUN on the Celerra that will host CIFS shares. I am using the EMCOPY utililty to copy data from a physical server over to the CIFS LUN. When I run the utility and begin copying data, I start receiving errors concerning the ATA LUN in use by VMware:

"Lost access to volume xxxxx due to connectivity issues. Recovery attempt is in progress......"

If I stop the EMCOPY process, the VM datastore immediately reconnects to the ATA LUN and all is fine. If I re-start the EMCOPY process, the errors start again. I had our network guy check the links and he said it appears one of those trunk group ports are being saturated and the others are not.

Do I need to add an additional network portal to the iSCSI target??---I only have one assigned currently. I am assuming it has to be my port configuration causing the issue. Anyone have any ideas as to what I can check? I ran the command below all connections seem OK.

server_sysconfig server_2 -virtual -info trk0
server_2 :
*** Trunk trk0: Link is Up ***
*** Trunk trk0: Timeout is Short ***
*** Trunk trk0: Statistical Load Balancing is IP ***
Device     Local Grp   Remote Grp Link LACP Duplex Speed
------------------------------------------------------------------------
cge0       10000       2560       Up    Up   Full     1000 Mbs
cge1       10000       2560       Up    Up   Full     1000 Mbs
cge2       10000       2560       Up    Up   Full     1000 Mbs

Thanks.

Rainer_EMC · Answer

Take a look at the link utilization on both the Celerra and the switch side

make sure that the IP addresses you use for both clients and servers are spaced appropriately so that they don’t end up all on the same link

dynamox · Answer

crank emcopy down by using /th:1 ..default is 64. It will copy slower but won't overwhelm your connection. Worth a shot

DanJost · Answer

Ask your network admin what the LACP/Etherchannel load balancing mechanism is.  If it is set to route based on destination MAC, you'll only use a single link in the LACP bundle not matter what.   Depending on the switch, you will likely have better options like source MAC, IP, tcp port, etc.  The more metrics the better but low-end fixed configuration switches are pretty limited in what they can do.

ddavisguard · Answer

Rainer.....

All of the VMware servers are located on a dedicated VLAN (we call storage) and the iSCSI port on the Celerra is located in that same VLAN as well.......guest VM's and users clients all have their separate VLAN's too.

ddavisguard · Answer

Dnyamox,

Thanks for the tip about decreasing the threads used by EMCOPY.......that should help me get the data over to the new LUN with disrupting our VMware environment.

Do you know if I should add another IP address (network portal as they call it there) to the iSCSI target in my Celerra configuration? Would that help to load balance the connections across the other links in the trunk? Coming from our old CX320C which had 8 ip addresses for iSCSI -- using only one on the Celerra seems to be asking for trouble!

I am only running VMware on the Celerrra now and I don't want to run into any trouble once I enable two CIFS servers handling approx. 100 shares and home directories for 800 users.

ddavisguard · Answer

Dan....thanks for the reply. We are using Source XOR Destination MAC address for load balancing on our Cisco 6509. The switch does have the following options available to us:

dst-ip        Dst IP Addr
dst-mac       Dst Mac Addr
dst-port      Dst TCP/UDP Port
mpls          Load Balancing for MPLS packets
src-dst-ip    Src XOR Dst IP Addr
src-dst-mac   Src XOR Dst Mac Addr
src-dst-port Src XOR Dst TCP/UDP Port
src-ip        Src IP Addr
src-mac       Src Mac Addr
src-port      Src TCP/UDP Port

DanJost · Answer

Take a look at this document: http://www.cisco.com/en/US/tech/tk389/tk213/technologies_tech_note09186a0080094714.shtml

In a 3-port LACP/etherchannel, you get a 3:3:2 distribution so one of the ports (even under perfectly even distribution) will get 2/3 of the traffic that the other two receive. You might need a specific card to do TCP/UDP-based load balancing (mentioned in link above). Port-based load balancing is probably the best as almost all protocols use a fixed port on one side and a random port on the other. src-dst-ip is the most common that I've seen though it isn't perfect either if the VMware servers "hash" to the same port. There is a test command you can run to see which path it'll take. For your CIFS clients (if you have a sufficient number) IP-based load balancing works great. With two or three ESX servers running iSCSI you might still have them all end up on the same path. The Celerra will "reflect" or reply on the same path (I think this is the default perhaps someone else can chime in). You'll want your network guru to check this out. He might have a reason to keep the src-dst-mac but he may have never considered/thought about changing it.

It is common to overlook this - and the Cisco defaults (in a routed VLAN environment) often peg traffic to the same member of an etherchannel/LACP. Just because you have an etherchannel/LACP set up does not mean you get x-times-number of links in bandwidth.

Dan

Rainer_EMC · Answer

IMHO you first need to analyze how well the links are balanced for each direction before making a good decision on how to improve

multiple IPs might help but some OS's never use more than one session pr LUN or per target

Celerra

Celerra 3 Port Trunk Group being Saturated

Was this post helpful?