Dell Unity: How to change the LACP trunk or bond load-balancing algorithm (Dell Correctable)
Summary: Link Aggregation Control Protocol (LACP) traffic is balanced for writes to the Unity SPs over an LACP trunk or bond, but is not evenly balanced when responses are sent to read requests. ...
Symptoms
Under certain networking conditions and environments, the default algorithm may default to using a single interface.
Examples:
When the source MAC address is the same (aka when using a router), the MAC will always be the same and the port that is used for transmissions will always be the same.
Also, under special circumstances, different MACs may also result in the same value.
For example, if MACs always end in an even number (0,2,4,6,8,A,C, or E) and there are two ports in the LACP trunk or bond, calculation of the hash will direct traffic through the same port every time as well.
LACP trunks or bonds are showing traffic is not balanced, using a single interface instead of all interfaces evenly.
This can be confirmed on the production network (by the Network Switch SysAdmins), or by looking at the graphical network display in Unisphere under SYSTEM > Performance.
netstat -i' output in the service shell.
Cause
LACP on the Unity uses layer2 as its default "xmit_hash_policy."
Using Layer2+3 as the "xmit_hash_policy" is intended to provide a more balanced distribution of traffic than layer2 alone, especially in environments where a layer3 gateway device is required to reach most destinations.
Reference: https://www.kernel.org/doc/Documentation/networking/bonding.txt
Layer2 uses XOR of hardware MAC addresses and packet type ID field to generate the hash.
The formula is
hash = source MAC XOR destination MAC XOR packet type ID slave number = hash modulo slave count.
Layer2+3 uses a combination of layer2 and layer3 protocol information to generate the hash.
The hash is generated using a combination of the XOR of the hardware MAC addresses and the IP addresses.
The formula is
hash = source MAC XOR destination MAC XOR packet type ID hash = hash XOR source IP XOR destination IP hash = hash XOR (hash RSHIFT 16) hash = hash XOR (hash RSHIFT 8) And then hash is reduced modulo slave count.
Both Layer2 and Layer2+3 are 802.3ad compliant.
Resolution
For Unity OE Code 4.3 and above:
Change the xmit_hash_policy with the svc_network_bond command.
Dell Unity™ Family Version 4.3: Service Commands Technical Notes - page 74.
Usage:
svc_network_bond [-h|--help] -d <device> {-s -o <option> -v <value>} {-g [-o <option>]}
The syntax would be similar to the below example:
service@(none) spb:~> svc_network_bond -s -d bond23 -o xmit_hash_policy -v 2
The acceptable values for xmit_hash_policy are:
0 or layer2 Default setting
This parameter uses the XOR of hardware MAC addresses to generate the hash.
1 or layer3+4 Uses upper layer protocol information (when available) to generate the hash.
This allows for traffic to a particular network peer to span multiple slaves, although a single connection will not span multiple slaves.
2 or layer2+3 Uses a combination of layer2 and layer3 protocol information to generate the hash - Mode 2 or Layer2+3 algorithm is 802.3ad compliant.
For Unity OE Code 4.2.3.9670635 and older:
Contact Dell Customer Service, and reference this KBA number.
For Unity OE Code 5.3.x or higher
Making the change does not require the service shell and no reboot is required.
Here is an example setting the Unity Array in our lab's xmit_hash_policy.
This Unity Array is configured with an LACP trunk known as bond22.
SSH in to the Unity array using the service account.
First, check what its xmit_hash_policy is set to.
# svc_network_bond --get --device bond22 -o xmit_hash_policy INFO: Selected device: bond22 INFO: Option to show: xmit_hash_policy INFO: Execution code: 0 xmit_hash_policy=0 #
Next, set the xmit_hash_policy to 2
# svc_network_bond --set --device bond22 -o xmit_hash_policy -v 2 INFO: Selected device: bond22 INFO: Option to modify: xmit_hash_policy INFO: Requested value: 2 WARNING: Do you want to proceed? [yes/no]: yes <<<<<< sometimes y works and sometimes it fails on the first attempts. Retry then. INFO: Execution code: 0 INFO: Option 'xmit_hash_policy' has been successfully changed. #
# svc_network_bond --get --device bond22 -o xmit_hash_policy INFO: Selected device: bond22 INFO: Option to show: xmit_hash_policy INFO: Execution code: 0 xmit_hash_policy=2 #
Additional Information
netstat and arp require the service shell to be enabled.
Example 'netstat -i' output, which shows the members of a bond and can be used to determine if traffic is being hashed.
In this example, 'bond20' (the LACP bond) is made up of interfaces 'eth20' and 'eth21'.
Note the difference in the TX-OK column, which represents outbound traffic from the Unity.
21:12:03 service@(none) spb:~> netstat -i Kernel Interface table Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg bond20 9000 0 101724658 0 11 0 126087418 0 0 0 BMmRU cmin0 9000 0 14341258 0 0 0 11301712 0 0 0 BMRU eth2 1500 0 0 0 0 0 0 0 0 0 BMU eth3 1500 0 0 0 0 0 0 0 0 0 BMU eth10 1500 0 0 0 0 0 0 0 0 0 BMU eth11 1500 0 0 0 0 0 0 0 0 0 BMU eth12 1500 0 0 0 0 0 0 0 0 0 BMU eth13 1500 0 0 0 0 0 0 0 0 0 BMU eth20 9000 0 52249885 0 1 0 38317 0 0 0 BMsRU eth21 9000 0 49474773 0 10 0 126049101 0 0 0 BMsRU eth22 1500 0 0 0 0 0 0 0 0 0 BMU eth23 1500 0 0 0 0 0 0 0 0 0 BMU eth_int 9000 0 14341055 0 0 0 11301598 0 0 0 BMRU eve_br0 1500 0 16 0 0 0 3656 0 0 0 BMRU lo 65536 0 963282566 0 0 0 963282566 0 0 0 LRU mgmt 1500 0 1405994 0 64 0 360538 0 0 0 BMRU mgmt_vdev 1500 0 356150 0 64 0 326216 0 0 0 BMRU srm 1500 0 135650 0 64 0 5 0 0 0 BMRU vetheve1 1500 0 16 0 0 0 3647 0 0 0 BMRU
Example output of 'arp' on service shell which shows the MAC addresses (HWaddress) for each IP address.
These should be different for LACP to accurately load balance traffic across multiple physical ports.
There are several interfaces that must filter by the interface you are using.
Use
'ip addr' to find the "Iface" that is assigned to the IP address you want to investigate.
19:23:33 service@(none) spa:~> arp Address HWtype HWaddress Flags Mask Iface 10.98.25.61 ether 00:25:b5:02:01:fc C bond20 10.98.25.60 ether 00:25:b5:02:00:1c C bond20 10.98.25.70 ether 00:25:b5:02:00:dc C bond20 10.98.25.63 ether 00:25:b5:02:01:8c C bond20 10.98.25.65 ether 00:25:b5:02:01:6c C bond20 10.98.25.62 ether 00:25:b5:02:01:dc C bond20 10.98.25.64 ether 00:25:b5:02:01:9c C bond20 10.98.25.67 ether 00:25:b5:02:01:0c C bond20 10.98.25.66 ether 00:25:b5:02:01:7c C bond20 10.98.25.59 ether 00:25:b5:02:00:0c C bond20 peer ether 8e:92:80:4d:2d:02 C eth_int 10.98.25.69 ether 00:25:b5:02:00:fc C bond20 10.98.25.1 ether 00:08:e3:ff:fd:90 C bond20 10.98.25.68 ether 00:25:b5:02:01:1c C bond20