multipath is only using one (1) eth interface -- why?

Question

Hi,
platform:
Red Hat Enterprise Linux AS release 4 (Nahant Update 4)
2.6.9-42.ELsmp #1 SMP Wed Jul 12 23:32:02 EDT 2006 x86_64 x86_64 x86_64 GNU/Linux

I have multipath setup, having installed the s/w from the Dell cd (including mpp). iscsi-l indicates multiple paths to the storage, but I want the i/o to go over the multiple eth interfaces -- why doesn't it?

Also, multipath seems to ignore the settings in /etc/multipath.conf

Relevant info:

[root@kip md3000]# iscsi-ls -l ******************************************************************************* SFNet iSCSI Driver Version ...4:0.1.11-4(15-Jan-2007) ******************************************************************************* TARGET NAME : iqn.1984-05.com.dell:powervault.60019b9000d5ad0d00000000469feb82
TARGET ALIAS :
HOST ID : 0
BUS ID : 0
TARGET ID : 0
TARGET ADDRESS : 192.168.130.102:3260,2
SESSION STATUS : ESTABLISHED AT Thu Nov 8 14:24:12 PST 2007
SESSION ID : ISID 00023d000001 TSIH 800e
DEVICE DETAILS:
---------------
LUN ID : 0
Vendor: DELL
Model: MD3000i Rev: 9650
Type: Direct-Access ANSI SCSI revision: 05
LUN ID : 31
Vendor: DELL Model: Universal Model: Xport Rev: 9650
Type: Direct-Access ANSI SCSI revision: 05 ******************************************************************************* TARGET NAME : iqn.1984-05.com.dell:powervault.60019b9000d5ad0d00000000469feb82
TARGET ALIAS :
HOST ID : 3
BUS ID : 0
TARGET ID : 0
TARGET ADDRESS : 192.168.11.101:3260,1
SESSION STATUS : ESTABLISHED AT Thu Nov 8 14:24:12 PST 2007
SESSION ID : ISID 00023d000004 TSIH 36
DEVICE DETAILS:
---------------
LUN ID : 0
Vendor: DELL Model: MD3000i Rev: 9650
Type: Direct-Access ANSI SCSI revision: 05
LUN ID : 31 Vendor: DELL Model: Universal Model: Xport Rev: 9650
Type: Direct-Access ANSI SCSI revision: 05 ***************************************************************************** ::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
[root@kip md3000]# multipath -l -v3
load path identifiers cache
#
# all paths in cache :
# 360019b9000d5ad0d000011c2470d05b6 4:0:0:0 sda 8:0 1 [active] DELL /MD Virt
cciss!c0d0 blacklisted
cciss!c1d0 blacklisted
dm-0 blacklisted
dm-1 blacklisted
md0 blacklisted
ram0 blacklisted
ram10 blacklisted
ram11 blacklisted
ram12 blacklisted
ram13 blacklisted
ram14 blacklisted
ram15 blacklisted
ram1 blacklisted
ram2 blacklisted
ram3 blacklisted
ram4 blacklisted
ram5 blacklisted
ram6 blacklisted
ram7 blacklisted
ram8 blacklisted
ram9 blacklisted
===== path info sda (mask 0x1) =====
bus = 1
dev_t = 8:0
size = 1992560640
vendor = DELL
product = MD Virtual Disk
rev = 9650
h:b:t:l = 4:0:0:0
tgt_node_name =
uid = 360019b9000d5ad0d000011c2470d05b6 (cache)
#
# all paths :
# 360019b9000d5ad0d000011c2470d05b6 4:0:0:0 sda 8:0 1 [active] DELL /MD Virt
params = 0 0 1 1 round-robin 0 1 1 8:0 1000
status = 1 0 0 1 1 A 0 1 0 8:0 A 0
mpath0 (360019b9000d5ad0d000011c2470d05b6)
[features="0"][hwhandler="0"]
\_ round-robin 0 [active]
\_ 4:0:0:0 sda 8:0 [active]

::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
cat /etc/multipath.conf
defaults { user_friendly_names yes }
devices {
device {
vendor DELL*
product MD3000*
path_grouping_policy group_by_prio
getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
features "1 queue_if_no_path"
path_checker rdac prio_callout "/sbin/mpath_prio_tpc /dev/%n"
hardware_handler "1 rdac"
failback immediate
}
}
blacklist { device
{ vendor DELL product "Universal Xport" }
device { vendor DELL product Virtual.* } }

::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

[root@kip md3000]# ls -l /dev/sd*
brw-rw---- 1 root disk 8, 0 Nov 8 14:24 /dev/sda
brw-rw---- 1 root disk 8, 1 Nov 8 14:24 /dev/sda1

[root@kip md3000]# ls -l /dev/mapper
total 0
crw------- 1 root root 10, 63 Nov 8 14:23 control
brw-rw---- 1 root disk 253, 0 Nov 8 14:24
mpath0 brw-rw---- 1 root disk 253, 1 Nov 8 14:24 mpath0p1

Thanks!

REgards, Chas.

cjtompsett · Answer

There is a utility called SMdevices that is installed with all of the MDSM software. This will display the luns seen by the server as well as which raid controller they are on. This will only display the path that is not being masked. This software will also display the access luns (Lun 31). The access luns show up as sg devices and are not masked. You should see a Lun 31 for each path established to the MD3000i. So depending on the number of paths to the MD3000i over iSCSI you can see between 1-4 Lun 31s shown from the results of running the SMdevices utility.

Alternativly you can check to see how many iSCSI sessions have been established by going into the MDSM GUI, Click the iSCSI tab, and then click View/End iSCSI Sessions.

-cjtompsett

cdye · Answer

cj, thanks. I've confirmed that we have established iscsi sessions over both interfaces, but when we issue i/o, only one (1) of the nic's takes traffic. I'm not sure if this behavior is configuration, or is the multipath driver itself simply incapable of distributing the load? Here is a little more info:

[root@kip ~]# ls -l /dev/sg*
crw-rw---- 1 root disk 21, 0 Nov 8 14:24 /dev/sg0
crw-rw---- 1 root disk 21, 1 Nov 8 14:24 /dev/sg1
crw-rw---- 1 root disk 21, 2 Nov 8 14:24 /dev/sg2
[root@kip ~]# ls -l /dev/sd*
brw-rw---- 1 root disk 8, 0 Nov 8 14:24 /dev/sda
brw-rw---- 1 root disk 8, 1 Nov 12 14:06 /dev/sda1
[root@kip ~]# ls -l /dev/dm*
brw-r----- 1 root root 253, 0 Nov 12 14:02 /dev/dm-0
[root@kip ~]# ls -l /dev/mapper/*
crw------- 1 root root 10, 63 Nov 8 14:23 /dev/mapper/control
brw-rw---- 1 root disk 253, 0 Nov 12 14:02 /dev/mapper/mpath0
[root@kip ~]# multipath -ll
mpath0 (360019b9000d5ad0d000011c2470d05b6)
[features="0"][hwhandler="0"]
\_ round-robin 0 [prio=1][active]
\_ 4:0:0:0 sda 8:0 [active][ready]

Any insights most welcome. REgards, Chas.

dining_philosop · Answer

The MD3000i is designed as a asymmetric active-active storage controller. This means that for a given virtual disk all IOs are processed by a given controller (which is the owning controller for the virtual). If you have multiple virtual disks you can distribute them across the controllers so that both controllers are servicing IOs. Both Gbe ports on a given controller can accept IO for a virtual disk owned by the controller.

So:
1) If the sessions are established to separate controllers and you do IO to a virtual disk, you will only see activity on the session to the controller that has ownership of the virtual disk. If you disconnect the session, the virtual disk will fail over to the other controller and IO will be rerouted. Then you will see activity on the other session.
2) To load balance with controller redundancy you will need two sessions per controller for Linux (a total of 4 sessions to the MD3000i) (since the Linux initiator does not support multiple connections per session). Then on the sessions to the ports on the same controller you should see activity as long as there is enough traffic.

Message Edited by dining_philosophers on 11-27-2007 01:28 PM

cdye · Answer

dining_philosophers:

Do you have two controllers? Yes.

How many sessions do you have? eight (8) sessions, but all coming over eth2. We turned off eth2 and seen connections get established on eth3

Is the session from eth2 to one controller (say C0)?
Is the session from eth3 to the other controller (say C1)?

No -- eth2 has two (2) sessions to each port on each controller.

To use eth3 you will need to create a session from eth3 to C0?

Yes, but it seems that eth3 will only establish a connection for failover.

Just fyi, our multipath.conf looks like this:

devices {
device {
vendor DELL*
product MD3000*
path_grouping_policy multibus
getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
features "1 queue_if_no_path"
path_checker rdac
prio_callout "/sbin/mpath_prio_tpc /dev/%n"
hardware_handler "1 rdac"
failback immediate
}
}

dining_philosop · Answer

No -- eth2 has two (2) sessions to each port on each controller.
To use eth3 you will need to create a session from eth3 to C0?
Yes, but it seems that eth3 will only establish a connection for failover.

You need to add a session from eth3 to a controller so that the host can send IO down either eth2 or eth3 for a virtual disk that is owned by that controller. The best way to so this for say controller C0 is:
Session 1: eth2 to C0/Port 0
Session 2: eth3 to C0/Port 1

Message Edited by dining_philosophers on 11-27-2007 02:58 PM

dining_philosop · Answer

Do you have two controllers?
How many sessions do you have?
Is the session from eth2 to one controller (say C0)?
Is the session from eth3 to the other controller (say C1)?
To use eth3 you will need to create a session from eth3 to C0?

cdye · Answer

dining_philosophers,

thanks for the info -- good to know.

Even so, our immediate objective is to use both nic's on the client (eth2 and eth3 in my case) to perform i/o.

Right now, our throughput maxes out at about 128mb/s -- i.e. we are saturating the link (over eth2 in my case) -- absolutely no i/o goes over eth3.

What I want to do is use both eth2 and eth3 to perform i/o.

FWIW, we have:
- both eth's and all iSCSI ports on the same subnet
- a single gigE switch between the client and the md3000i
- a single virtual disk

Thanks.

REgards,
Chas.

cdye · Answer

dining_philosophers,

understood.

the conundrum is: why won't eth3 establish any connections at all unless eth2 is off?

if you know a way to explicitely tell eth3 to get with the program, we're all ears ;-)

short if multipath.conf, I don't know of anywhere else to tell multipath to use both eths...

thanks.

REgards,
Chas.

cdye · Answer

All,

Just fyi, we now have i/o balanced evenly across eth2 and eth3.

The trick:

eth2 is on subnet A
eth3 is on subnet B

controller 0 port 0 is on subnet A
controller 0 port 1 is on subnet B
controller 1 port 0 is on subnet A
controller 1 port 1 is on subnet B

02:43:46 PM IFACE rxpck/s txpck/s rxbyt/s txbyt/s rxcmp/s txcmp/s rxmcst/s
02:43:49 PM eth2 6951.00 4294.33 9654943.33 353963.33 0.00 0.00 0.00
02:43:49 PM eth3 6958.00 4301.67 9663673.33 354588.67 0.00 0.00 0.00

Previously we had tried (and failed) with:
- everything on the same subnet
- controller 0's ports on subnet A, controller 1's ports on subnet B

Thanks for your help.

REgards,
Chas.

dining_philosop · Answer

Just to be clear the establishment of sessions is not controlled the configuration file for the failover driver. It is controlled by the iSCSI initiator.

PowerVault

multipath is only using one (1) eth interface -- why?

Was this post helpful?