We are trying to install a MD3200 and are having a few issues with setting up the failover ability (between 2 cards and 4 ports). I've installed a few MD3000s and always used the multipath driver. After reboot everything would show up correctly and I could partition the virtual disk without any issues.
From what I can tell the MD3200 now uses rdac to handle the failover ability. I followed the steps and partitioned the /dev/mapper device but we are still getting corrupted blocks. Fdisk -l is still showing multiple devices /dev/sdd and /dev/sdf with 4TB when there is only 4TB total.
We are running CentOS 5.5 and renamed the redhat-release before installing. Can we not use the multipath driver? Do we still have an issue with rdac because it is showing twice in fdisk? I also see the 20MB sizes which I believe are the raid controller cards? This usally went away after installing the Dell Storage manager on the MD3000s. I've called Dell on this and haven't made much headway with it so I thought I would post up here.
The MD3200 and the MD3000 are both OEM'ed LSI Engenio storage and both use "RDAC", where RDAC is the protocol to handle fail over between the paths.
Technically Dell are not supporting the RDAC or more correctly known as the RDAC Multi-Path Proxy driver with the MD3200 and suggest you use the dm-multipath driver instead. Personally I have never been convinved about the reliability of dm-multipath under stress or when stuff goes wrong. We have seen many instances of where dm-multipath connected machines have thrown complete wobblers where RDAC mpp based ones don't when both are connected to the same storage array at the same time. It is also far more complex to setup, and the two downsides of the RDAC mpp driver, installing a new version for a new kernel and removing unused disks from a running system are easy enough to overcome. I can post the information if anyone is interested.
If you still wish to keep with the RDAC driver, then you are going to need a more up to date version that will support the MD3200. As Dell are not providing a suitable driver I would suggest going to
You want the 09.03.0C05.0331 version which is the latest. Does not matter which link you get it from as it is the same thing. I can confirm that version works with an MD32xxi using a Qlogic iSCSI HBA.
That's funny that you mention that Dell doesn't want to use the rdac because the white papers that they gave me all suggest to use it.
I can't seem to find the dm-multipath driver on their install CD so I'm not sure what is going on.
I've downloaded the driver you mentioned from LSI and it appears to be working since fdisk -l isn't showing both ports as disks anymore. What I don't see anymore is in multipath -ll all 4 ports. We have two Dell HBAs with 2 ports connected on each one. Only one HBA is now showing.
mpath1 (<<removed_by_me>>) dm-4 DELL,VirtualDisk
\_ round-robin 0 [prio=1][active]
\_ 3:0:0:0 sdb 8:16 [active][ready]
You are confusing the term RDAC. Specifically RDAC is a protocol used over the scsi bus (whether provided by SCSI, FC, iSCSI or SAS) to handle device failover. To make use of this on Linux there are two approaches. First a driver provided by LSI known as the RDAC MPP driver or sometimes the RDAC driver, this only supports LSI Engenio based arrays of which the MD3000 and MD3200 are amoung, along with the IBM DS3000/4000/5000 line and various other vendors. The second is the Linux Device Mapper Multipath driver that through the use of various helper agents supports a wide range of storage devices including RDAC protocol based ones.
With the MD3200 the offically supported driver has changed from the RDAC MPP driver as used with the MD3000 to dm-multipath. You will see no mention of the the RDAC MPP driver in the offical support matrix.
If you are using CentOS 5.5 then the necessary dm-multipath drivers are including in the OS.
More confusion is occuring because firstly the dm-multipath driver does not hide the /dev/sd* devices, but generates new ones called /dev/mapper/mpath* leaving the /dev/sd* ones intact. Just don't try and use them or you will get all sorts of errors.
The RDAC MPP driver does hide these devices and creates the /dev/sd* devices for the virtual disks. If you install lsscsi you can see this by running the lsscsi command. In addition the RDAC MPP driver does not know or understand anything to do with the dm-multipath utilities, and dm-multipath should be turned off. You can check on the status of paths and things with mppUtil, mppUtil -S is a useful one and /opt/mpp/lsvdev is also useful for listsing the virtual disks created by the mpp driver. You can also use hot_add to scan in newly mapped luns to the OS.
One of the weaknesses of the mpp driver is that no methdd is provided to remove disks without rebooting, though this can be overcome by echoing stuff into the right places in /sys. I have a perl script that does it all for you given a device name, but only tested on RHEL5 and will not work on RHEL4.
I really appreciate all of the information you have provided me. It is a lot better then what I have been getting directly from Dell support. I do see all 4 ports using the mppUtil -s
H3C0T0 Active Active hdavrtarrayt01
H1C0T0L000 Up H2C0T0L000 Up
H1C0T1L000 Up H2C0T1L000 Up
But fdisk -l is now only showing the one disk, before it would show both 4.5TB (even though we had one disk). Also the 20MB files are missing.
Our issue is that if we get the driver from LSI and we run into issues, Dell usually looks for a way out of having to look at the issue by saying it is not supported so we really don't want to go outside of what they provide on the CD. If something is screwed up on the CD, Dell needs to fix it.
At this point for our customer we are going to send it back and get a MD3000. I am going to move the HBAs to another machine to see if we can get this sorted by Dell support but for now will have to use the older hardware.
Installation and configuration of the Device Mapper Mulitpath for Linux can be found on page 167 of this document.
This following document describes what has changed in the Linux multipathing configuration between the MD3000 and MD3200.
Linux Multipathing Solutions: MPP vs. DM-RDAC
As I said previosuly that is the correct behaviour for the RDAC mpp driver from LSI. You only get to see the one device. If you install lsscsi (yum install -y lsscsi) and run the lsscsi command you will see all the four paths that way as well.
As I said previously as well if you are running CentOS 5.5 you don't need anything from the CD. It is made quite clear in the documentation that all the necessary dm-multipath drivers are included in the operating system. That said CentOS 5.5 is not suported either so why worry...
Note that whether to use MPP or MD-RDAC on either a MD3000 or MD3200 is purely a support decision made by Dell. Both products support both solutions at a technical level.
Thanks for the replies. So going back to my original issue. We had everything setup correctly in CentOS 5.4 and yet when we put the storage array under any kind of load it took the mount into read-only. So there is either a compatibility issue with the drivers or something else in the documentation is missing. I'm not sure what it may be but since we are over our deadline for our customer and don't have the freedom to mess around with this, we are sending it back to Dell and will order a MD3000.
Maybe with the next customer we'll plan enough time to get the kinks worked out of it.
Interesting. We are seeing really poor read performance and spontaneous controller reboots under "heavy" read load on our MD3200i. I quote the use of "heavy" there as in fact a dd of a large file is only able to sustain ~30MB/s read performance, where as with write we get more like 170MB/s.
Might be worth checking the event log and checking to see if you can see any controller restarts (look for something like start of day routine completed).