MD3200 rdac help

The MD3200 and the MD3000 are both OEM'ed LSI Engenio storage and both use "RDAC", where RDAC is the protocol to handle fail over between the paths.

Technically Dell are not supporting the RDAC or more correctly known as the RDAC Multi-Path Proxy  driver with the MD3200 and suggest you use the dm-multipath driver instead. Personally I have never been convinved about the reliability of dm-multipath under stress or when stuff goes wrong. We have seen many instances of where dm-multipath connected machines have thrown complete wobblers where RDAC mpp based ones don't when both are connected to the same storage array at the same time. It is also far more complex to setup, and the two downsides of the RDAC mpp driver, installing a new version for a new kernel and removing unused disks from a running system are easy enough to overcome. I can post the information if anyone is interested.

If you still wish to keep with the RDAC driver, then you are going to need a more up to date version that will support the MD3200. As Dell are not providing a suitable driver I would suggest going to

    http://www.lsi.com/rdac/

You want the 09.03.0C05.0331 version which is the latest. Does not matter which link you get it from as it is the same thing. I can confirm that version works with an MD32xxi using a Qlogic iSCSI HBA.

That's funny that you mention that Dell doesn't want to use the rdac because the white papers that they gave me all suggest to use it.

I can't seem to find the dm-multipath driver on their install CD so I'm not sure what is going on.

I've downloaded the driver you mentioned from LSI and it appears to be working since fdisk -l isn't showing both ports as disks anymore. What I don't see anymore is in multipath -ll all 4 ports. We have two Dell HBAs with 2 ports connected on each one. Only one HBA is now showing.

mpath1 (< >) dm-4 DELL,VirtualDisk
[features=0][hwhandler=0][rw]
\_ round-robin 0 [prio=1][active]
 \_ 3:0:0:0 sdb 8:16  [active][ready]

You are confusing the term RDAC. Specifically RDAC is a protocol used over the scsi bus (whether provided by SCSI, FC, iSCSI or SAS) to handle device failover. To make use of this on Linux there are two approaches. First a driver provided by LSI known as the RDAC MPP driver or sometimes the RDAC driver, this only supports LSI Engenio based arrays of which the MD3000 and MD3200 are amoung, along with the IBM DS3000/4000/5000 line and various other vendors. The second is the Linux Device Mapper Multipath driver that through the use of various helper agents supports a wide range of storage devices including RDAC protocol based ones.

With the MD3200 the offically supported driver has changed from the RDAC MPP driver as used with the MD3000 to dm-multipath. You will see no mention of the the RDAC MPP driver in the offical support matrix.

http://support.dell.com/support/edocs/systems/md3200/en/matrix/matrix.pdf

If you are using CentOS 5.5 then the necessary dm-multipath drivers are including in the OS.

More confusion is occuring because firstly the dm-multipath driver does not hide the /dev/sd* devices, but generates new ones called /dev/mapper/mpath* leaving the /dev/sd* ones intact. Just don't try and use them or you will get all sorts of errors.

The RDAC MPP driver does hide these devices and creates the /dev/sd* devices for the virtual disks. If you install lsscsi you can see this by running the lsscsi command. In addition the RDAC MPP driver does not know or understand anything to do with the dm-multipath utilities, and dm-multipath should be turned off. You can check on the status of paths and things with mppUtil, mppUtil -S is a useful one and /opt/mpp/lsvdev is also useful for listsing the virtual disks created by the mpp driver. You can also use hot_add to scan in newly mapped luns to the OS.

One of the weaknesses of the mpp driver is that no methdd is provided to remove disks without rebooting, though this can be overcome by echoing stuff into the right places in /sys. I have a perl script that does it all for you given a device name, but only tested on RHEL5 and will not work on RHEL4.

I really appreciate all of the information you have provided me. It is a lot better then what I have been getting directly from Dell support. I do see all 4 ports using the mppUtil -s

H3C0T0       Active                     Active           hdavrtarrayt01                                                       
             H1C0T0L000  Up             H2C0T0L000  Up                                                                        
             H1C0T1L000  Up             H2C0T1L000  Up  

But fdisk -l is now only showing the one disk, before it would show both 4.5TB (even though we had one disk). Also the 20MB files are missing.

Our issue is that if we get the driver from LSI and we run into issues, Dell usually looks for a way out of having to look at the issue by saying it is not supported so we really don't want to go outside of what they provide on the CD. If something is screwed up on the CD, Dell needs to fix it.

At this point for our customer we are going to send it back and get a MD3000. I am going to move the HBAs to another machine to see if we can get this sorted by Dell support but for now will have to use the older hardware.

Installation and configuration of the Device Mapper Mulitpath for Linux can be found on page 167 of this document.
http://support.dell.com/support/edocs/systems/md3200/en/OM/PDF/MD3200.pdf

This following document describes what has changed in the Linux multipathing configuration between the MD3000 and MD3200.
Linux Multipathing Solutions: MPP vs. DM-RDAC
http://www.dell.com/downloads/global/products/pvaul/en/powervault-md32x0-md32x0i-linux-multipathing-solutions-en.pdf

-cjtompsett

Note that whether to use MPP or MD-RDAC on either a MD3000 or MD3200 is purely a support decision made by Dell. Both products support both solutions at a technical level.

Thanks for the replies. So going back to my original issue. We had everything setup correctly in CentOS 5.4 and yet when we put the storage array under any kind of load it took the mount into read-only. So there is either a compatibility issue with the drivers or something else in the documentation is missing. I'm not sure what it may be but since we are over our deadline for our customer and don't have the freedom to mess around with this, we are sending it back to Dell and will order a MD3000.

Maybe with the next customer we'll plan enough time to get the kinks worked out of it.

Interesting. We are seeing really poor read performance and spontaneous controller reboots under "heavy" read load on our MD3200i.  I quote the use of "heavy" there as in fact a dd of a large file is only able to sustain ~30MB/s read performance, where as with write we get more like 170MB/s.

Might be worth checking the event log and checking to see if you can see any controller restarts (look for something like start of day routine completed).

We didn't get as bad performance as what you have stated, might be because we are using Dell's HBA's and you are using the Qloqic ones.

I'm sorry I can't check the logs for you at the moment...I removed the HBA's from the server. I'll try to install the storage software on another box and connect to it since I have the ethernet management ports enabled.

The Qlogic has nothing to do with it, we get the same asymmetric read/write performance using Broadcom 5709 nics with both software initiator and hardware accelerated version.

Has anyone been able to implement this successfully without any issues? We have followed the whitepaper installing the MD3200 and even tried it with one card. Within 10 minutes of us running Oracle's Orion test which simulates an OLTP database and simultaneously running dd the filesystem went into read only. Support had been anything but helpful and the TAC has done little in helping us. I am going on a last ditch effort to get this resolved tomorrow before we end up looking at HP possibly for our storage needs.

It just seems to us that Dell support has very little experience in troubleshooting advanced issues with the MD3200.

Npolite... what whitepaper are you referring to? Please can you provide a link?

I have a couple of notes after reading the thread:

a) MPP (RDAC) and DM (Linux default device mapper) are implemented in different fashions. The guides linked above discuss DM implementation, how to use it, why you would see multiple /dev/sd* devices, how to use /dev/mapper/mpath/ devices instead etc. It is important to note that Dell has optimized everything for working with DM and not MPP (especially some timings around failover). So, MPP will not give you the best failover performance on linux

b) MPP and DM should not be used at the same time. I recommend using DM and using only the /dev/mapper/mpath devices and not the /dev/sd* devices

c) I am not sure why the disks are becoming read only. The only explanation that I can think of is that you are using multiple paths to a virtual disk on the /dev/sd* device listing causing overlap.

We have been actively working this issue with Dell support for over 6 months and have not made any progress. We can pretty much drop the filesystem to read-only on the fly now that we have done it so many times. They have even asked us to use smaller disk groups and run consistency checks before and after on the disk group. Has anyone been able to get a stable environment running Redhat?

I'm almost willing to try Windows 2008 and see if we hit the same problem.

We also tried Redhat 6. It look a little longer to drop the filesystem but we did.

I'm just thankful that we are not in any need for storage arrays at the moment because we don't have a solution from Dell and would have to consider HP as an alternative until we can get a solution to this issue.