Poor performance - PV220S, PE2650, RAID 10

What is the BIOS on your 2650?  What PCI slot is your PERC in.

The A15 BIOS supposedly had some fixes (unpublished of course) for this, problems with that motherboard and certain slots... and write performance.  I am planning to do some testing in my lab to see if this is affecting us as well.  If you can, try moving your PERC to slot 1, and slot 3, and repeat your test, and see what happens.

Looks like we're running A10 on the 2650. As to the PCI slot, I don't know - the server in question is in a co-location facility, and it's a production server involved in a 24/7 operation, so at this point I'd really like to avoid any significant downtime.

I'll try installing the A15 BIOS over the weekend (life? what life?) and see if that makes any difference. Thanks for the pointer.

Just finished with the A15 BIOS update, and no significant change... the write speed has improved marginally, but that may simply be due to the fact that it's past midnight local time and there's less load on the system than when I performed the benchmark earlier (during business hours). The 2650 is a rack-mount unit, I don't know that there's any option as to which PCI slot the PERC 3/DC is installed in, and to further complicate matters, it's in a remote co-lo facility, so it's not a trivial matter to "pop the hood" on the system.

Any other suggestions, or is it time to call Dell support?

Good luck!  I was hoping that would address it.  I have read so many complaints on write performance with a 2650.  There are three PCI slots in a 2650.... the PERC3/DC can go in any of those three.  Like I said, I dont know that it will help, but if you search around the forums, there are others who have gotten different results based on the PCI slot chosen.

When I complete my testing, I will post my results.

Message Edited by Quino on 08-09-2003 10:02 AM

Just curious - how can you do performance testing when you don't know the workload on your system?  If your previous test was during the day when you had a large user workload, then how do you know they haven't already used up your available I/O performance?

Good question - I've run the benchmark at varying times over a course of weeks, both at peak and off-peak hours, and have had the same performance results consistently - the numbers that I've posted to this thread are the most frequent range, and are actually very close to the top end of the performance curve - I've managed to test during heavy load times and seen performance significantly lower.

Hope that answers your question.

Just checked with our co-lo guys, and it appears that the PERC 3/DC is in PCI slot 1. Now, according to the 2650 documentation, slot 1 runs at 100 MHz on bus 1, slot 2 runs at 133 MHz on bus 1 if no card is in slot 1 (otherwise 100 MHz), and slot 3 runs at 133 MHz on bus 3. I don't see anything in the PERC 3/DC documentation that specifies whether it will make any difference as to the speed of the PCI slot - should this make any difference?

I'm going to plan to find some time to take the system offline and try moving the PERC card to slot 3, see if that makes any difference. Will post back to this thread with results. Thanks to all for your advice.

Hi

I have moved the PERC3 /DC cards around to the different slots. It made no differance. Hope this saves you some time.

Hi I intend on talking to my Dell rep on this issue. If he comes back with anything useful I will forward it on to you.

Drat... was hoping to hear the opposite from someone who had tried it...

Back to the old drawing board... thanks for your input!

Hello,

I'm using also a similar configuration (Dell PE2650, PowerVault 220s, 8 disks, 18Gb, 15K in RAID 10 config, connected to Perc3/DC with splitted bus in 220s, using 2 SCSI cables). Performance is also slow (reads about 8-10Mb/s when reading 4Kb blocks, but over 100Mb/s when reading 512Kb blocks, write is not tested, but I expect similar values). We are using Rational clearcase, which is a typical DBMS look-a-like application.

First, I upgraded to firmware E14, and did not see any difference compared to E.10. Also the newerst SCSI drivers were used + latest firmware for the 3/DC controller. Nothing helped.

As far as I discovered from the Dell documentation: RAID 10 should have better performance, and multiple LUNS may help (http://www.dell.com/us/en/esg/topics/power_ps2q02-wong.htm), stripe size should be set to what is needed for your business, i.e. small stripe size for random reads/writes, bigger for more sequential reads/writes (http://www.dell.com/us/en/esg/topics/power_ps1q02-perc.htm).

However, I did not test this for now, since I have to plan maintenance time for that. I asked Dell support for info, and they suggested to contact the account manager and/or Dell sales, however I do not think to get that technical info I want...

Currently I did not get a clear answer on what (negative) effects the RAID10 solution from Perc3/DC (RAID 1 concatenated) has on performance compared to a real mirrored stripe (RAID10). The strip should just give the enhanced performance by using more spindles/disks in a volume, and the mirroring the reliability.

Kees

We have purchased (earlier in the year and not yet deployed) a 220S attached to a pair of 2650s via Perc 3/DC.  The intended application is OLTP. Prior to purchasing the system and after much reading and review of the various Dell documents (Wong, et al), and after guidance from Dell’s sales engineers (the technical ones) we had resolved to use RAID-10 for its speed (via striping/multi-spindles) and fault tolerance capabilities. 

However, just by chance, was reading some online docs for the ArrayManager v3.4 and ran across a confusing reference to the substitution of a RAID “1+concatenation” method instead of the desired RAID-10 that every other document indicates is supported.

Note that the PERC 3/DC User’s Guide defines the six official RAID levels that are supported.  These are exactly as expected, with no mention of the use of 1+concatenation as a substitute method for true RAID 10.  In addition, in Dell documents / charts that address the relative merits of various RAID levels, in NOT ONE is the RAID 1+concatentation discussed.

This has led to further review of Dells documents.  Here is what I have found so far.

In Dell OpenManage Array Manager 2.6 User's Guide (Dec 2000) there is no mention of the substitution.  Only the six generally recognized formats are listed as being supported.

However, in Dell OpenManage Array Manager 2.7 User's Guide, dated: January 2001 …
This is the first document that introduces the replacement of the desired RAID-10 with the unwanted RAID 1+concatenation concept.  Following is relevant content:

Concatenation

Concatenates n disks as one large virtual disk with a capacity of n disks.
Data fills up the first disk before it is written to the second disk  (So there goes all the benefits of multi-spindle performance…no wonder that folks are not seeing the performance referred to in PeterWong’s 220S whitepaper).
No redundancy data is kept. When a disk fails, the large virtual disk fails.
No performance gain.
No redundancy

Note: On the PERC 2/SC, 2/DC, 3/DCL, 3/DC, and 3/QC controllers, RAID-10 is implemented as RAID-1 Concatenated. See the next topic RAID Level 1-Concatenated (Concatenated mirror). RAID Level 1-Concatenated (Concatenated mirror)
RAID-10 on PERC 2/SC, 2/DC, 3/DCL, 3/DC, and 3/QC controllers is implemented as RAID-1 Concatenated.
RAID-1 Concatenated is a RAID-1 array that spans across more than a single pair of array disks. This combines the advantages of concatenation with the redundancy of RAID-1. No striping is involved in this RAID type.
Also, RAID-1 Concatenated can be implemented on hardware that supports only RAID-1 by creating multiple RAID-1 virtual disks, upgrading the virtual disks to dynamic disks, and then using spanning to concatenate all of the RAID-1 virtual disks into one large dynamic volume.

If one did not have the docs, or simply used the Array manager online, it is not clear to me that there would be any notice whatsoever of the technology substitution.  Furthermore, there is no mention of this substitution in the supporting documents for the other available methods of array configuration (BIOS Config Utility, Dell Manager, WebBIOS Config utility).  So, if I understand this substitution correctly, here is what the end result would have been in our case:

What we wanted (the Desired RAID 10):  8 Drives total, two stripe sets of 4, mirrored.  This would have data being actively striped across the four drives.  Very fast and great for OLTP.  Fault tolerance courtesy the mirror.

What we would end up with (Unwanted RAID 1+Concatenation):  If I understand the process, while we thought we would be getting the RAID 10, the system would in fact deploy four drives allocated to a concatenation series, meaning that only one drive would be used until the 36GB was filled, then data would be applied to the second drive.  Fault tolerance via the mirror.  I can understand the posts in this forum that are complaining about the poor responsiveness.  Instead of four active drives, you appear to have one.  This would seem to be a material mis-representation of the systems capabilities.

Finally, some questions I have that perhaps some of you folks can help me out with:

1. Since this substitution is only referenced (as far as I have found) in the ArrayManager documents, does this apply only if one is using ArrayManager?  (ie:  if using the BIOS Config utility, do you end up with what you wanted – the RAID-10?)

2. Or, is there a general defect in the PERC x/DC controller such that no matter what method you select to build/administer your arrays, you end up in the same spot?  Does the PERC x/DC work with non-dell arrays?

3. Is there a general defect in the 220S enclosure? Has anyone used a 220S with non-PERC controllers?

4. Has anyone received any updates from Dell?  The complete lack of moderator input on this thread is a bit troubling.

Really open to input here (Sorry about the posts an multiple threads – seems different folks are experiencing different symptoms).  We were counting on this system to deliver high data processing performance AND uptime/reliability/fault tolerance.  The posts here are troubling.  As it stands, I could not recommend that we deploy.

Hello,

My feeling is that the problem is in the PERC 3 controller, but I'm not sure. If the webbios docs are read (http://docs.us.dell.com/docs/storage/perc3sc/webbios/en/webbios_p3.html), you will something about spanning (which is IMO similar to concatenating). See also (http://docs.us.dell.com/docs/storage/perc3sc/ug/en/bios.htm). A remark is in the document that the PERC 3 family only supports spanning for RAID 1 and RAID5. No word about striping and mirroring.

You may be right that the Array Manager is the cause that RAID 10 (or better said a mirrored stripe, which is actual 0+1) is implemented as concatenated RAID 1 (thus not having any performance gain by writing to the different disks in a stripe).

I'm in contact with Dell to get more information. Note that other vendors (like Compaq/HP with its SmartArray 6402 controller really provides hardware RAID 0+1, i.e. mirrored stripe). However, to put a HP/Compaq card in a Dell sounds not very nice to me. I will first await the response of the Dell people. If Dell will not provide a solution, I'm thinking about either 2 RAID0 volumes (one for left 4 disks of PV220s, and one for right 4 disks) + software mirror, or RAID5 for 7 disks :-(, or buying a better controller card which really supports mirrored stripes.

BTW. I didn't search on Adaptec's website for more info about RAID 0+1. In fact the PERC3 is a adaptec 39160 + additional daugther card. May be they may provide more information about how RAID10 was implemented.

Kees

Kees:

Thanks for the tip on the Adaptec Product.  4+ hours into reading about the Adaptec product line, I have come to the following:

The Adaptec 39160 is only a SCSI controller.  It has NO RAID capabilities itself.  Quote from the Adaptec online help:

Q:  Can the Adaptec SCSI Card 39160 do hardware RAID?

A:  The card is not capable of hardware level RAID.  If you are looking for Ultra160 Dual Channel SCSI card that is capable of hardware RAID, you might want to look into Ultra160 or Ultra320 SCSI RAID from Adaptec.

So, at this point, I am left with the assumption that any implementation of RAID in the Dell units is via some other method and indepedant of Adaptec.  Another quote from the Adaptec site indicates that they have no software RAID products.

The more I read, the more stumped I am getting.  (The intra/inter company inconsistancies in defining RAID 10 vs. 0+1, etc. is not helping.  Would be really nice if we could agree on terms here....)

Will keep advised.

 

 

Hi,

Excuse me, I'm wrong, but our Dell's PE 2650 report that two adaptec 39160 are in it (in Windows 2000's device manager), one perc 3/dc and one perc 3/di. The 3/di controller is on the motherboard (may be also on a 39160 based?). One 39160 is in the system for the tape device, and one perc3/dc. The perc3/dc is a regular PCI card  with a daughter card in. I think you are right it is not from adaptec, but from LSI/Ami/Dell (see http://support.euro.dell.com/nl/nl/filelib/system.asp?sid=PWE_FOS_XEO_2650&m=o&p=WNT51) I thought that somewhere else in the forum somebody spoke about using LSI drivers, but this doesn't make much sense now. There is (of course) nothing to find about concetenated RAID 1 on www.lsilogic.com. Looking to the pictures used in the MegaRaid 1600 manuals, I may guess Dell is using a LSI 'look-a-like' controller for the PERC3/DC. The only thing I found in the MegaRaid manuals is that RAID10 is called spanning of raid 1 volumes (this may sound similar to concatenated raid 1), instead of mirrored stripe (which is actually RAID0+1). But anyhow, LSI is also saying that RAID10 is intended for high performing mid-sized databases (see hardware manual of e.g. the MegaRaid1600 or 1650).

However, as soon as I get a reply back from Dell, I will post it here. I'm just wondering why dell is showing performance tests for a powervault 220s, which shows that RAID10 outperformed RAID5 (www.dell.com/us/en/biz/topics/power_ps2q02-wong.htm). This should only possible if some kind of striping is done. But probably we made a mistake :-(

Kees