MD3000 + 2x MD1000's SLOW! Help

Question

System: PowerEdge R900 64bit 4x Quad 2.4Ghz 6Meg cache 8x 2.5' drives All drives are 73Gig 15K SAS PERC 6/i RAID 1 (2x) for OS RAID 1+0 (6x) for Data   The MD's all have 15x SAS 73Gig 15K drives   R900 has two SAS 5/e HBA's connected to the MD3000 and the MD3000 has two MD1000's connected to it.    For testing i created 3x RAID 1+0 with 14 drives each and have 3 global hot spares.  Doing some normal transfer testing i see that the intigrated RAID 1+0 with the 6 drives can push about 357Megs a second while the MD3000 RAIDs can only push about 80-110megs a second.  I then did SQLIO testing and again found that the intigrated RAID can do 3x more IO's then the external RAIDs.  I'm at a loss now.  I've messed with all the settings i could find in the MD storage manager but there isn't much.  am i missing something?  Our old PV220's blow these MD's out of the water with speed?  I must have something setup wrong.   please advise?  thanks   -king

mot99 · Answer

Make sure you have the latest SAS 5/E driver which is marked urgent and came out a few days ago.     Also latest MD3000 firmware hot off the press:   http://support.dell.com/support/downloads/download.aspx?c=us&l=en&s=gen&releaseid=R174600&SystemID=PWV_MD3000&servicetag=&os=WX64&osl=en&deviceid=10810&devlib=0&typecnt=0&vercnt=4&catid=-1&impid=-1&formatcnt=1&libid=46&fileid=237224   The MD1000 should have this latest version as well:

SAILORKING · Answer

mot99,   thanks for the reply.   yes usually when i get new HD i insure i'm on the latest everything.  is there a certain way to make disk groups or even connect to the MD's?  I pulled them out of the box and plugged them in and they worked but i really have yet to insure i daisy chained them properly.

Dell-Jeff G · Answer

The support matrix has proper cabling diagrams. http://support.dell.com/support/edocs/systems/md3000/en/supmatrx/SupportMatrix.pdf

SAILORKING · Answer

Jeff,

Thanks for the reply.

Figure 2 through Figure 4 show redundant, high-availability cabling configurations on one and two hosts. If any one path fails, at least one HBA is still connected to one RAID controller.

Above is what i have. Jeff is their any documentation on actual setting up the RAID that covers how to change the stripe size and the cache policies and actually how to set the MD's up to insure they are at their maximum performance settings? Should MD be slower then a PERC 6\i or slower then the PV220's? I mean i can't get past 110megs a second on a normal data transfer at i get over 300 with the Perc 6\i setup in the same raid configuration with the internal R900 drives. I confused. i must be doing something wrong. what should i see with numbers on a basic install?

thanks

-king

DavidDay22 · Answer

From benchmarking the MD3000, it behaves like the caching policy is not set correctly.

SAILORKING · Answer

David,   exactly.  I'm going to start playing arounf with the SMCli today.  i'll let you know my results but if you have already ventured down that path let me know.  thanks   -king

SAILORKING · Answer

David,   yeah my PERC 6\i is fine its the RAID controllers in the MD3000.  I use two HBA's SAS 5/E's to connect to them.  thats where the problem is.  I know my setup or something is wrong because these performances would never fly.  however if i + dell can't resolve it, then its time to rethink the MD's.  But i know it can't be.  hopefully i can figure it out and i'll post when i do.   -king

DavidDay22 · Answer

I have just setup a Poweredge 2950, with a MD3000.  With the 72 G 15K drives on the PERC6/i I have been able to get about 800 MB/sec and about 700 MB/sec on the MD3000.  I have 15 146 G 15K drives in the MD3000, and it is setup with 2 RAID arrays.  The PERC 6i is also setup with a RAID 10 array.  What bothers me is that the disk latency seems to rise on the MD3000, slowing I/Os down.

dzenz · Answer

In general it's difficult to provide comments on performance related questions without more detail on the complete setup, including: cabling, host OS, virtual disk configuration, and the application that is being used to measure performance. MD3000 data can be copied/pasted from the MDSM storage array profile into a post.   For typical usage the cache parameters on the MD3000 really do not need to be tweaked as the defaults are good as is for the vast majority. YMMV so do feel free to experiment!   Write cache is enabled by default but will be temporarily disabled if the battery is missing or expired (either would put the array into a Needs Attention status). Also, as the RAID controllers manage the cache read/write % utilized dynamically there are no required parameters for this.   Dave

DavidDay22 · Answer

I had a Dell engineer review my setup for about an hour, and my conclusions are that the MD3000 with 15 146 G 15K drives is pretty good in read speed, but not so good in small random writes.  The write caching doesn't seem to give same performance as the PERC6 I have internally on my server.  My arrays were setup in RAID 10 and RAID 0, overall whether it was 2 disks in RAID 0 or 12 disks in RAID 10, the maximum number of io/sec topped out at about the same number.  Here are my test results using SQLIO for a 12 disk RAID 10 array:   Test IO/sec Random write 8KB 4,321.39 Random write 64KB 1,857.56 Random write 128KB 1,117.00 Random write 256KB 631.11 Seq write 8KB 7,943.94 Seq write 64KB 1,897.39 Seq write 128KB 1,009.08 Seq write 256 KB 528.56 Random read 8 KB 35,536.21 Random read 64 KB 9,831.23 Random read 128 KB 5,744.66 Random read 256 KB 2,876.53 Seq read 8KB 36,548.33 Seq read 64KB 8,761.72 Seq read 128KB 4,600.73 Seq read 256KB 2,382.96   For comparison here are my results for a 4 disk RAID 10 with a PERC6i controller:   Test IO/sec Random write 8KB 41,187.19 Random write 64KB 8,292.15 Random write 128KB 4,066.13 Random write 256KB 1,942.65 Seq write 8KB 49,129.36 Seq write 64KB 8,276.62 Seq write 128KB 4,074.46 Seq write 256 KB 1,910.54 Random read 8 KB 47,128.68 Random read 64 KB 12,262.52 Random read 128 KB 6,233.61 Random read 256 KB 3,194.80 Seq read 8KB 61,689.58 Seq read 64KB 12,244.57 Seq read 128KB 6,221.71 Seq read 256KB 3,186.80

SAILORKING · Answer

Davidday22,   The numbers you are getting is exactly what i'm getting.  Is the MD3000+2x MD1000 just a dog?  Have you been able to find anything out.   I'm at a loss.  it is time to see what DELL says.

DavidDay22 · Answer

I had to put my server into production, and so far it has run pretty well.  I have 32G of memory and this may be mitigating the slower performance on the 8K random writes.  Since I tried a variety of disk and RAID combinations, I think the limiting factor is throughput of the HBA.  They are only 4X and a 2950 has 2 8X slots.  I will be interested in what Dell has to say.

dining_philosop · Answer

A couple of high level concepts about random IO performance. With all RAID controllers true random IO is a function of the number of disk drives that are attached to the controller (this assumes that there is no caching). This is because unlike sequential IO tests and cached IO tests, a random IO test is most affected by the seek latency of disk drives. The seek latency means that the number of IOPs that you are seeing is consistent with the performance of the number of spindles that you have configured (+ additional benefit from cache).

Cache helps when there is "locality of reference". This means that there are enough accesses to the same location on the virtual disk so that the controller gets a cache hit. With most real applications there is locality of reference and hence cache helps and that is why you are not seeing issues in your production system (that you anticipated based on the test that you did). With most test tools the random IO is so random that you don't see the locality of reference and hence any benefit from cache.

Since you used a 4 HDD R10 for PERC 6i and 12 HDD R10 for the MD controller, was there a difference in size of the virtual disk that was configured for the test?

AlexSelles · Answer

I've recently setup a MD3000 connected to a single PE1950 with 2 SAS5/E cards installed. The MD3000 is running 15x 146Gb 15K SAS drives. I have one hot spare and the remaining 14 drives configured as raid10. I'm currently getting a sequential read speed of 150mb/s and burst of 200-220. The sequential read speed is only 50mb/s faster than an old PV220S (14x 73Gb 15K SCSI raid5) that we have.

The information that I've seen in this thread suggests that the MD3000 is capable of, and we were expecting much higher speeds than I've been able to get. I have raised this with Dell support, and I'm waiting on some answers...

Can anyone suggest some configuration changes that would/could give me better performance from this server?

The setup is;

PE1950 dual quad x5355 2.66Ghz 4Gb ram, Win2003 R2 Ent x86 SP2

specific config of the MD is

set storageArray mediaScanRate=15;

set physicalDisk[0,0] hotSpare=true;

set storageArray cacheBlockSize=4;

set storageArray cacheFlushStart=80;

create virtualDisk physicalDisks=(0,1 0,2 0,3 0,4 0,5 0,6 0,7 0,8 0,9 0,10 0,11 0,12 0,13 0,14) raidLevel=1 userLabel="Disk" owner=1 segmentSize=128;

set storageArray cacheFlushStop=80;

set virtualDisk["Disk"] cacheFlushModifier=10;
set virtualDisk["Disk"] cacheWithoutBatteryEnabled=false;
set virtualDisk["Disk"] mirrorEnabled=true;
set virtualDisk["Disk"] readCacheEnabled=true;
set virtualDisk["Disk"] writeCacheEnabled=false;
set virtualDisk["Disk"] mediaScanEnabled=true;
set virtualDisk["Disk"] consistencyCheckEnabled=true;
set virtualDisk["Disk"] readAheadMultiplier=1;
set virtualDisk["Disk"] modificationPriority=high;

PowerVault

MD3000 + 2x MD1000's SLOW! Help

Was this post helpful?