R610 Perc6i abyssmal disk subsystem performance issues - need help.

two quick comparos I have handy:-

7CSD62S: R610 Perc6i Win2003sp2 non-aligned partitions but something else very fishy, ultra slow diskperf
CrystalDiskMark 3.0 (C) 2007-2010 hiyohiyo
           Sequential Read :    29.596 MB/s
          Sequential Write :   342.225 MB/s
         Random Read 512KB :     3.959 MB/s
        Random Write 512KB :    40.069 MB/s
    Random Read 4KB (QD=1) :     0.177 MB/s [    43.3 IOPS]
   Random Write 4KB (QD=1) :     1.705 MB/s [   416.2 IOPS]
   Random Read 4KB (QD=32) :     0.169 MB/s [    41.2 IOPS]
  Random Write 4KB (QD=32) :     1.088 MB/s [   265.5 IOPS]
  Test : 4000 MB [C: 8.1% (3.5/43.0 GB)] (x9)
  Date : 2010/11/29 16:35:35
    OS : Windows Server 2003  SP2 [5.2 Build 3790] (x86)

6CSD62S: R610 Perc6i Win2003sp2 aligned partition offsets
CrystalDiskMark 3.0 (C) 2007-2010 hiyohiyo
           Sequential Read :   496.602 MB/s
          Sequential Write :   346.665 MB/s
         Random Read 512KB :    89.084 MB/s
        Random Write 512KB :   114.979 MB/s
    Random Read 4KB (QD=1) :     1.242 MB/s [   303.3 IOPS]
   Random Write 4KB (QD=1) :     4.423 MB/s [  1079.9 IOPS]
   Random Read 4KB (QD=32) :     9.548 MB/s [  2331.2 IOPS]
  Random Write 4KB (QD=32) :     2.906 MB/s [   709.5 IOPS]
  Test : 4000 MB [C: 10.6% (5.2/48.8 GB)] (x9)
  Date : 2010/11/29 13:19:31
    OS : Windows Server 2003  SP2 [5.2 Build 3790] (x86)
 

HDTune graph

http://img442.imageshack.us/i/hdtuner610perc6i4x146gb.png/

as noted above:

3 servers, same hw specs, same BIOS version, same firmware version, same driver

- one with very different (read: poor) performance.

Pretty pathetic reads, it  acting like the" WB  Enabled" is not being being used. The HD test seems to show it running in WT

Don't know if running the Dell diagnostic will find anything, as the array switched to another server acts normally, you might try the diags.

You might try re-setting WB via the cli even if it shows as already enabled. 

<ADMIN NOTE: Broken link has been removed from this post by Dell>

If no help, I would have Dell send out a replacement. Doubt this issue is  going to be be a OS level software issue, with those low read speeds. Did you try the tests in safe mode, if your disk tests do not function at safe mode level, get a simple DOS based disk test such as Cosbi (no software added to server, safe for server use). This should rule out most software/driver interferance. The only software I know which could cause such an an issue is AV software (other then a bad driver), still very unlikely with those test results, disable it and test, if it is installeded

File Copy HERE Ver. 0.52 HDD test."

http://4peeps.com/ivb/index.php?showtopic=4108

Hi,   Cheers for the prompt reply & suggestions. My understanding was that Cosbi was a windows app, and any dos based disk testing might be limited to 32Gb fat32 partitions (rather than the whole disk).

(why does this Dell forum editor persist in using some funky html formatting...)

Cosbi  Ver. 0.52 is a small disk benchmark, will work on any 32 bit Windows or DOS, system does not add any files to the system, never tried it on a GPT disk, but has worked on any 32 bit  NTFS system I tried it on, all much greater than 32 gig. I was not referring to the upgraded suite, as I do not like using Windows benchmark  apps which add files/drivers to client's production systems, only to the DOS based version referred to in the phrase "File Copy HERE Ver. 0.52 HDD test."
.  Good luck

Hi guys,

Ok, a little update.

RAID controller on another R610 swapped into this problem unit - same performance issues on this system, other system maintains existing decent performance. [these are two identical Perc6i w/BBU, same fw]

On a hunch, I've trashed the 5 disk raid5 array and rebuild 5 separate (single disk raid0) arrays. 4 give stellar performance, and one gives the up&down (hero to zero) see-saw performance. HDTune gives a visual rep of the behaviour I'm talking about.  pics attached (1 good drive, 1 bad drive). I've seen no SMART warnings for this disk. SeaTool Enterprise (generic test) passes this disk.

Yet to go thru the Dell diagnostics (that I'm sure Dell support will want to use) or check for hdd fw upgrades [but the Perci reported them as all the same fw if I remember correctly]. I've also got to swap drives slots around, in case it is a bad connector/cable or something.

a good drive for reference/comparison

146Gb 10k SAS drives in this unit.

"RAID controller on another R610 swapped into this problem unit - same performance issues on this system, other system maintains existing decent performance. [these are two identical Perc6i w/BBU, same fw] "

Assuming you just swapped the controller and not the disks....

From the HD tach, I have never seen the up/down graph like that, it is as if the elevator seeks are not functioning on reads, as if the head returns for every seek of info. As to testing, I have little faith in software based diags if drives are to be used on a raid adapter; expensive hardware testers are the most reliable and only the manufacturers can afford them.

 Have you tried building a raid 5 out of the drives which perform well as raid 0 drives, leaving out the one under-performing drive? Perhaps a small non critical component on the drive controller is malfunctioning, but not critical enough to show in diags. Raid adapters handle physical disk errors, and obvious disk controller errors, but do not do often react to non critical disk controller issues.

Picking at straws...The only other thing I can think of is the interrupt sharing, very doubtful. Possible two devices on the mobo are fighting each other. This rather doubtful as swapping the raid card should have reset the situation, but you might try the berg pin reset or a battery pull. 

Give you an A++ ++ for patients and thorough testing, this is a most frustrating situation,  I would rather have a simple total raid failure, if I have a backup then an issue like this. If the above does not work, and your still under warranty, dump the issue on Dell, You have done everything which could be expected of a mortal IT person. 

Ps, this is another good site to get help with raid issue 

http://forums.2cpu.com/forumdisplay.php?f=26

I've since run the Dell hw diags (from within System Services) 

[the problem disk configured as a single disk RAID0 array hanging off the Perc6i]

Hard Drive   Read Test  failed.   Error code: 4400:051C

Call logged with Dell, new hdd shipped out. Repeating the same test on the new drive now.

Let us know the results

Thanx