PE T320 and S110 RAID 10 - Server 2012 - Abysmal Disk Subsystem Performance

Open a case with Dell Support ... you can't do that here.

Hi,

For these numbers can you quote the IO sizes & Q depths running @ the time when you took the readings?

-Deepu

Sorry to bring up such an old thread, but were you ever able to figure out a solution to this problem?  We are seeing slow performance on R220 servers with S110 controllers running RAID 1, but have no issue with our T110's.  Latest drivers/firmware/bios do not seem to be of benefit.  

I know this is old, but I am desperate.  What ended up being the resolution to this? I am having this exact same issue today and Dell is telling me it is a hardware vs. software RAID issue, and I know it is not.

If I recall correctly I just had to wait for the background array initialization to finally complete (it may take several hours or more) and then the performance improved a bit.  Download Dell OMSA (Open Manager Server Administrator) and install it on the server in question.  It will allow you to monitor the progress of the Background Init process on the drive array.  Once the init process is complete, you should see performance improve.

www.dell.com/.../OMSA

FWIW, I just jumped onto the server that I started this thread about and ran a Crystal Diskmark again and the numbers are as follows:

4 x 500 GB SATA drives in RAID-10

-----------------------------------------------------------------------

CrystalDiskMark 3.0.2 x64 (C) 2007-2013 hiyohiyo

                          Crystal Dew World : http://crystalmark.info/

-----------------------------------------------------------------------

* MB/s = 1,000,000 byte/s [SATA/300 = 300,000,000 byte/s]

          Sequential Read :   189.667 MB/s

         Sequential Write :   127.642 MB/s

        Random Read 512KB :    36.863 MB/s

       Random Write 512KB :    92.901 MB/s

   Random Read 4KB (QD=1) :     0.569 MB/s [   139.0 IOPS]

  Random Write 4KB (QD=1) :     2.929 MB/s [   715.1 IOPS]

  Random Read 4KB (QD=32) :     2.898 MB/s [   707.5 IOPS]

 Random Write 4KB (QD=32) :     2.939 MB/s [   717.4 IOPS]

 Test : 1000 MB [D: 11.9% (99.2/832.8 GB)] (x5)

 Date : 2015/12/24 13:43:00

   OS : Windows Server 2012 Server Standard Edition (full installation) [6.2 Build 9200] (x64)

IMHO, the Dell S110 controllers are rather poor performers.  They were barely adequate in performance when they were introduced and since then mechanical drives are faster and SSDs are prevalent.  Sadly the S110 and associated are still being offered by Dell and the performance of these controllers is very sub par.

I now use the Dell T20 for many of my smaller Server builds.  It is not saddled with the S110 controller and instead is a pure Intel RST solution.  I find the Intel RST performance is about as good as it gets without a dedicated CPU for the RAID controller.

Here are some numbers for the T20 in RAID-10 with 4 Seagate 1TB Constellation drives and Intel RST:

-----------------------------------------------------------------------

CrystalDiskMark 3.0.4 x64 (C) 2007-2015 hiyohiyo

                          Crystal Dew World : http://crystalmark.info/

-----------------------------------------------------------------------

* MB/s = 1,000,000 byte/s [SATA/300 = 300,000,000 byte/s]

          Sequential Read :   309.817 MB/s

         Sequential Write :   295.623 MB/s

        Random Read 512KB :    44.745 MB/s

       Random Write 512KB :   114.239 MB/s

   Random Read 4KB (QD=1) :     0.600 MB/s [   146.5 IOPS]

  Random Write 4KB (QD=1) :     2.312 MB/s [   564.5 IOPS]

  Random Read 4KB (QD=32) :     4.226 MB/s [  1031.7 IOPS]

 Random Write 4KB (QD=32) :     2.405 MB/s [   587.1 IOPS]

 Test : 1000 MB [D: 28.5% (977.0/3433.0 GB)] (x5)

 Date : 2015/12/24 14:05:01

   OS : Windows Server 2012 R2  [6.3 Build 9600] (x64)

and just for comparisons sake, a T20 with two 1TB Crucial SSD's in RAID-1 and Intel RST:

-----------------------------------------------------------------------

CrystalDiskMark 3.0.3 x64 (C) 2007-2013 hiyohiyo

                          Crystal Dew World : http://crystalmark.info/

-----------------------------------------------------------------------

* MB/s = 1,000,000 byte/s [SATA/300 = 300,000,000 byte/s]

          Sequential Read :  1043.186 MB/s

         Sequential Write :   487.407 MB/s

        Random Read 512KB :   805.658 MB/s

       Random Write 512KB :   411.275 MB/s

   Random Read 4KB (QD=1) :    31.286 MB/s [  7638.1 IOPS]

  Random Write 4KB (QD=1) :    94.559 MB/s [ 23085.6 IOPS]

  Random Read 4KB (QD=32) :   549.455 MB/s [134144.3 IOPS]

 Random Write 4KB (QD=32) :   264.189 MB/s [ 64499.3 IOPS]

 Test : 1000 MB [D: 11.5% (100.7/875.7 GB)] (x5)

 Date : 2015/12/24 14:19:55

   OS : Windows Server 2012 R2 Server Standard (full installation) [6.3 Build 9600] (x64)

As you can see, the Dell T20 with SSD's and Intel RST make a great combo.  Equip the Xeon T20 with 32 GB of RAM and SSD's and you've got a great system for running several VM's.

Do you have a number for the CPU usage by the Intel RST during heavy usage?

With 100% mechanical drives the CPU usage is 6% or less, even with QD=32.  

With SSD's it gets a bit higher, but you're moving substantially more data.  Sequential is about 10% for the RAID-1 pair, QD=32 can be as high as 30% but it only seems to use 1-2 threads.  With a 4 or 8 thread processor it does not seem to impact performance at all.  In systems where I have configured a RAID-1 SSD pair along with another RAID-1 mechanical pair I'm able to run Crystal DiskMark simultaneously testing each pair and the numbers stay pretty consistent.  

If you look at the numbers in my last post, comparing QD=32 IOPs, Mechanical vs SSD, the SSD numbers are over 100x higher.  I'd expect to use a bit more of the CPU to move 100x more data.

And please keep in mind that these numbers are from a synthetic benchmark.  In the real world I've had no trouble running 4 or 5 VM's on one of these T20's, equipped with a Xeon CPU, 32 GB RAM, 1 TB SSD's in RAID-1 along with 2 TB SATA Mechanicals also in RAID-1.  

I hope this information is helpful.

Thanks for the reply. Our server takes a full ten minutes from power on until crtl-alt-del prompt and the hard drives are constantly being hit, even with nobody on the system. I have uninstalled everything, turned off all indexing and shadow copies and the drives are still constantly active. Dell swapped the motherboard and cable to no affect, so I am not sure what we are going to do next. I am in the same situation. I have a 10 year old clone server in my scrap heap that is faster than this thing.

My experience with this issue was way more than just poor performance.  As Andy mentioned, my servers were completely unusable - like running a machine pegged at 100% CPU (but hardly any CPU actually being used).  I haven't looked at this since I posted.  We ended up forcing Dell to replace the servers with ones that included the low end hardware raid controller option.  They run fine.  I hoped/assumed it was a driver issue that Dell would have solved by now (been over a year).  Unfortunately it looks like Dell has not released a newer driver for the S110 controller for Server 2012.  If I remember right, Server 2008R2 ran fine - it was just Server 2012 that was horrible.  Maybe try the latest BIOS (Dec 2015)?