Slow Performance with vSphere and MD3000i

Question

Hi,

we have got a new MD3000i last week and I had really fun playing with it. I like it when hardware is simple to set up and with this tutorial it was a peace of cake getting our two esxi-servers up and running againt the storage.
But after some initial testing of failover and other stuff, I did started benchmarking the whole system. And I found out I only get about 30MByte/s read-only. So I added Jumbo-Frames, tried connecting the servers directly via crossover, enabled and disabled round robin but the best I could get with HDTune (and verified it via smCLI or esxtop) was about 50MB/s. Which is fine for most servers, but we want to add a virtualized fileserver and then 50MB/s is not enough.
Out of curiosity I then ran 2 instances of HDTune and the throughput I recieved nearly doubled. With 4 instances I had the full 110MB/s and the switch was at running 98% load.
Now comes the part where I need help:
How do I get 100MB/s just for one instance of the benchmark (or a filecopy etc.)?

Would be great if someone here could help me out ;)

Thanks in advance!

Chris

david4hand · Answer

I'm having similar issues. Received a new MD3000i and two PC-5424 switches two weeks ago. Dual controllers, 7 15KRPM 300GB SAS drives and 8 7200RPM 500GB Nearline drives. Each are configured large RAID5 volumes (one 500GB drive is set as hot spare). Dell did a remote setup for us (basically just confirmed I had everything setup correctly). I've configured and reconfigured everything 20+ times. I have tried JF, no JF, MRU, Round Robin, etc. The max I can get is about 50MB/sec. Copying to/from our NFS server (a cheap ReadyNAS device) to the SAN is only 20-30MB/sec. Copying to/from NFS to/from local storage is 70-80MB/sec.

JOHNADCO · Answer

Not horrible.. Probably should be better.... Are you two new to sans? Sustained is never really that impressive, it's the IOPS that count here.

I can do some testing as well. We just installed our first Vsphere host yesterday.

cpt86 · Answer

Its my first SAN but I read a lot and expected more than 30MByte/s. If it would be around 80-90MB/s I would be totally happy and I don't need anything more.

The funny thing just is that in the same VM running against one LUN i get the performance i wish just by starting a second or third benchmark instance.
If it would be seperate VMs against different LUNs or different controller I would have a clue but every monitoring tool says to me I get around 100-110MB/s from one VM over one switch port to the storage. Just under some strange conditions. I want these 100MB/s with just one benchmark instance ;)

Edit: IOPS and latency are fine though...

david4hand · Answer

It's my first SAN as well, but I've been using a ReadyNAS pro for the past year (both iSCSI and NFS). I get 70-90MB/sec transfers on the ReadyNAS via NFS, so I was expecting something more from a entry level $12K SAN.

I would think that 7x 15K RPM SAS drives in RAID 5 would be able to saturate a 1Gb/sec link. Especially since the ReadyNAS with regular 7200RPM SATA drives comes close.

My main issue with throughput is the speed at which I can backup my data from the SAN. Maybe I need to use something other than GhettoVCB, but at 30-50MB/s takes 50% longer to backup from the SAN to our NFS server than it did from local storage.

Using HD Tune Pro I get the following IOPS (Random Access test).
512 bytes 346 IOPS / 2.9ms / 0.169 MB/s
4 KB 338 IOPS / 3.0ms / 1.321 MB/s
64 KB 253 IOPS / 3.9ms / 15.859 MB/s
1 MB 91 IOPS / 10ms / 91.627MB/s
Random 130 IOPS / 7.7ms / 66.002 MB/s

cpt86 - how does this compare to what you get?

david4hand · Answer

FWIW I found this thread: http://communities.vmware.com/thread/73745

My performance seems on par with others. I'm going to try disabling Jumbo Frames again. It seems many people find JF actually hinder performance (but reduce % CPU a bit).

UPDATE: I disabled JF and my performance increased by about 10-15%. CPU usage only went up 2%.

UDATE2: With JF disabled I tried enabling Round Robin (which killed performance before). Performance went up another 10% on IOMETER, but dropped on HD Tune Pro.

Kong Yang · Answer

david4hand and cpt86- Thank you for sharing your MD3000i + VMware vSphere 4 performance.

Your performance can vary depending on your workload profile as you have observed with multiple benchmark instances and different benchmarks. Your IOPs and throughput seem low though. A RAID 5 volume should be able sustain 150 IOps per spindle with smaller IO packet size. Are you using TCP Offload b/c offload can negatively impact performance?

Next, since you are performing backup, your IO profile should be large, sequential reads from your MD3000i and sequential writes to your NFS server. So Jumbo Frames should help you. The one caveat with JF is that you need to have it enabled end-to-end. Otherwise it can negatively impact performance. With JF enabled, does the command "vmkping -s 9000 from your ESX server return correctly?

As for RR storage path policy, your trade-off is between path thrashing and having enough IOPs to maximize the bandwidth utilization on your multiple paths. Please refer to:
http://www.delltechcenter.com/page/A+%E2%80%9CMultivendor+Post%E2%80%9D+on+using+iSCSI+with+VMware+vSphere
Particular, Question #3.

david4hand · Answer

I'm pretty sure JF were setup correctly. I followed the exact setup from this document: http://www.delltechcenter.com/page/VMware+ESX+4.0+and+PowerVault+MD3000i.

I could never vmkping with a size of 9000, it maxed out at 8992 I think. JF were set on the NIC, vmknic, vswitch, MD3000i and PC 5424 switches. From what I've experienced though, the headache of JF doesn't justify the performance gain (if any).

Thanks.

JOHNADCO · Answer

I found JF to be a trade off, my sustained throughput went up, but it seemed at the expense of soem IOPS.

I must not have downloade the same version of HDtune. the version I have gave better results, but I think it's because we run 14 drives in our disk groups.

I did seem to be able to get better aggregate throughput, but with only one HDTUNE runnung I hit about 70mb per second. I am probably not testing what I need to test here. I am going to load IO meter eventually, but this upgrade to Vsphere from 3.5 / 2.5VC is killing me.

david4hand · Answer

I would hope that 7 15K RPM SAS drives in RAID 5 could sustain better than 50-60MB/sec. For some reason my performance has gone to crap. Everything seemed 'acceptable' with one LUN and one ESXi box connected, so I went ahead and carved out 5 more LUNS and connected our second ESXi box. Performance is now down about 30% (maxing out at 250 IOPS on HD Tune Pro at 4K).

JOHNADCO · Answer

My version of HDTune isn't giving me IOPS.

cpt86 · Answer

I had some conversations with VMWare now and it all was a measurement error.
You should use a RDM device and IOmeter for the benchmark and the following settings:
Disk Targets:
Maximum Disk Size: 1000
# of Outsatind I/Os: 16

Access Specification:
100% write
2MB Transfer Request Size
100% Sequential

With these settings I got full speed.

Thanks for your help though!

Jeff Sullivan · Answer

cpt86 -- thanks for the follow up post!

koen.vdvelde · Answer

Hi cpt86, can you please let us know what you mean with 'full speed' ? tia, Koen.

cpt86 · Answer

@koen About 110MByte/s on the fast array, about 80MBytes/s on the slow array.

brandstockil · Answer

Hi Isn't that 30MB/s down to 32bit windows limitation?

FluidFS

Slow Performance with vSphere and MD3000i

Was this post helpful?