Highlighted
JustMy2Cents1
1 Nickel

R900 poor performance on VMware

Hi,

I'm having horrible bad performance on Dell R900 hosts with VMware. Whatever task that creates writes to disks is extremely slow. For example unpacking a 500MB zip takes 4 to 5 minutes on R900 while the same job is done in one minute on my laptop. When testing there was only one vm running on the R900 host, so its not a load thing. The hosts have 4x4core CPU's, 64GB RAM, 4 HBAs, SAN is EMC VMAX and I've used even LUNs from SSD disks. VMware has checked the configs of ESX and EMC checked the SAN. Both were ok.
0 Kudos
9 Replies
JOHNADCO
4 Tellurium

RE: R900 poor performance on VMware

How long did it take to format the datastore? Assuming VMFS here. Trouble shooting this gets way more complicated if your using RDM.
So when you look at performance for the VM and select datastore do you see high latency?


0 Kudos
Kong Yang
3 Silver

RE: R900 poor performance on VMware

Hi JustMy2Cents,

Do you have any perfmon data from the VM or any esxtop data from the R900 host that you can share? In particular, the disk transfer/sec, disk writes/sec, disk reads/sec, sec/disk transfer, disk bytes per transfer.

Regards,

KongY@Dell

0 Kudos
JustMy2Cents1
1 Nickel

RE: R900 poor performance on VMware

Hi,

Sorry for the delay but I had to migrate couple petabytes of data.

@JOHNADCO
Formatting a 550GB LUN seems to take about 40 seconds and yes we are using VMFS

@KongY

Here are the results of perfmon. I'm running IOMeter with 4k blocks, 100% random with 67%read and 33%write.

Counter average minimum maximum

Disk Transfers/sec 5150kB 4200kB 5878kB
Disk Writes/sec 1730kB 1350kB 1950kB
Disk Reads/sec 3500kB 2780kB 3870kB
Avg Disk Bytes/transfer 4097kB 4091kB 4158kB

From esxtop I get 5253 CMDS/s, 3532 READS/s, 1720 WRITES/s, 13.80 MBREAD/s, 6.73 MBWRTN/s, 0.21 DAVG/cmd, 0.01 KAVG/cmd0.22GAVG/cmd and 0.0 QAVG/cmd

IOMeter gives 4883 Total I/Os per sec, 19 Total I/Os per sec, Average I/O response is 0.4 ms, Maximum I/O response is 620ms CPU utilization is 10% during the tests.

All 5 nodes acts in a similar way regardless how many virtual machines are running on the same box. The very next cluster with other hosts gives four to ten times better results.

-0,02$
0 Kudos
JOHNADCO
4 Tellurium

RE: R900 poor performance on VMware

I think VMware is performing pretty well here....

"Maximum I/O response is 620ms"

This is troublesome.... But your only seeing it at the VM level. How many vCPU's assigned to the guest used for testing?
0 Kudos
JOHNADCO
4 Tellurium

RE: R900 poor performance on VMware

I have to ask here... Your iSCSI storage network is isolated and dedicated right?
0 Kudos
JustMy2Cents1
1 Nickel

RE: R900 poor performance on VMware

I can only agree. I see similar behavor on all virtual machines running on the R900 hosts. Those test results above are from a VM having 4vCPUs and it is Windows2008R2. I have tried with dedicated Disk/CPU/RAM resources but that doesn't make any remarkable differences. Results with windows 2003 VM with 2vCPUs are similar.

Our storage is FC, not iSCSI.

-0,02$
0 Kudos
Kong Yang
3 Silver

RE: R900 poor performance on VMware

Hi 0.02$

When you say "all 5 nodes act the same way.." are all 5-nodes in that cluster R900s? What host servers are in the other cluster? Differences in microarchitecture could explain the perf differences. And when you say 4 to 10 times better results, what's the metric that you're referencing? Is it 4-10 times better response times or more IOPs & therefore, more throughput?

Also, can you please share your IOMeter .icf config file? Thank you.

There are many factors that will come into play to affect overall performance. For example, if your workload profile is single threaded, it won't be able to take advantage of spreading threads across multiple vCPUs.

Your max response time was high. Can you please check your CPU % Wait on host? Also, how many interrupts per second are you seeing in perfmon? Is it 1-to-1 with your # of IOs?

Regards,

KongY
0 Kudos
JustMy2Cents1
1 Nickel

RE: R900 poor performance on VMware

Yes, all nodes in this slow cluster are R900's. The other clusters that I've tested have PE2950's , HP DL380G5, Dell R710 and some other models too. My notebook that beats up these R900's is Lenovo Thinkpad T61P.

My home "cluster" has one home built i7 workstation and HP XW4600 workstation using iomega StorCenter IX4-200d box as shared storage and even this el cheapo gives better performance than R900 cluster with EMC VMAX.

I'm suspecting that model R900 has something seriously wrong in the architecture. Maybe I'll feed them to the Dell sales guy as he sold something that these hosts are not doing.
0 Kudos
Dev Mgr
6 Gallium

Re: R900 poor performance on VMware


@jossy_rascon wrote:

Designed with powerful virtualization capabilities, the PowerEdge R900 provides outstanding performance in a four-socket server with Intel® Xeon® Six-Core Processors and up to 256 GB of memory. The R900’s Xeon 7400 series processors support VMware® vMotion™ for migration of data from one server to another with no interruption of the application, enabling load balancing in real time.


You are replying to a 7 year old thread with marketing stuff from probably 10+ years ago, and add no useful information to this thread. Please refrain from posting unrelated posts in threads.

Member since 2003

0 Kudos