R900 poor performance on VMware

Hi JustMy2Cents,

Do you have any perfmon data from the VM or any esxtop data from the R900 host that you can share? In particular, the disk transfer/sec, disk writes/sec, disk reads/sec, sec/disk transfer, disk bytes per transfer.

Regards,

KongY@Dell

Hi,

Sorry for the delay but I had to migrate couple petabytes of data.

@JOHNADCO
Formatting a 550GB LUN seems to take about 40 seconds and yes we are using VMFS

@KongY

Here are the results of perfmon. I'm running IOMeter with 4k blocks, 100% random with 67%read and 33%write.

Counter average minimum maximum

Disk Transfers/sec 5150kB 4200kB 5878kB
Disk Writes/sec 1730kB 1350kB 1950kB
Disk Reads/sec 3500kB 2780kB 3870kB
Avg Disk Bytes/transfer 4097kB 4091kB 4158kB

From esxtop I get 5253 CMDS/s, 3532 READS/s, 1720 WRITES/s, 13.80 MBREAD/s, 6.73 MBWRTN/s, 0.21 DAVG/cmd, 0.01 KAVG/cmd0.22GAVG/cmd and 0.0 QAVG/cmd

IOMeter gives 4883 Total I/Os per sec, 19 Total I/Os per sec, Average I/O response is 0.4 ms, Maximum I/O response is 620ms CPU utilization is 10% during the tests.

All 5 nodes acts in a similar way regardless how many virtual machines are running on the same box. The very next cluster with other hosts gives four to ten times better results.

-0,02$

I think VMware is performing pretty well here....

"Maximum I/O response is 620ms"

This is troublesome.... But your only seeing it at the VM level. How many vCPU's assigned to the guest used for testing?

I have to ask here... Your iSCSI storage network is isolated and dedicated right?

I can only agree. I see similar behavor on all virtual machines running on the R900 hosts. Those test results above are from a VM having 4vCPUs and it is Windows2008R2. I have tried with dedicated Disk/CPU/RAM resources but that doesn't make any remarkable differences. Results with windows 2003 VM with 2vCPUs are similar.

Our storage is FC, not iSCSI.

-0,02$

Hi 0.02$

When you say "all 5 nodes act the same way.." are all 5-nodes in that cluster R900s? What host servers are in the other cluster? Differences in microarchitecture could explain the perf differences. And when you say 4 to 10 times better results, what's the metric that you're referencing? Is it 4-10 times better response times or more IOPs & therefore, more throughput?

Also, can you please share your IOMeter .icf config file? Thank you.

There are many factors that will come into play to affect overall performance. For example, if your workload profile is single threaded, it won't be able to take advantage of spreading threads across multiple vCPUs.

Your max response time was high. Can you please check your CPU % Wait on host? Also, how many interrupts per second are you seeing in perfmon? Is it 1-to-1 with your # of IOs?

Regards,

KongY

Yes, all nodes in this slow cluster are R900's. The other clusters that I've tested have PE2950's , HP DL380G5, Dell R710 and some other models too. My notebook that beats up these R900's is Lenovo Thinkpad T61P.

My home "cluster" has one home built i7 workstation and HP XW4600 workstation using iomega StorCenter IX4-200d box as shared storage and even this el cheapo gives better performance than R900 cluster with EMC VMAX.

I'm suspecting that model R900 has something seriously wrong in the architecture. Maybe I'll feed them to the Dell sales guy as he sold something that these hosts are not doing.

---

@jossy_rascon wrote:

Designed with powerful virtualization capabilities, the PowerEdge R900 provides outstanding performance in a four-socket server with Intel® Xeon® Six-Core Processors and up to 256 GB of memory. The R900’s Xeon 7400 series processors support VMware® vMotion™ for migration of data from one server to another with no interruption of the application, enabling load balancing in real time.

---

You are replying to a 7 year old thread with marketing stuff from probably 10+ years ago, and add no useful information to this thread. Please refrain from posting unrelated posts in threads.

Resolution
Each step below provides instructions and a link to a document, for performing the step and taking corrective action as necessary. The steps are ordered in the most appropriate sequence to isolate the issue and to identify the proper resolution. They are also ordered in the most appropriate sequence to minimize data loss. After completing each step, evaluate the virtual machine performance again. Work through each troubleshooting step in order, and do not skip a step.

Verify that the reduced performance is unexpected behavior. When a workload is virtualized it is common to see some performance reduction due to virtualization overhead. Troubleshoot a performance problem if you experience these conditions:

The virtual machine was previously working at acceptable performance levels but has since degraded
The virtual machine performs significantly slower than a similar setup on a physical computer
You want to optimize your virtual machines for the best performance possible

Verify that you are running the most recent version of the VMware product being used. For download information, see the VMware Download Center.

Check that VMware Tools is installed in the virtual machine and running the correct version. The version listed in the toolbox application must match the version of the product hosting the virtual machine. To access the toolbox, double-click the VMware icon in the notification area on the task bar, or run vmware-toolbox in Linux. Some VMware products indicate when the version does not match by displaying a message below the console view. For more information on installing VMware Tools, see Overview of VMware Tools (340).

Review the virtual machine's virtual hardware settings and verify that you have provided enough resources to the virtual machine, including memory and CPU resources. Use the average hardware requirements typically used in a physical machine for that operating system as a guide. Adjustments to the settings are required to factor-in the application load: higher for larger loads such as databases or multi-user services, and lower for less intense usage such as casual single-user application like e-mail or web clients. Consult your operating system and application documentation for more information.

Ensure that any antivirus software installed on the host is configured to exclude the virtual machine files from active scanning. Install antivirus software inside the virtual machine for proper virus protection. For more information, see Investigating busy hosted virtual machine files (1003849).

Check the storage sub-system on the host and verify that it is configured for optimal performance. For information, see Troubleshooting hosted disk I/O performance problems (1008885).

Verify that there are enough free resources on the host to satisfy the requirements of the virtual machine. In VMware hosted products resources must be shared by both the host operating system and all running guests. For more information, see Investigating hosted virtual machine resources (1003848).

Disable the CPU power management features on the host. In some cases, these features can cause CPU performance issue with virtual machines. For more information, see Virtual Machine Clock Reports Time Unpredictably on Multiprocessor Systems (2041).

Verify that host networking issues are not impacting the performance of the virtual machine. For more information, see Verifying host networking speed (1009527).

Verify that the host operating system is working properly and is in a healthy state. When the host is not working correctly it may draw excessive resources from the guests.

 

Greeting,

Rachel Gomez