Unsolved

This post is more than 5 years old

4 Posts

26960

January 31st, 2013 03:00

R510 SAS RAID 10 Hosting Hyper-V poor disk speed

Setup:

R510 PERC H700: 8 virtual disks in total. System drive\OS on RAID1, other 7 VD on 4 SAS 600GB as RAID10

Read Policy: Adaptive Read Ahead

Write Policy: Write Back

Disk Cache Pol: Disabled

OS: W2K8 R2 ENT, Hyper-V + failover clustering

Each VM hosted in separate volume on RAID 10, SIOS DataKeeper Cluster Edition replicating the 7 volumes to two identical R510's

VM's: One Exchange 2010, one SQL2005, Citrix Server, File and Print (Os mixes of W2K8 and W2K3 64 Bit)

Problem:

I am seeing very high average disk queue lengths in all the 7 VM's (maxing at 100 in performance monitor) whenever I try and do large(ish) file transfers (2Gb plus). Other symptoms: Exchange VM slows right down if anyone emails attachments of +7Mb. File and Print Server, long time to enumerate folder structures etc... I've tried updating BIOS, RAID f/ware Broadcom network adapters (against my better judgement I've got a Broadcom LACP team upto my switch for VM LAN access). I'm also seeing pretty sluggish throughput with IPERF from other LAN computers into the Hyper-V (R510's).

Kind of running out of ideas here, any help appreciated

12 Elder

 • 

6.2K Posts

January 31st, 2013 09:00

Hello gordyr

other 7 VD on 4 SAS 600GB as RAID10

Is this a typo or do you have 7 RAID 10 virtual disks sliced across 4 HDDs?

How many users are there?
In your testing is this during production hours or off peak hours when no one is using the server?

against my better judgement I've got a Broadcom LACP team upto my switch for VM LAN access

Could you explain in more detail how you have this set up? Is the cluster traffic on the same VLAN as the regular traffic? Is it configured as a LAG or did you use the Broadcom Advanced Control Suite to create a team? Do you have one NIC that all of this traffic is going across?

Thanks

4 Posts

February 1st, 2013 00:00

Hi Daniel

7 VD= 7 virtual disks, spanned across 4 physical disks configured as a RAID10 (each VD is a Windows volume that hosts it's own VM)

^ I'm guessing this is where my bottleneck is in terms of disk I/O

How many users? 15 tops

Broadcom BACS4 configured LACP team, upto an Extreme XOS switch with ports configured as a L3 LACP port team.

The server itself has 4 nics (2 broadcom, 2 intel) the intel nics are configured on a different subnet for exclusive use by the underlying volume replication app "SIOS DK Cluster", the broadcom team I am using for cluster vm migration and LAN access

12 Elder

 • 

6.2K Posts

February 1st, 2013 09:00

7 VD= 7 virtual disks, spanned across 4 physical disks configured as a RAID10 (each VD is a Windows volume that hosts it's own VM)

Okay, you have 1 RAID 10 virtual disk with 7 windows volumes/partitions. That is fine.

Broadcom BACS4 configured LACP team, upto an Extreme XOS switch with ports configured as a L3 LACP port team.

I think this is your problem. You have a Link Aggregation Group/Port-Channel/Channel-Group created on the switch, and you have a team created in BACS. You should configure one or the other. If you have a BACS team and a LAG then they will conflict and cause a lot of packets to be improperly routed and resent.

LAGs and teams function differently. A LAG will not spread a data stream across multiple ports, but a team will. If exchange is pushing through a 7MB attachment then the LAG sees that as a single data stream and will only use one port. If you have teaming configured as well then teaming will attempt to spread that traffic across multiple ports. The switch will not accept the packets sent on the other ports by the team, and they will be resent over and over until both the LAG and team pass the data in a method both can agree on.

The other concern I have is with multicast traffic from the cluster. Even though the cluster traffic is segregated on a different subnet it may still multicast across the switch. Is the cluster traffic on its own VLAN?

Aside from the team in BACS I think your issues lie with traffic segregation on the switch. If it is a Dell switch then I would suggest creating a post in the networking forums. If it is a 3rd party switch then I would suggest contacting the vendor on how to configure for multicast traffic. I know that some of our switches are not able to properly handle multicast traffic.

Thanks

6 Operator

 • 

1.8K Posts

February 2nd, 2013 15:00

TO add...

Teaming will not increase throughput from a single client, only helps with multiple accesses from multiple clients. On the NIC interface unless your using IP6, disable it. also with your low number of WKS, your likely to be better off with disabling flow control in the NIC properties. Have you run chkdsk? Turn off SMB signing.

Have you read this?   tweak carefully....

http://msdn.microsoft.com/en-us/library/windows/hardware/gg463392.aspx

 

4 Posts

February 6th, 2013 01:00

Thanks for the suggestions Daniel, I'll look at my BACs config. I have had some feedback on the MSDN forum re: 4 spindle RAID 10 with 7 VD not being an efficient config for Hyper-V, I may take a look at testing the VM on a RAID1 to see if I can improve disk I/O

12 Elder

 • 

6.2K Posts

February 6th, 2013 09:00

I have had some feedback on the MSDN forum re: 4 spindle RAID 10 with 7 VD not being an efficient config for Hyper-V

I think there is some confusion regarding terminology. Yes, 7 Virtual Disks/Sliced arrays is bad for performance. Based on what you described I don't think you actually have 7 Virtual Disks/Slices. I think you have 7 partitions within windows on a single RAID 10 virtual disk. There is a huge difference between 7 virtual disks and 7 partitions. Virtual disks are created on the RAID controller. Partitions are created inside the operating system.

Thanks

4 Posts

February 7th, 2013 00:00

no it's 7 VD's configured at the RAID utility, not 7 partitions configured within Win

No Events found!

Top