Hyper-V High Availability Cluster Fails when a single Switch Fails

We have the following kit configured as a two-node Hyper-V High Availability Cluster on Windows Server 2008 R2.

• 2 x Dell PowerEdge R610s
• 2 x Dell PowerConnect 6224 Switches, with switch interconnects
• 1 x Dell EQL-PS4000XV iSCSI SAN Array

Each of the R610s has 10 NICs available to it, spread as follows:

• 4 on motherboard (Broadcom - purchased with machine)
• 2 on add-in card (Broadcom - purchased with machine)
• 4 on add-in card (Intel - added at a later date to increased NIC count)

NIC ports are dedicated as follows on each of the machines:

• 1 x Management Network (Cluster Use: Enabled)
• 2 x iSCSI Network (using MPIO) (Cluster Use: Disabled)
• 2 x VM Network (bound together using a NIC team) (Doesn't Show in Failover Cluster Manager)
• 1 x Cluster Heartbeat / Private Comms (Cluster Use: Internal)
• 1 x Live Migration Traffic (Cluster Use: Disabled)
• 1 x Clustered Shared Volume Traffic (Cluster Use: Internal)

Before deploying the cluster to production I have been simulating the failure of various aspects of the infrastructure.

The problem that we have is that the cluster will not survive a failure of one of the switches.

I believe that it is not always the same switch that causes the failure, but relates to a failure of the switch coinciding with the node that is currently owning the cluster name and IP address resources.

The failure ultimately ends up with one node being removed from the cluster, with the following events showing in the event log:

• 1177 - The Cluster service is shutting down because quorum was lost.
• 1129 - Cluster network 'MGT' is partitioned.
• 1129 - Cluster network 'CHB' is partitioned.

From this I deduce that the cluster has "given up" because it can no longer communicate on either the management (MGT) or Heartbeat (CHB) networks.

At present the MGT and CHB NICS are wired up as follows:

• Node 1
  • MGT -> Switch 1
  • CHB -> Switch 2
• Node 2
  • MGT -> Switch 2
  • CHB -> Switch 1

However, this seems to result in each network becoming partitioned, causing the cluster to fail when the switch fails.

At present, there is no routing between the subnets used for MGT and CHB networks.  The CHB is an isolated subnet purely for use by the cluster.

We saw a similar problem in our earlier testing, whereby live migration (CLM) and Clustered Shared Volume (CSV) networks would become partitioned when configured in the same way.  To resolve this, we moved all CLM NICs to sit on one switch, and all CSV NICs to sit on the other. 

This means that a whole network was lost when a switch fails, rather than becoming "partitioned" - which the cluster seems to be happier with!

Anyway, after setting the scene, my questions are as follows:

• Is there any best practice guidance for NIC to switch wiring for a Hyper-V HA cluster? (e.g. should all NICs on a given cluster network be connected to the same switch?)
• Should we re-locate the MGT and CHB networks so they each sit on a single, separate switch - as we did with the CLM and CSV NICs?
  • If so, how do we manage the cluster if both MGT interfaces are on the switch that has failed?
• Would there be any merit in configuring routing between the MGT and CHB networks, and adding a default gateway to the CHB network? My hope is that this may permit communication to continue in the case of a failure on one switch. However, I thought only one NIC should contain settings for a default gateway, which is why we have not set this up already.

Any thoughts or advice on the above would be greatly received.

Thank you for your time.

Roland