HyperV losing network connectivity

Question

Hi,

Like to seek help on our production machine Dell PE T620 (with tag =<ADMIN NOTE: Service tag removed per privacy policy>),

We might be having problems on the gigabit broadcom extreme network card or perhaps Windows.
We’ve setup HyperV on it:
1st VM running windows2012 R1 standard server Active directory server domain, DNS and DHCP services
2nd VM running legacy windows2003 standard server, for development/test environment & batch jobs
Both VMs are using the virtual switch manager to share the network connection.

Often times, We’re getting intermittent network problems on the VMs, ping test of very high latency (300ms-2000ms)
and RTOs to worst of NO connections on both VM1 and 2.
There are good days and bad days, worst case we had to restart VM1 and 2 to restore its operation and stability.
What could cause HyperV to loose network connectivity on the VM1 and 2?

The network (built in Intel Gigabit) on the bare metal OS seems to be stable, when the incident happens.

Another warning We're getting an exclamation mark (blue) on the DHCP service in the 1st VM,
but there are available IPs that are still usable.
Would appreciate any help.

Kelvin

APweb · Answer

Make sure you have up to date firmware for the Broadcom adapter and the most current drivers.

Early Broadcom drivers were horrible with 2012 R1, I used to experience very slow/intermittent operation with Hyper-V and very slow iSCSI data transfers (offloads did not work well at all and actually slowed things down further, I ended up disabling most of them to get some performance back).

I had a long running case open with Dell who were not particularly helpful as they do not provide the drivers themselves (packaged up as Dell updates, but the drivers are third party from Broadcom/QLogic it seems).

So if you are using old drivers start there, you may find the problem magically disappears. Depending on the Broadcom version you have, it may now be rebadged via firmware update to QLogic.

jkchoa · Answer

Ok, we've checked with our vendor and they gave us this link to update the driver, its currently on 16.2.0.4

http://www.dell.com/support/home/my/en/mybsd1/Drivers/DriversDetails?driverId=KW7KT

My colleague also found this link

https://support.microsoft.com/en-us/kb/2986895 saying about the drivers and the VMQ settings on HyperV. Should we change the VMQ or stick to the drivers update?

APweb · Answer

I've posted an update for you but probably because of the links to drivers I included it has been referred for moderation, hopefully it will appear soon. Edit: Now appears above!

APweb · Answer

Yes sounds like the problem you are experiencing.

Check firmware level as well:

www.dell.com/.../DriversDetails

I have seen some Broadcom driver updates spontaneously reboot the server half way through installation as well, so best to apply during a normal maintenance window/out of hours.

Of course it is up to you if you want to disable VMQ to see if the problem is resolved, but as I said previously the earlier Broadcom drivers were pretty poor performance wise so if it was my server, I would be looking to update the firmware and drivers.

There are more recent drivers/firmware for the NetXtreme than this, but they are not published under the T620 page as yet. Funnily enough I just applied them to an R730xd this morning. These were Network_Driver_TD70Y_WN64_20.2.0_20.02.04.01 and Network_Firmware_21DWR_WN64_20.2.17 respectively. Of course own risk, not recommended for production until officially published on the T620 page, etc.

You may also wish to consider a general update of firmware/drivers for the server using the update utility: www.dell.com/.../DriversDetails

Good luck!

jkchoa · Answer

Hi, I’ve installed the driver last Saturday and its version is now 17.0.0.3 but we’re still getting intermittent RTOs or destination unreachable on the HyperVs (chsidbdvm1 172.20.0.32) that uses the Broadcom netxtreme card. Should we go for the firm ware next?

APweb · Answer

Drivers and firmware go together with Broadcom/QLogic. When updating one you should update the other (normally in the description it will say that the firmware should be used with X version drivers).

jkchoa · Answer

I just installed the firmware recently.  HOpe this will solve the intermittent issue.

jkchoa · Answer

It seems my effort on fixing this issue is not paying off.  We're still experiencing intermittent on the HyperV that are using the broadcom netxtreme NIC.

PowerEdge Hardware General

HyperV losing network connectivity

Was this post helpful?