Highlighted
swade2569
1 Copper

PE T320 Hyper-V Guest latency fixed by BIOS update to Host

Jump to solution

Twice now I have had the Exchange 2010 guest on my W2K8 Standard Server with Hyper-V suddenly suffer a massive amount of latency and become practically locked up out of the blue.  After hours of troubleshooting, rebooting, and researching - as a last-ditch effort I noticed that the BIOS was out of date.  I upgrade BIOS and then everything magically works again.

Anywhere from 2-4 months pass and rinse & repeat. 

I'm going out on a limb here but I suspect that there's something about the system that gets flushed out during a bios update that is causing the problem, but I'm at a loss to determine what that is.  I want to proactively avoid having my customer's Exchange server fail, but it goes against my better judgment to apply BIOS updates proactively.  We've had the motherboard swapped out already due to a bad DRAC, and the problem keeps happening.

I could use some help in identifying what else I can do to prevent this from happening in the future.  TIA

0 Kudos
1 Solution

Accepted Solutions
LSLC
1 Copper

RE: PE T320 Hyper-V Guest latency fixed by BIOS update to Host

Jump to solution

Well we haven't seen the issue return at all since we changed the System Profile from "Performance per Watt" to "Performance".  I would start there first.  It's been three months and none of our T320's do this anymore.  We're 90% sure this is the solution.

20 Replies
Moderator
Moderator

RE: PE T320 Hyper-V Guest latency fixed by BIOS update to Host

Jump to solution

Hello swade2569

A BIOS update should flush cache and memory throughout the system.

Are you having issues with the host operating system or any other VMs, or is it limited to this one VM?
Are you using any technology that bypasses the hypervisor like SR-IOV?
When the VM locks up have you noticed if the memory or CPU utilization within the VM or host OS are maxed out?
Are there any errors in the hardware log of the iDRAC/BMC?
Any other info you can provide to help narrow down the cause?

Thanks

Daniel Mysinger
Dell EMC, Enterprise Engineer

Get support on Twitter @DellCaresPRO

0 Kudos
swade2569
1 Copper

RE: PE T320 Hyper-V Guest latency fixed by BIOS update to Host

Jump to solution

The host OS seems to run well enough.  I don't see a lot of latency there.  There are three VM's running.  A W2K3 server which has no problems, The W2K8 Exchange server which runs just awful when this happens and a W2K8 DC which spikes every 3-4 seconds but runs ok.

Not using anything like SR-IOV.

The memory on the Exchange VM is not maxed out.  I have it set dynamic and at worst it only uses about 2.5 GB.  The CPU however is running at about 90-95% (as indicated on a host perfmon session, not the VM's resource monitor).

No hardware errors are being registered. 

I suspect cache and memory are only going to be flushed on a BIOS update or complete shutdown, which isn't very helpful since this server is not readily accessible.  We reboot it once a month at minimum but that doesn't seem sufficient.

 

0 Kudos
Moderator
Moderator

RE: PE T320 Hyper-V Guest latency fixed by BIOS update to Host

Jump to solution

Troubleshooting issues that occur so infrequent are very difficult. I would recommend assigning more CPU capacity to the VM if it is maintaining close to 100% utilization. Also, read through this post. This person had a very similar issue that was corrected by changing the VHD to static rather than a dynamically expanding drive:

http://community.spiceworks.com/topic/318917-hyper-v-unresponsive-for-about-20-30-minutes

Thanks

Daniel Mysinger
Dell EMC, Enterprise Engineer

Get support on Twitter @DellCaresPRO

0 Kudos
LSLC
1 Copper

RE: PE T320 Hyper-V Guest latency fixed by BIOS update to Host

Jump to solution

We've had the identical problem you have with ALL of our Dell T320's,  We have 16 of them, so it happens infrequently on any single T320, but it comes up at least once every fortnight on average for us.  Since this happens to us fairly frequently, we have been able to learn a lot about the nature of the problem.  Hopefully what we've learned can help you.  We still haven't solved the problem, and Dell is completely clueless to the cause or even possible reason for this issue.

  • Reboots don't fix it
  • Electrically disconnecting and discharging the power supplies for over ten minutes solves the symptoms
  • Disconnecting the motherboard from the power supplies solves the symptoms
  • BIOS firmware updates solves the symptoms
  • Power supply firmware updates solves the symptoms

By "solve the symptoms ",I mean it stops the sluggish performance and Hype-V VM's return to normal performance.   However in a month or two, the problem comes back randomly.  Unfortunately every time we apply a BIOS update, it's like burning a match, and we're running out of BIOS updates to apply.  We're already at 2.0.22 on all of our servers.  The last thing we're in the process of trying is the system profile.  If you F2 into the BIOS and select System Profile, change the profile from Performance per Watt to Performance.  It solves the symptoms, and so far we haven't had any servers that have this setting changed go symptomatic on us.  We're trying to do good science and have a control group and changed group.  We have only been trying this for about a month.  I'll report back in four months to let you know if this is an actual fix.

rmcompsol
1 Copper

RE: PE T320 Hyper-V Guest latency fixed by BIOS update to Host

Jump to solution

Having similar issues! Have you determined if changing the System Profile from "Performance per Watt" to "Performance" fixes the issue?

LSLC
1 Copper

RE: PE T320 Hyper-V Guest latency fixed by BIOS update to Host

Jump to solution

Well we haven't seen the issue return at all since we changed the System Profile from "Performance per Watt" to "Performance".  I would start there first.  It's been three months and none of our T320's do this anymore.  We're 90% sure this is the solution.

rmcompsol
1 Copper

RE: PE T320 Hyper-V Guest latency fixed by BIOS update to Host

Jump to solution
Just thought I would include more specifics about my issue in case others find this thread: Dell PowerEdge T320, Xeon E5 @ 2.4GHz, 16GB Memory, 4x600GB SAS 15K, RAID 10, Small Business Server 2011 So far this system has been hit a few times by the "latency" issue described in this post. Out of nowhere the server seems to grind to a halt, as if the CPU is running at 100MHz instead of 2400MHz. The system still responds to commands and runs programs; just at a snail's pace. As mentioned in a previous post by LSLC, Rebooting the server does NOT fix the issue! However, I can confirm that installing a new BIOS update alleviates the symptoms (currently running 2.1.2). I also followed the recommendation to change the BIOS System Profile from "Performance per Watt" to "Performance". I'll be monitoring this to see if this is a long-term fix (perhaps the latest BIOS update fixes the issue).
0 Kudos
Onyx3821
1 Copper

RE: PE T320 Hyper-V Guest latency fixed by BIOS update to Host

Jump to solution

Hey guys, I'm having the SAME symptoms on a T320 a customer of ours runs.    We're running the Hyper-V Core on this with several guest VMs.   When there's a freeze up, rebooting doesn't fix the problem, either.  I don't see a relief on these symptoms until going in through the DRAC and doing a power off/power on.   I'm wondering if all you guys that are saying that the BIOS upgrades fix this are just seeing a coincidence of a power reboot freeing up the resources.

I'd like to hear if LSLC has seen the problems are still gone after all this time.

0 Kudos
LSLC
1 Copper

RE: PE T320 Hyper-V Guest latency fixed by BIOS update to Host

Jump to solution

I absolutely recommend changing the System Profile to "Performance"  as your first step in solving the problem.  We have had no further issues with our T320's after doing this.  BIOS upgrades only temporarily fix the problem, so does draining the power supplies.