server 2008 r2 core network disconnects/drops often

I have a 1950 and a r510 both with server 2008 r2 core loaded. Both do exactly the same thing, the connections from RDP will drop every so often (few minutes) then reconnect. I have the hyper-v role installed on there and even the hyper-v systems drop the network connections. its very frustrating. I grabbed the latest Broadcom drivers from their site, does not seem to have helped. The systems are very vanilla, nothing much on them. Any other systems connected to the same switches do not exhibit this. In fact they are connected to different switches. The 1950 had server 2008 before, not core, and never exhibited those issuses. It had Hyper v and hosted 2 virtual servers just fine. Now I want to host additional systems and I figured Core would give it a smaller update footprint. I just can't make it reliable enough.  I bought the r510 hoping to do the same.

Anyone have ideas? Both are patched up to date.