So just after I posted this I realized I have a USB 3.0 to ethernet adapter laying around so I tried that as well. Same issue.
This led me to the conclusion that the issue has to be software related and most likely somewhere in the Windows networking stack. With that in mind I took to Google and found this article: blogs.technet.microsoft.com/.../.
And sure enough, Receive Window Auto-Tuning Level was set to disabled. Setting it to normal solved the issue.