Well, I'm not sure what the most recent generation of cards is requiring, but I can say that an nvidia 1080 is likely to pull too much power in *some* games.
I installed one in my x51 and it was great until I went to play some conan exiles. That game seems to randomly do something that causes the card to draw too much power for the system to keep up, which causes the system to randomly reboot.
Something that I didn't find obvious, is that on the cards like the 1080 that I have, is that the card will actually overclock itself *higher than the advertised specs*.
In other words, if it says on the spec sheet says that the card will run at a max gpu speed of 1000mhz and the memory will run at up to 2000mhz, then that is actually what the card will *usually* clock/overclock itself up to.
However, in some situations, if it isn't overheating and whatever various variables and conditions are met, then it may clock itself *beyond* those specs if it feels like it should, for whatever reason.
I was initially quite alarmed when I started trying to find out why my system was rebooting, and started looking at some logs generated by gpu-z, because according to my logs, the card kept zooming up past what was supposed to be it's limit. (i was worried about overheating because I had not realized that I should have bought a blower style card) but it wasn't overheating during those times, so I started looking into it and after doing some research I learned that it's somewhat normal for that series of cards.
However, that introduces the problem that the card can speed itself up enough that it draws more power than advertised as well.
And apparently, the 300watt power supply just couldn't provide enough amps to keep the card happy in all situations, and 3 or 4 for of my games (conan exiles, and I *think* ark survival evolved, and 1 or 2 others) happened to cause those situations to occur.
Other games, even ones that seemed much more graphically intensive, didn't cause reboots; it was only a select few that caused the problem.
Anyway, I considered doing some DIY engineering to take care of the problem, but in the end I ended up just buying one of the "graphics amplifier" boxes to attach to my x51.
It's basically an external case for your video card, that has it's own dedicated power supply.
As an bit of an added bonus, the graphics amplifier also happens to have some extra usb ports, so, win-win
Anyway, I haven't noticed any slow-down or anything from having the card mounted in the external box, and I've had no power related problems whatsoever with the system since then.
And, as another couple of bonuses, without a big powerful *and HOT* video card inside the main system, the x51 runs cooler, and has enough space to mount an extra ssd in the area where the video card would usually sit.
So if you like the x51 but want to put a modern high end graphics card in it, I'd suggest getting a graphics amplifier so you don't have to deal with worrying about all the potential power delivery and heat issues.
Anyway, whatever you choose, take care and good luck

GTX-1060 is the usual best one. I've read users trying GTX-1070, but sometimes it over-amps and x51-r3 shuts-down during hot-and-heavy gaming.

This setup is done. It is really too power-limited, and cooling-space limited ... to take it any further.