Welcome to the Dell Community @majpogi 

This is what I have found.

You can upgrade to a Xeon Quad Core but you would need to replace the system board. "Soldered"

Dell Precision 17 (7710) Motherboards:

https://www.parts-people.com/index.php?action=category&id=142&subid=563&refine=motherboard

Maximum memory 64GB:

Dell Precision 17 7000 series (7710) Owner's Manual:

https://www.dell.com/support/manuals/en-us/precision-m7710-workstation/prec7710_om/technical-specification?guid=guid-c8ed19d1-97bb-441c-b0d7-49fd9d5ed312&lang=en-us

NOTE: Install either one, two, or four memory modules in the memory module slots to ensure optimum system performance. Installing three memory modules can lead to system performance issues.

The Quadro M5000M is a go.

NVIDIA Quadro Series Graphics Driver:

https://www.dell.com/support/home/en-us/drivers/driversdetails?driverid=514x3&oscode=wt64a&productcode=precision-m7710-workstation

You may need to replace the heatsink???

Dell Precision 17 (7710) Replacement Notebook Heatsinks In Stock Parts List:

https://www.parts-people.com/index.php?action=category&id=142&subid=563&refine=heatsink

Best regards,

U2