CPU - dead end;  anything you can install will be a slight upgrade at best (and you're limited to the third generation CPUs, so they're all on the order of a decade old).

There's a GPU upgrade discussion here

https://www.dell.com/community/Alienware-General-Read-Only/m17x-R4-Graphics-upgrade-to-New-Gen-Graphics/td-p/5573165

As for drives -- any 2.5" SATA SSD will be quite an upgrade from your current hard drive.

Max RAM is 4X8 = 32 G.

CPU - i7-3920XM or i7-3940XM (the 3920XM is usually better value and tends to overclock better)

GPU options (GTX 980M, GTX 1060, GTX 1070, Quadro RTX 3000, a few others). The 120hz display is preferred for the 10 series cards

RAM - 32gb DDR3L 2133Mhz

Storage - Dual 2.5" SSD's in RAID 0 will get the best performance. You also have an mSata SSD slot for extra storage.

Thank you for your suggestion. I appreciate it.

Thank you for your suggestion. I will look for the suggested parts online and hope it is within my budget. Appreciate your help.

Just note that there are different form factors with some GPU options. You really need to find the 'standard' MXM 3.1 Type B card like the GTX 1070 pictured below.

Some manufacturers have a slightly different form factor and while they can still work, you may need to modify the laptop chassis and possibly the heatsink to get to to physically fit. See image below of an MSI GTX 1070

These cards can still be quite expensive so doing the upgrade is often not worth it. They really need to be paired with one of the top 3rd gen mobile CPU's and these can also still be expensive. You really need to love your old laptop to drop this sort of money on older tech. Some of us just enjoy it so the cost isn't such a big deal. The M17x R4 was a fantastic laptop. 

I'd suggest heading over to the NotebookReview forum if you want to learn more about upgrading these older machines. 

Sorry to be late to the party with this thread but I have been upgrading my M17xR4 for over a year now and have hit a dead-end GPU-wise. Your posts have been super helpful so I thought I'd give it a shot messaging you for advice. Also, sad days that the Notebook review forum has closed down! There's so many useful articles about these issues I can't access anymore.

I've tried 3x GTX 980m GPUs bought through eBay from China and they all experience this problem or something similar when inserted into my Alienware and Windows detects that it's a 980m in Device Manager. They've all been returned for a refund but I'd really like to upgrade from my 680m to something with more VRAM / a little more power. 2GB just isn't cutting it.

My current specs:

Alienware M17xR4: 1TB 2.5inch SSD(was 2x in RAID 0 but used second SSD for another PC), 2TB HDD, 32GB 1600mhz DDR3 RAM, Intel Core i7-3630QM, NVIDIA GeForce GTX 680m 2GB.

I love this laptop so much because it's so fix-able and upgradable. But the GPU issue has been causing me major problems. I hope I can get something above a 680m.

@NNS Max The 980M's from China are hit and miss. Apparently the are clone versions of an OEM 980M. I believe that they but the 980M die in bulk and the PCB's are made by some backyard guy so the quality is very poor. You should have more luck trying to source one locally. 

Another option is the 970M, yeah it's not as good as the 980M but most of them have 6gb vRAM so it's much better than your 680M. The 1060 or 1070 are then your higher options. Again, try to find these locally rather than from China.

Also, now that NBR has closed down there is NotebookTalk.net

This forum was born from the ashes of NBR. They also have a comprehensive archive of the old NBR threads so the old information isn't completely lost. 

Thanks for the reply, that's all really useful info!.

I just installed an 880m into my laptop and it won't power on anymore. If I plug the power cable in, the blue ring on the power brick connector turns off any the laptop appears dead. After re-installing my 680m, the problem went away and the laptop powers on again. Do you have any idea what might be causing this? Or do you think it's just a dead video card?

@NNS Max It is 100% a short on the 880M. This is a really common issue with this card (I've had at least 3 that have done the same thing). The motherboard detects the short and trips the power supply to protect itself. Someone who knows what they're doing would be able to use a multimeter to find the shorted component on the 880M and replace it but that is beyond my skills unfortunately. 

@mattyb3​ I noticed you said you can add a mSata SSD, I have looked everywhere trying to search for it on my 2013 Alienware M17xR4 model but can't seem to find the slot. I'm assuming i'd have to do a pretty deep teardown to find it. I would appreciate it if you could reply letting me know where this slot is located.