I don't think the 1080 will work in that system. The following posts should help.

https://na.alienwarearena.com/ucf/show/203628

https://www.dell.com/community/Alienware-General-Read-Only/UPGRADING-ALIENWARE-M17X-R4/td-p/5537633

https://www.dell.com/community/Alienware-General-Read-Only/Upgrading-my-m17x-r4-graphics-card/td-p/5557699

Hi @M17xR4User,

For the M17XR4 the highest video card that has been validated would be an Nvidia GTX 680, unofficially the highest will be an Nvidia GTX 980.

Hello mates,

I have also a 2012 Alienware M17 R4 and it has an AMD Radeon HD7970M (Wimbledon XT) GPU in it.

What are the most up to date options for upgrading?
My preference would be to buy directly from Dell
but I am open to other solutions if there are none here.

Ideally,
I would like to get something in the range of the GTX-1080 or higher.

AMD List:
1. Radeon HD 8970M (Neptune XT).
2. Radeon R9 M295X (Amethyst XT).
3. Radeon R9 M395X (Amethyst XT).
4. Radeon R9 M485X (Antigua XT).
5. Radeon RX 5700M (Navi 10).

nVidia List:
1. GeForce GTX 680MX.
2. GeForce GTX 780M.
3. GeForce GTX 880M.
4. GeForce GTX 980.
5. GeForce GTX 1080.
6. GeForce GTX 1660 Ti.
7. GeForce RTX 2080 Super.

Confirmed update options for the M17x above the 680M and 7970M are;

GTX 770M, GTX 780M, GTX 860M, GTX 870M, GTX 880M, GTX 970M, GTX 980M, GTX 1060, GTX 1070

Radeon 8970M, R9 M290x

There are other cards like the Quadro M4000 and Firepro M4000M that can work as well. 

To run the 10 series Nvidia cards you will require an Unlocked  BIOS. You will need a modded Nvidia driver for any of the Nvidia cards above the 680M.

The RTX 5000 (quadro version similar to the RTX 2080) has been shown to work in the AW 18 which means it should work in the M17x R4 but I do not know if it has been tested.

Only the Zotac version of the GTX 1080 has the GPU die in the right spot for the Alienware heatsink. They are very rare and they require more power than the MXM socket can provide so they are heavily throttled. 

I have never heard of anyone trying and Radeon card higher than the M290x but if there are standard MXM 3.0b versions of the higher cards then they should work.