On regards to the memory the system will be able to support 32 GB running up to 1600 MHz and for the video card we just validate the followings:

Nvidia 460M

Nvidia GTX 560M

Nvidia GTX 580M

AMD 6970M      

AMD  6990M

You can try any other video card but I cannot assure that it will work properly. 

maybe some adjustments for the new graphic...i have motherboard

Manufacturer Alienware

Model M18xR1 (CPU1)
Version A05
Chipset Vendor Intel
Chipset Model Sandy Bridge
Chipset Revision 09
Southbridge Vendor Intel
Southbridge Model HM67
Southbridge Revision B3

I will recommend you to check the power consumption of the video card and maybe check in a modification forum to see if someone can provide you further information for your upgrade. 

You can go up to 32gb of Ram, I have corsair vengeance in mine, and makes quite a difference. 

Graphics wise, although not validated by Dell, plenty of people have SLI GTX 680M cards running, usually Clevo spec cards. Validated by Dell is SLI GTX 580M (easier to stay in same family of cards). 

Processor wise, the 2690XM (extreme unlocked) is the fastest your board supports.

An SSD if you don't already have one will offer some speed too.

Hi Cenobitez, based on your previous answer, I think you can help me with my upgrade. I have a M18XR1 and an Aurora R4. I want to replace electromechanical HDD for SSD either in my Aurora or my M18. 1 SSD (256GB) for system startup and 1 SSD (2TB) for data storage in each one of them. Also I want to expand RAM up to 32GB. Please can you help with the trademark and model of SSD and RAM parts?. I caanot provide current configuration for Aurora because of the dead HDD, but my M18X current configuration is Intel(R) Core(TM) i7-2670QM CPU @ 2.20GHz. BIOS: A05. L2 Cache: 256MB. RAM: 8GB (2GBx4). GTX 560M. I really appreciate your help. Regards.[;)]