Did you do any research before attempting the upgrade? Alienware use fairly 'closed' system designs which means you can't usually upgrade GPUs to a next gen model.
I'll be surprised to hear you could upgrade past the 285m model.
Just to clarify. Yes, the GTX 560M is compatible with the Alienware M15X model.
You mentioned that all 3 LED lights are flashing:
The fans start and keyboard lights up, but the Scroll, Caps, and Num indicators are flashing.
This indicates that the problem is being caused by a faulty Motherboard according to the No Post LED Error Codes. Check the following article for more information.
I recommend you test the computer with the original GTX 260M video card. If the issue persists it means the Motherboard needs replacing.
Dell | Alienware Technical Support
Opening your case does not void your warranty! Check out Alienware Artificial Intelligence
I can't always reply to PM's, if you need extra assistance visit: