What CPU's can I upgrade my Alienware X51 with?

Not quite sure which revision do you own but I will list the processors compatible for each revision:

X51R1

I3-2120 3.3GHz

I5-2320 Quad Core, 3.0GHz

I5-3450 Quad Core, 3.10GHz

i7-2600 Quad Core, 3.4GHz

I7-3770 Quad Core,3.4GHz

X51R2

I3-4130 3.4 GHz

I3-4150 3.5 GHz

i5-4430 Quad Cores, 3.0 GHz

I5-4440 3.1 GHz

I5-4460 3.2 GHz

i7-4770 Quad Cores 3.4 GHz

I7-4790 3.6 GHz

How do I tell if mine is R1 or R2?

You can check it in the order number, in the system information or in the BIOS.

If you don’t find it send me a PM with your service tag and brief description so I can check if for you.

You most likely an R2 since the i7-3770 didn't work. But checking in the BIOS is not a bad idea to be safe.

Or inputting your service tag/express service code onto Dell's support site.

The R1 uses an LGA1155 socket while the R2 uses an LGA1150 socket - he would not be physically able to install the i7-3770 (LGA1155) in the R2 - he probably has the R1 and just needs a BIOS update.

Seem as if googling the motherboard already revealed the answer, but I didn't know this bit about the physical socket being that different.

The motherboard is one that supports LGA 1155 sockets. 

I was thinking that too, but I didn't know if he had the X51 running with a CPU already in there.

The physical bit is something that slipped my mind. Better to keep it in mind.

I already have an i3 2120 in my motherboard so it must be an R1 PC. What would make the cpu not work.

Having an old BIOS. All you need to do is update the BIOS.