Thanks for your fast reply!

The motherboard has no HDMI/DP Port and I tested both DP and HDMI on the OEM Dell 3060Ti. There is no difference between Dell in general and Alienware cars I hope? 

What might be interesting is that on the RX580 I didn't notice the fan spinning, while on the A770 I at least noticed the RGB but didn't look for the fan. On the 3060 I saw both RGB and the fans working, so the connection seems to work. 

The rest of the system is completely OEM, it has 64GB (4x16GB) HyperX.

This is a typical issues with video output. As long as you have an OEM Dell card it should show video output, as long as you use the video output from the video card and not onboard video output of the motherboard.

It's also possible there's something else wrong with the system. Hard to say without further troubleshooting.

Memory can be an issue on these systems. Did it come with the OEM memory, or retail aftermarket memory?

Good general advice is to never purchase a system without seeing it boot properly.

You can try removing and reinserting the coin cell battery and see if that makes a difference.

Any blinks on the Alienware head (power button)?

Good point, I will try that.

Yes, it's the POST code that I mentioned. 7 blinks, but only on every first startup after unplugging it from the wallplug. After that there are no codes until I unplug it again. 

7 blinks means CPU failure afaik, so maybe the capacitors on the mainboard that power the CPU are toast? I can't really imagine that the CPU itself would die from not being used.

I would put in a brand new CR2032 coin cell battery since the rig has been sitting for a while. They are very inexpensive (relatively speaking). Once you get video with a Dell OEM graphics card . . . if you want to use a 3rd party graphics card, try disabling Secure Boot in BIOS.

Just tried a brand new cell after pulling the old one and waiting for about 1hr. No changes unfortunately.

With the exception of the CR2032 battery, nothing in your R11 is going to die from lack of use, including PSU, CPU, and capacitors. Whomever sold you the R11 is providing bad information. With power disconnected; take out the CR2032 battery and hold in the power button for 30 seconds to discharge flea power. Make sue the Dell OEM graphics card is securely installed and locked in the slot. Insure that all RAM modules are securely installed and locked in their slots. Check that the EPS 8-pin CPU power cable is secure in the motherboard header . . . also the ATX 24-pin motherboard power cable. Check that all cables are securely connected at the PSU end. There is no telling what the original owner did when they ripped out the RTX 3090. Plug in and power up . . . good luck!

Try all 3 display port outputs. If the computer can't get into windows and load drivers, only one may work.

Alright, I somehow lost access to my old account, but just to give you a quick heads up:

The problem was the mainboard itself.

As soon as I switched everything over to another mainboard everything worked perfectly.

Since I didn't feel like cutting out the I/O shield and figuring out the proprietary harness I build a MiniITX case now and I'm happy with everything!

10900KF, 3060Ti, 2TB NVMe, 2TB HDD, 32GB RAM (with another 32GB currently spare/to sell), all in an tiny formfactor for around 900 bucks, so I don't really feel scammed.

Thanks anyways for all your suggestions!