So as I was able to find, installing GPUs will require the riser and the kit and the whole nine yards.

I got the 1100W PSUs, I updated bios, updated iDRAC to latest versions.

Installed GPU and used the riser power cable to do it. I enable power to the machine and there it is, blinking amber light. If I disconnect the power from the GPU, even without completely removing it, the system works just fine. Power back to GPU? Machine won't even start. iDRAC is still accessible through the network but that's about it, it just refuses to boot and nothing regarding the issue shows on any logs. 

When I go to power configuration, the system is capped at 461 watts for some reason. The input wattage is over 1200w and output (of PSUs) is 1100w. There is more than enough for them to work properly. 

Why is power capped at 461 when I have two 1100W PSUs? 

I reset iDRAC to default twice and it is still capped at 461 and the machine refuses to boot only when GPU power is connected.