Disable all wake events in BIOS and Windows, check front panel wiring, and test with minimal components.

Disabled wake on usb and wake on lan, is there additional wake events?

Checked front panel wiring, looks complete.

What is that the 'Deep sleep' mode? could it affect anything? it is turned on for S4 and S5 power state

I've recently upgraded the GPU to Nvidia RTX 5060 Ti, could it be the issue?

Re: When connecting the PSU to AC, or when shutting down Windows, the PC wakes for a few seconds

this is an idiosyncratic psu-mobo behavior seen with some Dell pc. It can be roughly characterized as confused logic communication btn board and psu. I have seen two XPS n multiple Optiplex do that. imo this has nothing to do w Windows sleep setting. To prove the point you can remove boot drive and still observe same thing.

I've disabled the 'deep sleep' mode in the bios settings and for now things looks fine!

Any idea why that setting affected that phenomena?

also, any idea how much more power the PC will consume when it is turned off with that setting disabled?

Sometimes a malfunctioning motherboard.

Sometimes a small (now under-powered) power-supply. When you "replaced Power Supply" did you install another Dell one? What wattage is it?

Sometimes these machines have multiple ones available, and people like to install the small one for some reason. Or, with some minor upgrades, it pushed the machine over the limit it was barely meeting before.

@ValorHeart​ ,

1. I've used genuine dell power supply for that system,

2. i have both 200w and 500w model,

3. when I've used the lower wattage one, I've removed the GPU.

4. I believe that issue started since i've installed the Geforce RTX 5060 Ti, it doesn't seems to work very well with that motherboard,

5. but since i've turned off the 'Deep sleep' BIOS option, the issue does not repeat, so i guess it is compatibility issue rather than hardware issue

1. Good.

2. Just keep the Dell-OEM 500 watter in there.

3. Troubleshooting, got it

4. Yeah, 500w still leaves little to spare when a RTX 5060 Ti (almost a RTX-5070) hits 80%-90% utilization.

The GPU is rated at ~180W and recommended PSU is 600W. 

We usually run 850-watters in systems configured like that.

5. Oh good. So, issue is resolved? Or, you are still testing?

With the Dell-OEM-500w PS installed and the RTX 5060-Ti removed (just using on-board video) the computer should be able to turn-on completely and Post/Boot every-time reliably.

If not, you just keep trying to fix this simple-config until it does. Or, you give-up trying to fix this (now) 5-year-old computer. It's cool it's certified to run Windows-11, but it's almost classified as "Legacy" now-days. 

@Tesla1856​ So the issue is resolved, unfortunately I don't understand the root cause for the issue, but from reading in the internet, it seems like the newer GPU's vBios doesn't communiate well with older mother boards, which can explain the reason the deep sleep mode is being disturbed.

Thanks!