Orange lights on power on (board problem)

sounds like the psu is dying although not completely dead yet.  You can use a spare psu to test the board and if all of a sudden the power button springs to life, that proves it.  Another thing to do w old dell is to replace cmos battery w a brand new cr2032.

Amazon has PSU testers.  Caveat being you would probably need 8-pin female to 24-pin male adapter to use it.  Dell has proprietary 8-pin MB connection in the 9020.  24-pin end would plug into tester and 8-pin end to PSU.

First, like redxps630 said, change size 2032 CMOS battery.  Please let us know if anything changes.

Many thanks for replies!

All right, shall start with battery change... however, changing any hardware of PC I just bought recently doesn't seem right to me. Ok this PC wasn't new, but how come PSU may get broken at all? This model isn't very old, and I don't do mining, or other extreme.

Oh, and on more thing. One of 10 times PC starts normally, for some reason. Does it give any additional  clue of what might happen?

9020 released in 2013 is not a new model.  You bought it recently but its stock psu is almost 10 years old.  Psu failing has nothing to do w mining or heavy duty use.  Even if it sits on shelf unused it may fail eventually due to internal component degeneration.  That is a quality factor.  Oem psu are not designed to last 15-20 years.  Even name brand psu only offer 5 year warranty.

A dying or failing psu typically can still power on system one or two times out of 7-10 trials.  The fact it occasionally works but is getting harder and harder is telltale sign of dying psu.  Often this occasional working condition fools user into believing that psu is still healthy or good.  I had a Dell pc that user told me the power button was finicky, he had to press multiple times to turn on pc.  Turns out psu was failing. After replacing w another used but good psu, pc works like a charm.

Thanks for reply.
Yeah maybe you're right. I'm not into PC hardware much. 
My Dell has another issue, though - graphic card's (AMD) display port doesn't work well. Gives me glitches with monitor. Had to replace it to old VGA, lost monitor sound. But no glitches. Thing is display port cable itself is OK - I switched it with another PC and it does work ok there. So, display port is broken too. Something's wrong about it all...

Just in case if I have to replace PSU. Which one do I buy? Should I contact local Dell store, or any used PSU will do fine? Guess it has to be SFF size, right? And no idea about power (watt?), either...

A cheaper test now is replace cmos battery (cr2032 $2) then clear cmos settings by motherboard RTCRST jumper.  see motherboard diagram. Move blue jumper from passwd to RTCRST x 5 sec then move back (pc power should be unplugged).

RTCRST

If no change, then replace psu. Dell oem psu for 9020 sff available on eBay (used).  255w $12.85 https://www.ebay.com/itm/255114067341?

Thanks a lot for this detailed explanation and pics! Really appreciate.

All right, shall try battery first.

Very important to know what GPU are you using in case it's overloading the PSU.  Have you tried multiple starts with your PC without the GPU installed and using onboard graphics?

Indeed, though said already, replace battery first.  Places that sell refurbs and used often don't replace the battery.  I had to replace the battery the first on one refurb I bought.

Thanks for advice!
I'll try the battery first and then probably w/o GPU. But  though that if I have AMD graphics - that means I do NOT have onboard GPU (Intel)?

See the manual-- Support for OptiPlex 9020 | Documentation | Dell US

That model comes in diff sizes so choose yours. It does have integrated video but the video card must be physically removed from the board to use the Intel integrated.

Just because an AMD GPU is being used doesn't mean your 9020 doesn't have onboard graphics.  Similar to what MaryG said, removing graphics card will activate onboard graphics.

Changed the battery, at last.
Unfortunately, did not help. Same problem.
Also, although I am really grateful for clear and detailed answers here, I do wish I was told one more thing - that my BIOS settings will be reset and PC won't start after that. Took me quite some time to find out that happened because HD mode got changed to RAID. Turning it back to AHCI started my PC at last but had to worry. No idea why HD mode changed at all but worth mentioning.

Since the problem remains I only write to update, and to prevent topic from automatic closing (if that happens here)
Next step I am going to do is to try internal graphic card only. ATI card doesn't seem to work ok anyway.
And then probably power unit.

Tried to remove AMD graphic card.
Problem with PC start/orange error lights remains.
But now I have another problem - no sound.
AMD card was connected via VGA cable, so I heard the sound coming from PC itself. Now internal graphic card is also connected via VGA cable - but no PC sound anymore, at all.
Sound setting of Windows 10 shows everything works ok, while fix utility shows some "universal audiodriver" error in the end which cannot be fixed of course.
How do I get the sound work? I want to run system with internal graphics for some time, for some reasons.
Please help. Thanks!

Sorry for all the problems you're having with the 9020.

It might be worth it to reload Win10.  Who knows how Win10 came if the PC was bought used.

Also, I haven't heard about it from others, 3 Optiplex's and I've never had BIOS change to RAID when changing the battery.  But I do understand the frustration and glad you found it.

Also, which jacks and speakers are you using to try to get sound from the PC?  Sound doesn't travel through VGA output.