sounds like the psu is dying although not completely dead yet.  You can use a spare psu to test the board and if all of a sudden the power button springs to life, that proves it.  Another thing to do w old dell is to replace cmos battery w a brand new cr2032.

Amazon has PSU testers.  Caveat being you would probably need 8-pin female to 24-pin male adapter to use it.  Dell has proprietary 8-pin MB connection in the 9020.  24-pin end would plug into tester and 8-pin end to PSU.

First, like redxps630 said, change size 2032 CMOS battery.  Please let us know if anything changes.

Many thanks for replies!

All right, shall start with battery change... however, changing any hardware of PC I just bought recently doesn't seem right to me. Ok this PC wasn't new, but how come PSU may get broken at all? This model isn't very old, and I don't do mining, or other extreme.

Oh, and on more thing. One of 10 times PC starts normally, for some reason. Does it give any additional  clue of what might happen?

9020 released in 2013 is not a new model.  You bought it recently but its stock psu is almost 10 years old.  Psu failing has nothing to do w mining or heavy duty use.  Even if it sits on shelf unused it may fail eventually due to internal component degeneration.  That is a quality factor.  Oem psu are not designed to last 15-20 years.  Even name brand psu only offer 5 year warranty.

A dying or failing psu typically can still power on system one or two times out of 7-10 trials.  The fact it occasionally works but is getting harder and harder is telltale sign of dying psu.  Often this occasional working condition fools user into believing that psu is still healthy or good.  I had a Dell pc that user told me the power button was finicky, he had to press multiple times to turn on pc.  Turns out psu was failing. After replacing w another used but good psu, pc works like a charm.

Thanks for reply.
Yeah maybe you're right. I'm not into PC hardware much. 
My Dell has another issue, though - graphic card's (AMD) display port doesn't work well. Gives me glitches with monitor. Had to replace it to old VGA, lost monitor sound. But no glitches. Thing is display port cable itself is OK - I switched it with another PC and it does work ok there. So, display port is broken too. Something's wrong about it all...

Just in case if I have to replace PSU. Which one do I buy? Should I contact local Dell store, or any used PSU will do fine? Guess it has to be SFF size, right? And no idea about power (watt?), either...

A cheaper test now is replace cmos battery (cr2032 $2) then clear cmos settings by motherboard RTCRST jumper.  see motherboard diagram. Move blue jumper from passwd to RTCRST x 5 sec then move back (pc power should be unplugged).

RTCRST

If no change, then replace psu. Dell oem psu for 9020 sff available on eBay (used).  255w $12.85 https://www.ebay.com/itm/255114067341?

Thanks a lot for this detailed explanation and pics! Really appreciate.

All right, shall try battery first.

Very important to know what GPU are you using in case it's overloading the PSU.  Have you tried multiple starts with your PC without the GPU installed and using onboard graphics?

Indeed, though said already, replace battery first.  Places that sell refurbs and used often don't replace the battery.  I had to replace the battery the first on one refurb I bought.

Thanks for advice!
I'll try the battery first and then probably w/o GPU. But  though that if I have AMD graphics - that means I do NOT have onboard GPU (Intel)?

See the manual-- Support for OptiPlex 9020 | Documentation | Dell US

That model comes in diff sizes so choose yours. It does have integrated video but the video card must be physically removed from the board to use the Intel integrated.