Upgrading to 8GB is definitely wise with Win10.  You'll probably see an improvement in speed as well.

Here's a link to Crucial accessories for your PC:  https://www.crucial.com/compatible-upgrade-for/dell/optiplex-5250-all-in-one 

For the most compatible RAM try the Crucial site here. And I would suggest buying an 8GB kit. That includes 2 matched RAM modules.

EDIT: Brad types faster than these old hands.

Optiplex 5250 is AIO with 2 DIMM slots accepting laptop sodimm memory.  Agree with JOcean that most elegant way of upgrade is matched sticks. But if money is a concern, get another ddr4-sodimm of same speed though different brand will work too.  I would just pop in another 4gb if I happened to have one lying around. Yes it was quick n dirty but you would hardly know any difference.

Two SODIMM slots; Non-ECC dual-channel 2400MHz DDR4 SDRAM (2133MHz performance on Intel 6th generation processors)