Make sure your power supply has been upgraded to the minimum requirement for that video card. Optiplex computers are for business not gaming so come with a small power supply. There are 4 diff models of the 780 so the psu size varies from 180w to 305w on the biggest MT model.

Newer AMD cards don't seem to work in older machines.

The incompatibility seems to start around the time of transition from Legacy to UEFI. I suppose around transition from old Radeon HD 5xxx,6xxx cards to newer Rx cards.

https://dell.com/community/Vostro/Vostro-420-Power-Supply-and-Graphics-Upgrade/m-p/6067239/highlight/true#M1277

@speedstep identified the problem as a missing VESA-Mode

https://dell.com/community/XPS-Desktops/XPS-8500-video-card-upgrade-questions/m-p/6103464/highlight/true#M13215

I suggest you stick to similar vintage AMD cards in these old machines, or try switching to Nvidia.

The card that will work with 780 MT is

XFX AMD Radeon HD 7750 FX775AZNP4

https://www.amazon.com/XFX-DisplayPort-PCI-Express-FX775AZNP4-FX-775A-ZNP4/dp/B007Z3T5JC

The missing MSDOS VESA MODE 103 goes back to INT 13  1981  MSDOS 1.0.  The primary benefit is compatibility and performance with DOS games. Older Dell Phoenix and AMI bios rely on this api for the F2 bios menus.

Cards and Bios that support this will run MSDOS all the way thru WINDOWS 10 as long as the other parameters are met.  This means the oldest processor supported is PENTIUM D 915 - 960.  Optiplex GX620 mini tower meets that spec.  Which is why this is the most DESIREABLE legacy machine dell ever made.  Supports Dos all the way thru WIN10 and supports SATA and PATA IDE and PS2 mice and Keyboards and Serial ports and Parallel ports and PCI-E video cards etc.  The oldest model in this series is the GX760 but the 760 does not have IDE ports for hard drives and optical drives like the GX620 does.

UEFI does not support 32 bit versions of windows like 98, 2000,  XP,  VISTA  let alone 8 bit MSDOS. or 16 bit WINDOWS 3.xx.

What that means is that the older cards have drivers that go back to WIN98 and WIN2000 with pointers to NTVDM.DLL  which supports DOS and VESA api.

32 bit windows VISTA/7/8/10 have NTVDM but 64 bit versions do not.

Vista and Below have been set to END OF LIFE END OF SUPPORT which means there will NEVER BE any drivers to support these old API's on Newer UEFI cards.

When you get to CLASS 3 UEFI there is NO Compatibility Support Module (CSM) and therefore ONLY the OEM you have will support THEIR cards but not 3rd party cards.

This is why you have to turn SECURE BOOT OFF to get 3rd party cards to work at all. Mode 103 is chosen to support SVGA or better monitors.