DPA: Error 500 returned when running reports

When attempting to run a Client Summary or Job Summary report, a popup window appears with an Error 500. The reports never run

Examining the Data Protection Advisor (DPA) Application's server log files, there were many warning messages stating that it could not insert data into tables or failing to run queries.
For example:

Failed executing updatesql=insert into bkup_ddclone .......Caused by: java.sql.SQLException: Error

And

Unhandled exception thrown from onMessage: java.lang.IllegalStateException: Failed processing agent report: BaseTransaction.rollback - ARJUNA016074: no transaction! ..........Caused by: com.emc.apollo.services.database.NativeQueryException: Failed executing querysql=select * from backupevent

The following errors were observed with examining the DPA Datastore logs:

"ERROR: could not extend file "pg_tblspc/16390/PG_9.6_201608131/16385/35561.12": No space left on device"
"ERROR: could not open file "pg_tblspc/16389/PG_9.6_201608131/16385/35570": No such file or directory".

Another error message in the Datastore logs is observed after a Datastore restart:

2019-08-07 13:22:26.172 EDT 11996 5d4b08d0.2edc 1 apollo:ERROR: could not open file "pg_tblspc/16389/PG_9.6_201608131/16385/35558.5" (target block 2322516): previous segment is only 13128 blocks

Examining the Datastore file system it is found that the file system where the database resides (...\services\datastore\data) has no space left.

The Error 500 popup window is a generic error from the DPA Application, stating that it failed to perform the last action or command.

The cause of the inability to run reports is that the database is out of disk space.

When the file system fills up, there are multiple issues that can occur.

If the DPA UI is already running, misbehavior and failures are seen in multiple areas in the UI, including with running reports. This is because the DPA Application must be able to query the database for information and data. This is not normally possible when the database file system is full.

The database could become corrupted or damaged in some way. Postgres caches some amount of data in shared memory at any point in time. If the file system fills up, then Postgres cannot commit (write) this data to the file system as it should. This can cause data that was to be inserted to be lost. It can also cause database files to be incomplete and damaged. Lastly, other data may not be written as it should, including database control files.

It is also likely that the Datastore may not be able to be restarted once the file system is full.

It requires that more file system space (disk) be provided to the Datastore database to resolve this problem. This can be done in any number of ways, including extending the file system or adding a new disk. Consult with the server's Administrators for assistance with adding space.

Once disk space has been added, one can then start the Datastore database again.

On restart, the database attempts to perform a recovery from the previous abnormal termination. The recovery startup attempts to make any corrections it can, so the database is functional and operating. This may or may not result in a normal and functioning database. It could result in a database that appears to run normally, but has internal structure issues or damage. Also as previously mentioned it likely results in some amount of lost data

After restarting the Datastore database, the database should be examined by Dell Technical Support to ensure that it is functioning normally and that there are no other after effects or issues.

It is also recommended that a Datastore export is performed to ensure that there is a viable copy of the database available going forwards. Preferably run the export command with the "--debug" switch added to it, so it provides debug output. Redirect this output to a file, so it can be reviewed afterwards for any issues.

Contact Dell Technical Support for more information and assistance.