Data Protection Advisor: Temporary files in pgsql_tmp folder on Datastore server increased rapidly and consuming all available space on filesystem
Summary: Files in pgsql_tmp folder on Datastore server increased rapidly and filling the disk.
This article applies to
This article does not apply to
This article is not tied to any specific product.
Not all product versions are identified in this article.
Symptoms
Filesystem on datastore server where Postgres is located is almost full.
Application service starts successfully but fails to process data and soon crashes.
The following message was found in the server.log on the Application server:
2022-11-15 08:41:00,700 WARN [org.jboss.jca.core.connectionmanager.listener.TxConnectionListener] (ReportEngineMessageListener #1:Backup Report Card:22b4e14e-a7a9-4b49-a244-54336df9661f) IJ000305: Connection error occured:
org.postgresql.util.PSQLException: ERROR: could not create temporary file "base/pgsql_tmp/pgsql_tmp921.5.sharedfileset/i165of8192.p0.0": No space left on device
An error followed the above warning:
2022-11-15 08:41:00,704 ERROR [com.emc.apollo.jdbc.exec.SQLExecutor] (ReportEngineMessageListener #1:Backup Report Card:22b4e14e-a7a9-4b49-a244-54336df9661f) Error executing query: Error. Query was:
SELECT t0.f_agent_name as "t0f_agent_name", t0.f_sub_name as "t0f_sub_name", t0.f_group_name as "t0f_group_name", t0.f_schedule_name as "t0f_schedule_name", t0.f_client_name as "t0f_client_name", t0.f_policy_name as "t0f_policy_name", t0.f_workflow_name as "t0f_workflow_name", t0.f_workflow_jobid as "t0f_workflow_jobid", t0.f_action_name as "t0f_action_name", t0.f_job_name as "t0f_job_name", t0.f_domain_name as "t0f_domain_name", t0.f_backup_set as "t0f_backup_set", t0.f_session as "t0f_session", t0.f_proxy as "t0f_proxy", t0.f_queuestart as "t0f_queuestart", t0.f_status as "t0f_status", t0.f_starttime as "t0f_starttime", t0.f_endtime as "t0f_endtime", t0.f_ntries as "t0f_ntries", t0.f_queuestart AS f_reporterstarttime, t0.f_queuestart AS f_reporterendtime, t0.f_id as f_id
FROM view_backupjob t0....
In the Datastore logs, similar messages will be found:
2022-11-15 07:29:49.001 GMT [7443] ERROR: could not write to file "base/pgsql_tmp/pgsql_tmp4401.6.sharedfileset/i19081of32768.p1.0": No space left on device
Checking on the Datastore server in <install_dir>/dpa/services/datastore/data/base/pgsql_tmp directory, there are numerous folders and files making up many GBs in size.
The date on the first files in the pgsql_tmp directory may correspond to a recent upgrade of the DPA software.
Application service starts successfully but fails to process data and soon crashes.
The following message was found in the server.log on the Application server:
2022-11-15 08:41:00,700 WARN [org.jboss.jca.core.connectionmanager.listener.TxConnectionListener] (ReportEngineMessageListener #1:Backup Report Card:22b4e14e-a7a9-4b49-a244-54336df9661f) IJ000305: Connection error occured:
org.postgresql.util.PSQLException: ERROR: could not create temporary file "base/pgsql_tmp/pgsql_tmp921.5.sharedfileset/i165of8192.p0.0": No space left on device
An error followed the above warning:
2022-11-15 08:41:00,704 ERROR [com.emc.apollo.jdbc.exec.SQLExecutor] (ReportEngineMessageListener #1:Backup Report Card:22b4e14e-a7a9-4b49-a244-54336df9661f) Error executing query: Error. Query was:
SELECT t0.f_agent_name as "t0f_agent_name", t0.f_sub_name as "t0f_sub_name", t0.f_group_name as "t0f_group_name", t0.f_schedule_name as "t0f_schedule_name", t0.f_client_name as "t0f_client_name", t0.f_policy_name as "t0f_policy_name", t0.f_workflow_name as "t0f_workflow_name", t0.f_workflow_jobid as "t0f_workflow_jobid", t0.f_action_name as "t0f_action_name", t0.f_job_name as "t0f_job_name", t0.f_domain_name as "t0f_domain_name", t0.f_backup_set as "t0f_backup_set", t0.f_session as "t0f_session", t0.f_proxy as "t0f_proxy", t0.f_queuestart as "t0f_queuestart", t0.f_status as "t0f_status", t0.f_starttime as "t0f_starttime", t0.f_endtime as "t0f_endtime", t0.f_ntries as "t0f_ntries", t0.f_queuestart AS f_reporterstarttime, t0.f_queuestart AS f_reporterendtime, t0.f_id as f_id
FROM view_backupjob t0....
In the Datastore logs, similar messages will be found:
2022-11-15 07:29:49.001 GMT [7443] ERROR: could not write to file "base/pgsql_tmp/pgsql_tmp4401.6.sharedfileset/i19081of32768.p1.0": No space left on device
Checking on the Datastore server in <install_dir>/dpa/services/datastore/data/base/pgsql_tmp directory, there are numerous folders and files making up many GBs in size.
The date on the first files in the pgsql_tmp directory may correspond to a recent upgrade of the DPA software.
Cause
During the recent upgrade, the DPA Datastore was not tuned correctly for the amount of RAM available on the server.
Running the following command shows the following:
The 512MB value is typical of default settings for PostgreSQL server.
This low memory setting caused DPA to write query information to temporary files in pgsql_tmp.
When the filesystem became full, the system killed the query which left behind the tmp files that eventually filled up all the free space on the filesystem.
This caused the error in the server.log relating to a SELECT query being unable to create temporary file in pgsql_tmp directory.
Running the following command shows the following:
# dpa.sh ds tune
EMC Data Protection Advisor
[INFO] Current Datastore Service tuned expecting 512MB of host memory
Command completed successfully.
EMC Data Protection Advisor
[INFO] Current Datastore Service tuned expecting 512MB of host memory
Command completed successfully.
The 512MB value is typical of default settings for PostgreSQL server.
This low memory setting caused DPA to write query information to temporary files in pgsql_tmp.
When the filesystem became full, the system killed the query which left behind the tmp files that eventually filled up all the free space on the filesystem.
This caused the error in the server.log relating to a SELECT query being unable to create temporary file in pgsql_tmp directory.
Resolution
Run the tune command on the Datastore server for the amount of RAM on the server.
For example, if the server is configured with 64GB of RAM, run the command:
Restart the Datastore service.
Upon restart, the files in pgsql_tmp will be automatically cleared by postgresql process.
Monitor the latest Datastore service log, when the following message appears, the Application can then be started.
2022-11-15 09:19:10.003 GMT [10897] LOG: database system is ready to accept connections
With more memory tuned for the Datastore service, there will be less requirement to write query data to temporary files in pgsql_tmp.
Please contact Dell Technical Support for further information.
For example, if the server is configured with 64GB of RAM, run the command:
# dpa.sh ds tune 64GB
EMC Data Protection Advisor
Set Datastore service memory usage to 65536MB : Confirm [Y|N]
y
Datastore Service successfully tuned.
The new memory utilisation for the datastore will not take effect until the datastore service is restarted.
Command completed successfully.
EMC Data Protection Advisor
Set Datastore service memory usage to 65536MB : Confirm [Y|N]
y
Datastore Service successfully tuned.
The new memory utilisation for the datastore will not take effect until the datastore service is restarted.
Command completed successfully.
Restart the Datastore service.
Upon restart, the files in pgsql_tmp will be automatically cleared by postgresql process.
Monitor the latest Datastore service log, when the following message appears, the Application can then be started.
2022-11-15 09:19:10.003 GMT [10897] LOG: database system is ready to accept connections
With more memory tuned for the Datastore service, there will be less requirement to write query data to temporary files in pgsql_tmp.
Please contact Dell Technical Support for further information.
Affected Products
Data Protection AdvisorArticle Properties
Article Number: 000205495
Article Type: Solution
Last Modified: 25 May 2023
Version: 3
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.