Data Protection Advisor: Temporary files in pgsql_tmp folder on Datastore server increased rapidly and consuming all available space on filesystem

요약: Files in pgsql_tmp folder on Datastore server increased rapidly and filling the disk.

이 문서는 다음에 적용됩니다. 이 문서는 다음에 적용되지 않습니다. 이 문서는 특정 제품과 관련이 없습니다. 모든 제품 버전이 이 문서에 나와 있는 것은 아닙니다.

증상

Filesystem on datastore server where Postgres is located is almost full.
Application service starts successfully but fails to process data and soon crashes.
The following message was found in the server.log on the Application server:
2022-11-15 08:41:00,700 WARN  [org.jboss.jca.core.connectionmanager.listener.TxConnectionListener] (ReportEngineMessageListener #1:Backup Report Card:22b4e14e-a7a9-4b49-a244-54336df9661f) IJ000305: Connection error occured:
org.postgresql.util.PSQLException: ERROR: could not create temporary file "base/pgsql_tmp/pgsql_tmp921.5.sharedfileset/i165of8192.p0.0": No space left on device


An error followed the above warning:
2022-11-15 08:41:00,704 ERROR [com.emc.apollo.jdbc.exec.SQLExecutor] (ReportEngineMessageListener #1:Backup Report Card:22b4e14e-a7a9-4b49-a244-54336df9661f) Error executing query: Error. Query was: 
SELECT t0.f_agent_name as "t0f_agent_name", t0.f_sub_name as "t0f_sub_name", t0.f_group_name as "t0f_group_name", t0.f_schedule_name as "t0f_schedule_name", t0.f_client_name as "t0f_client_name", t0.f_policy_name as "t0f_policy_name", t0.f_workflow_name as "t0f_workflow_name", t0.f_workflow_jobid as "t0f_workflow_jobid", t0.f_action_name as "t0f_action_name", t0.f_job_name as "t0f_job_name", t0.f_domain_name as "t0f_domain_name", t0.f_backup_set as "t0f_backup_set", t0.f_session as "t0f_session", t0.f_proxy as "t0f_proxy", t0.f_queuestart as "t0f_queuestart", t0.f_status as "t0f_status", t0.f_starttime as "t0f_starttime", t0.f_endtime as "t0f_endtime", t0.f_ntries as "t0f_ntries", t0.f_queuestart AS f_reporterstarttime, t0.f_queuestart AS f_reporterendtime, t0.f_id  as f_id
FROM view_backupjob t0....


In the Datastore logs, similar messages will be found:
2022-11-15 07:29:49.001 GMT [7443] ERROR:  could not write to file "base/pgsql_tmp/pgsql_tmp4401.6.sharedfileset/i19081of32768.p1.0": No space left on device

Checking on the Datastore server in <install_dir>/dpa/services/datastore/data/base/pgsql_tmp directory, there are numerous folders and files making up many GBs in size.
The date on the first files in the pgsql_tmp directory may correspond to a recent upgrade of the DPA software.

원인

During the recent upgrade, the DPA Datastore was not tuned correctly for the amount of RAM available on the server.
Running the following command shows the following:
# dpa.sh ds tune
EMC Data Protection Advisor
[INFO] Current Datastore Service tuned expecting 512MB of host memory
Command completed successfully.

The 512MB value is typical of default settings for PostgreSQL server.
This low memory setting caused DPA to write query information to temporary files in pgsql_tmp. 
When the filesystem became full, the system killed the query which left behind the tmp files that eventually filled up all the free space on the filesystem.
This caused the error in the server.log relating to a SELECT query being unable to create temporary file in pgsql_tmp directory.

해결

Run the tune command on the Datastore server for the amount of RAM on the server.
For example, if the server is configured with 64GB of RAM, run the command:
 # dpa.sh ds tune 64GB
EMC Data Protection Advisor
Set Datastore service memory usage to 65536MB : Confirm [Y|N]
y
Datastore Service successfully tuned.
The new memory utilisation for the datastore will not take effect until the datastore service is restarted.
Command completed successfully.

Restart the Datastore service.
Upon restart, the files in pgsql_tmp will be automatically cleared by postgresql process.
Monitor the latest Datastore service log, when the following message appears, the Application can then be started.
2022-11-15 09:19:10.003 GMT [10897] LOG:  database system is ready to accept connections

With more memory tuned for the Datastore service, there will be less requirement to write query data to temporary files in pgsql_tmp.
Please contact Dell Technical Support for further information.

해당 제품

Data Protection Advisor
문서 속성
문서 번호: 000205495
문서 유형: Solution
마지막 수정 시간: 25 5월 2023
버전:  3
다른 Dell 사용자에게 질문에 대한 답변 찾기
지원 서비스
디바이스에 지원 서비스가 적용되는지 확인하십시오.