IDPA vCenter upgrade fails with error "vCSACliInstallLogger - ERROR - vCenter Server Data Export failed"

Summary: This KB Article walks you through vCenter Upgrade failure caused due to export of vcenter db failing. This could be due to the vcenter DB corruption.

This article applies to This article does not apply to This article is not tied to any specific product. Not all product versions are identified in this article.
Check out other resources

Symptoms



On the ACM, you may notice the vCenter shows as failed.

Looking into the vCenter Upgrade log under "/data01/tmp/patch/logs/VCSA" directory, the following errors can be seen on vcsaclientinstaller log :

Progress: 42% Exporting VMware vCenter Server Database data...
2020-03-26 03:41:00,145 - vCSACliInstallLogger - ERROR - vCenter Server Data Export failed.
2020-03-26 03:41:00,145 - vCSACliInstallLogger - ERROR - Upgrade failed:
Status: ERROR
Progress: 42% Exporting VMware vCenter Server Database data...
Error:
    Problem Id: None
    Component key: upgrade_framework
    Detail:
        Error while exporting source SET tables: pg_dump: Error message from server: ERROR:  invalid page in block 9 of relation base/16384/25164

    Resolution: Read the log files and address the issues found in your source SET tables.
2020-03-26 03:41:00,145 - vCSACliInstallLogger - INFO - See /data01/tmp/patch/logs/vclogs/monitor-firstboot-progress-oczvL_.json file on local machine or /var/log/vmware/upgrade/export.json file on remote machine for details.


Additionally, on the vCenter server, Upgrade logs under "/var/log/vmware/upgrade" can also be verified and they may show the following error message:

VMware License Service data...
2020-03-26 17:09:01,732 - vCSACliInstallLogger - INFO - vCenter Server Data Export: Progress: 42% Exporting VMware vCenter Server Database data...
2020-03-26 17:09:12,296 - vCSACliInstallLogger - ERROR - vCenter Server Data Export failed.
2020-03-26 17:09:12,296 - vCSACliInstallLogger - ERROR - Upgrade failed:
Status: ERROR
Progress: 42% Exporting VMware vCenter Server Database data...
Error:
    Problem Id: None
    Component key: upgrade_framework
    Detail:
        Error while exporting source SET tables: pg_dump: Error message from server: ERROR:  invalid page in block 9 of relation base/16384/25164

    Resolution: Read the log files and address the issues found in your source SET tables.
2020-03-26 17:09:12,296 - vCSACliInstallLogger - INFO - See /data01/tmp/patch/logs/vclogs/monitor-firstboot-progress-tmcn2I.json file on local machine or /var/log/vmware/upgrade/export.json file on remote machine for details.
2020-03-26 17:09:12,305 - vCSACliInstallLogger - INFO - Gathering VC support log bundle. This can take a few minutes.
2020-03-26 17:09:12,583 - vCSACliInstallLogger - INFO - Uninstalling vmware-upgrade-requirements-* on 192.168.100.99.
2020-03-26 17:09:13,881 - vCSACliInstallLogger - ERROR - (u'Failed to uninstall %(rpm)s on %(host)s: %(error)s', {'host': u'192.168.100.99', 'rpm': u'vmware-upgrade-requirements-*', 'error': CommandError(u"Failed to run and wait for command in guest with error 'Command '/bin/rpm' exited with non-zero status 1'",)})


postgres.log on the vcenter shows the following error message: 

816 UTC 5e7ce0cc.7647 24294965  LOG: automatic analyze of table "VCDB.vc.vpx_event_arg" system usage: CPU 0.02s/1.53u sec elapsed 6.15 sec
2020-03-26 17:08:19.502 UTC 5e7ce180.2530 24295002  LOG: automatic analyze of table "VCDB.vc.vpx_event" system usage: CPU 0.00s/2.40u sec elapsed 2.79 sec
2020-03-26 17:08:58.326 UTC 5e7c377f.3892 0 VCDB vc LOG: unexpected EOF on client connection with an open transaction
2020-03-26 17:08:58.326 UTC 5e7c3796.3a7b 0 VCDB vc LOG: unexpected EOF on client connection with an open transaction
2020-03-26 17:08:58.327 UTC 5e7c3796.3a81 0 VCDB vc LOG: unexpected EOF on client connection with an open transaction
2020-03-26 17:09:09.530 UTC 5e7ce1b3.3c67 0 VCDB postgres ERROR: invalid page in block 9 of relation base/16384/25164
2020-03-26 17:09:09.530 UTC 5e7ce1b3.3c67 0 VCDB postgres STATEMENT: COPY vc.vpx_topn_past_month (counter_id, sample_time, integral_value) TO stdout;
SET zero_damaged_pages = on; select * from <corrupted table>; SET zero_damaged_pages = off;

VCDB=# select counter_id from vpx_topn_past_month;
ERROR: invalid page in block 9 of relation base/16384/25164

Cause

vCenter Postgres Database corruption caused vCenter Upgrade to fail during DB Export task. 

Resolution

Step 1: Login to the IDPA vCenter UI.

Step 2: Make a clone of vCenter VM.
  • Right click on VCSA VM --> Clone to Virtual Machine. 
  • Choose the name of the clone and other details and click on OK. 

Step 3: Perform a rollback for the vCenter VM on the ACM UI. 
  • Click on Troubleshooting Options checkbox. Click on Rollback for vCenter Upgrade.  

Step 4: SSH to vCenter VM and stop the vpxd service.
  • service-control --stop vpxd

Step 5: Login to vCenter Postgres DB.
  • /opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres
Step 6: Run the following queries/commands on the vCenter Postgres db:
  • SET zero_damaged_pages = on;
  • VACUUM FULL ANALYSE VERBOSE;
  • REINDEX table vpx_topn_past_month;
  • select * from vpx_topn_past_month;
  • SET zero_damaged_pages = off;
Step 7: Go to ACM UI, Click on Retry for vCenter Upgrade and monitor it. 


    Additional Information

    Involve VMware support if issue is unresolved after following the above steps. 

    Affected Products

    Integrated Data Protection Appliance Family

    Products

    PowerProtect DP4400, PowerProtect DP5300, PowerProtect DP5800, PowerProtect DP8300, PowerProtect DP8800, PowerProtect Data Protection Software, Integrated Data Protection Appliance Family, PowerProtect Data Protection Hardware , Integrated Data Protection Appliance Software ...
    Article Properties
    Article Number: 000173289
    Article Type: Solution
    Last Modified: 27 May 2025
    Version:  3
    Find answers to your questions from other Dell users
    Support Services
    Check if your device is covered by Support Services.