VCF on VxRail: Unable to launch SDDC UI while upgrading VCF to 4.4.x or 4.5.x

During a VMware Cloud Foundation upgrade to 4.4.x, the VCF upgrade is stuck at the SDDC MANAGER UI step and does not proceed.

• The VCF upgrade process gets stuck at the SDDC MANAGER UI step and may not proceed. 

• Refreshing the browser/UI displays this error message:

  {"message":"Session middleware wrapper failed initial and all retry attempts. Cannot continue.  Throwing error from session middleware - password authentication failed for user \"vcfui\""}

• SDDC Manager UI fails to launch due to middleware wrapper user vcfui failing to authenticate.

• /var/log/vmware/vcf/sddc-manager-ui-app/sddcManagerServer.log also reports failed authentication and Session middleware wrapper error

This is caused by a service restart occurring during the upgrade. The user is affected, and this results in a failed authentication.

There are two workarounds for the above issue.

Workaround 1:

• Reboot the SDDC manager and wait for the services to come up.

• The SDDC Manager UI should launch, and you can log in. If you are still unable to launch the SDDC manager UI, follow workaround 2.

• Run the pre-upgrade check and Retry the VCF upgrade.

• If the VCF upgrade runs successfully, workaround 2 can be skipped. If the issue recurs at the second attempt of the upgrade, follow workaround 2.

Workaround 2: 

• Take a snapshot of the SDDC Manager VM.

• SSH into SDDC Manager as vcf and su to root.

• Launch postgres: 

psql --host=localhost -U postgres

• Drop the affected DATABASE/USER:

DROP DATABASE IF EXISTS sddc_manager_ui; DROP ROLE IF EXISTS vcfui;
 
Should return output of:
DROP DATABASE
DROP ROLE

• Quit out of the postgres database:  

\q

• Restart SDDC Manager UI services:

systemctl restart sddc-manager-ui-db
systemctl restart sddc-manager-ui-app

• The SDDC Manager UI should launch, and you can log in.

• Run the pre-upgrade check and Retry the VCF upgrade.