VCF on VxRail: SDDC Manager Workload Domain Upgrade Precheck Failed at Step "VxRail Manager SSH Connection"

For a VCF on VxRail cluster

After upgrade to VxRail 7.0.410 or 7.0.411 release, then perform a new upgrade, SDDC manager upgrade precheck will fail at step "VxRail Manager SSH Connection" with Error Description "Auth fail."

Find "mystic" user cannot connect to VxRail Manager over SSH.

root@sddc-manager-controller [ ~ ]# ssh mystic@<VxM-IP>
FIPS mode initialized
Password:
Account locked due to 7 failed logins

Log in to VxRail Manager as root user using VM console in vCenter Run the following command.

pam_tally2 --user mystic

Observed that user mystic has been locked due to multiple login failures.

For standard VxRail cluster

After upgrade to 7.0.410 or 7.0.411, may also find "mystic" user cannot connect to VxRail Manager over SSH as it is locked.

Due to a SUSE Linux issue, after VxRail upgrade to 7.0.410 or 7.0.411, the config file /etc/pam.d/common-account is not successfully updated. (If the VxRail is directly deployed at 7.0.410 or 7.0.411, then it is not impacted by this issue.)

As a result, when establishing SSH connection from other VMs to VxRail Manager VM via mystic user, although the login is successful, the failed login count is still incremented. After it reaches the maximum allowed failed login count, the mystic user will be locked.

Note: Direct SSH to VxRail Manager is not impacted by this issue, for example successful SSH login from your laptop to VxRail manager VM does not cause the failed login count increasing. A VCF on VxRail environment hits this issue more frequently, because during upgrade precheck, the SSH connection is established from SDDC manager VM to VxRail manager VM.

This issue is fixed in VxRail 7.0.450 release, means when VxRail is already running on 7.0.450 then performing a new upgrade to a higher version, SDDC manager upgrade precheck will not hit this issue.

Check the below list to determine if your clusters are impacted by this issue:

• VxRail is upgraded to 7.0.410 or 7.0.411 then performing new upgrade to a higher version (including 7.0.450), SDDC manager upgrade precheck will hit this issue.
• VxRail is deployed at 7.0.410 or 7.0.411 (greenfield), then performing new upgrade to a higher version, SDDC manager upgrade precheck will not hit this issue.
• VxRail is running on a pre-7.0.410 version then performing new upgrade to a higher version, SDDC manager upgrade precheck will not hit this issue.
• VxRail is running on 7.0.450 then performing a new upgrade to a higher version, SDDC manager upgrade precheck will not hit this issue.

VCF on VxRail customers follow the below table to avoid SDDC manager upgrade precheck hitting this issue:

* If you have installed VCF 4.5 with VxRail 7.0.410 in greenfield then you can ignore the KB. Refer the KB only if you have upgraded to VxRail 7.0.410 from any other VxRail version.

If SDDC manager upgrade precheck already hit this issue, please also apply these workaround steps.

Workaround Steps:

=================

1. Add the following line to the /etc/pam.d/common-account file in VxRail Manager. (Using virtual machine console in vCenter with root account, since SSH connection does not work)

account    required    pam_tally2.so

2. Unlock the mystic user by running the following command in the VxRail Manager command console with root account.

pam_tally2 --user mystic --reset

3. Once mystic account has been unlocked, user should validate it and try to establish SSH session to VxRail Manager using the mystic credentials.
 

4. Proceed with workload domain upgrade precheck on SDDC manager.

User could lookup the correct passwords of VxRail Manager from the SDDC manager by running the lookup_passwords command as shown below.
 

root@sddc-manager-controller [ /home/vcf ]# lookup_passwords
Password lookup operation requires ADMIN user credentials. Please refer VMware Cloud Foundation Administration Guide for setting up ADMIN user.
Supported entity types: ESXI VCENTER PSC NSX_MANAGER NSX_CONTROLLER NSXT_MANAGER NSXT_EDGE VRSLCM VRLI VROPS VRA WSA BACKUP VXRAIL_MANAGER AD
Enter an entity type from above list: VXRAIL_MANAGER
Enter page number (optional):
Enter page size (optional, default=50):
Enter Username: administrator@vsphere.local
Enter Password:<password_of_administrator@vsphere.local>
        VXRAIL_MANAGER
        identifiers: 172.16.6.129,app01-vxrm.test.local
        workload: app01-md
                username: mystic
                password: <passord_of_mystic>
                type: SSH
                account type: SYSTEM
        VXRAIL_MANAGER
        identifiers: 172.16.6.129,app01-vxrm.test.local
        workload: app01-md
                username: root
                password: <passord_of_root>
                type: SSH
                account type: SYSTEM