Dell VCFonVxRail: SDDC manager workload domain upgrade precheck failed at step "VxRail Manager SSH connection"

For VCF on VxRail cluster

After upgrade to VxRail 7.0.410 or 7.0.411 release, then perform a new upgrade, SDDC manager upgrade precheck will fail at step "VxRail Manager SSH Connection" with Error Description "Auth fail".

Find "mystic" user cannot connect to VxRail Manager over SSH.

root@sddc-manager-controller [ ~ ]# ssh mystic@<VxM-IP>
FIPS mode initialized
Password:
Account locked due to 7 failed logins

Login to VxRail Manager as root user via VM console in vCenter. Run the following command

pam_tally2 --user mystic

Observed that user mystic has been locked due to multiple login failures.

 

For standard VxRail cluster

After upgrade to 7.0.410 or 7.0.411, may also find "mystic" user cannot connect to VxRail Manager over SSH as it is locked.

Due to a SUSE Linux issue, after VxRail upgrade to 7.0.410 or 7.0.411, the config file /etc/pam.d/common-account is not successfully updated. (If the VxRail is directly deployed at 7.0.410 or 7.0.411, then it is not impacted by this issue.)

As a result, when establishing SSH connection from other VMs to VxRail Manager VM via mystic user, although the login is successful, the failed login count is still incremented. After it reaches the maximum allowed failed login count, the mystic user will be locked.

Note: Direct SSH to VxRail Manager is not impacted by this issue, for example successful SSH login from your laptop to VxRail manager VM will not cause the failed login count increasing. So VCF on VxRail environment will hit this issue more frequently, because during upgrade precheck, the SSH connection is established from SDDC manager VM to VxRail manager VM.

This issue is fixed in VxRail 7.0.450 release, means when VxRail is already running on 7.0.450 then performing a new upgrade to a higher version, SDDC manager upgrade precheck will not hit this issue.

Check below list to determine if your clusters are impacted by this issue:

• VxRail is upgraded to 7.0.410 or 7.0.411 then performing new upgrade to a higher version (including 7.0.450), SDDC manager upgrade precheck will hit this issue.
• VxRail is deployed at 7.0.410 or 7.0.411 (greenfield), then performing new upgrade to a higher version, SDDC manager upgrade precheck will not hit this issue.
• VxRail is running on pre-7.0.410 version then performing new upgrade to a higher version, SDDC manager upgrade precheck will not hit this issue.
• VxRail is running on 7.0.450 then performing a new upgrade to a higher version, SDDC manager upgrade precheck will not hit this issue.

VCF on VxRail customers please follow below table to avoid SDDC manager upgrade precheck hitting this issue:

* If you have installed VCF 4.5 with VxRail 7.0.410 in greenfield then you can ignore the KB. Refer the KB only if you have upgraded to VxRail 7.0.410 from any other VxRail version.

If SDDC manager upgrade precheck already hit this issue, please also apply this KB workaround steps.

Workaround Steps:

=================

1. Add the following line to the /etc/pam.d/common-account file in VxRail Manager. (Using virtual machine console in vCenter with root account, since SSH connection does not work)

account    required    pam_tally2.so

2. Unlock the mystic user by running the following command in VxRail Manager command console with root account.

pam_tally2 --user mystic --reset

3. Once mystic account has been unlocked, user should validate it and try to establish SSH session to VxRail Manager using the mystic credentials.
 

4. Proceed with workload domain upgrade precheck on SDDC manager.

User could lookup the correct passwords of VxRail Manager from SDDC manager by running lookup_passwords command as shown below.

root@sddc-manager-controller [ /home/vcf ]# lookup_passwords
Password lookup operation requires ADMIN user credentials. Please refer VMware Cloud Foundation Administration Guide for setting up ADMIN user.
Supported entity types: ESXI VCENTER PSC NSX_MANAGER NSX_CONTROLLER NSXT_MANAGER NSXT_EDGE VRSLCM VRLI VROPS VRA WSA BACKUP VXRAIL_MANAGER AD
Enter an entity type from above list: VXRAIL_MANAGER
Enter page number (optional):
Enter page size (optional, default=50):
Enter Username: administrator@vsphere.local
Enter Password:<password_of_administrator@vsphere.local>
        VXRAIL_MANAGER
        identifiers: 172.16.6.129,app01-vxrm.test.local
        workload: app01-md
                username: mystic
                password: <passord_of_mystic>
                type: SSH
                account type: SYSTEM
        VXRAIL_MANAGER
        identifiers: 172.16.6.129,app01-vxrm.test.local
        workload: app01-md
                username: root
                password: <passord_of_root>
                type: SSH
                account type: SYSTEM