Dell VxRail: ESXi root account has been locked for 900s after failed login attempts.

The root account of one or more ESXi hosts has been locked due to several failed login attempts.
Unable to cannot connect to the node using SSH or the web UI.
Confirm the issue using the iDRAC console to the ESXi shell.

In vCenter, a warning message is shown similar to the following:

Remote access for ESXi local user account 'root' has been locked for 900s after 14 failed login attempts.

Figure 1: Remote access is locked

Logs similar to the following are found on the affected host:

/var/log/vobd.log

2020-04-03T17:27:58.790Z: [GenericCorrelator] 8202447897096us: [vob.user.account.locked] Remote access for ESXi local user account 'root' has been locked for 900 seconds after 32 failed login attempts.
2020-04-03T17:27:58.790Z: [UserLevelCorrelator] 8202447897096us: [vob.user.account.locked] Remote access for ESXi local user account 'root' has been locked for 900 seconds after 32 failed login attempts.
2020-04-03T17:27:58.791Z: [UserLevelCorrelator] 8202447897325us: [esx.audit.account.locked] Remote access for ESXi local user account 'root' has been locked for 900 seconds after 32 failed login attempts.

/var/log/auth.log

2020-04-03T17:29:06Z sshd[701694298]: Connection from 192.168.100.40 port 55682
2020-04-03T17:29:06Z sshd[701333862]: pam_tally2(sshd:auth): user root (0) tally 34, deny 5
2020-04-03T17:29:08Z sshd[701694298]: error: PAM: Authentication failure for root from 192.168.100.40
2020-04-03T17:29:08Z sshd[701694492]: pam_tally2(sshd:auth): user root (0) tally 35, deny 5
2020-04-03T17:29:08Z sshd[701694492]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.100.40  user=root
2020-04-03T17:29:10Z sshd[701694298]: error: PAM: Authentication failure for root from 192.168.100.40
2020-04-03T17:29:10Z sshd[701694298]: error: Received disconnect from 192.168.100.40 port 55682:3: com.jcraft.jsch.JSchException: Auth cancel [preauth]
2020-04-03T17:29:10Z sshd[701694298]: Disconnected from authenticating user root 192.168.100.40 port 55682 [preauth]

The root password for the node may have been changed, but the third-party monitoring software has not been updated with the new root password.

This causes multiple failed logins (sometimes hundreds or even thousands). This locks the root account for at least 15 minutes. Unable to SSH to the node or log in to the node web UI.

You can log in through the DCUI and the ESXi shell.

Starting with vSphere 6.0, account locking is supported for access through SSH and through the vSphere Web Services SDK. The Direct Console Interface (DCUI) and the ESXi Shell do not support account lockout. By default, a maximum of five failed attempts are allowed before the account is locked. The account is unlocked after 15 minutes by default.

To resolve this issue:

1. Connect to the iDRAC console and then to the ESXi shell. 
2. Enable the shell by logging in to the DCUI and enabling the ESXi shell under troubleshooting options.
3. You can also do a Cntrl-Alt-F1 to access the shell.
4. After connecting to the ESXi shell, run the commands below. The output should match the screenshot below, except the "From" entry says "unknown".

#pam_tally2 --user root
#pam_tally2 --user root --reset
#pam_tally2 --user root

Figure 2: ESXi commands and output
 

• After running the above commands, log in to the ESXi node web UI. 
• Go to Monitor and then Events. You should see an IP address that was trying to log in that is listed as failed. 
• You must identify the application based on the IP address that is listed here. Either stop it or configure it with the correct credentials.

For more information, reference ESXi Passwords and Account Lockout.