PowerFlex: Root Account Lockout During Upgrade - Password is not Synced \Cached

• Root‑account lockouts on a subset of ESXi hosts during patch upgrade.
• Upgrade jobs fail for the affected nodes.
• Log entries in deployment.log showing "Invalid login attempt".
• ASMManager and thin‑deployer logs contain "Failed to update OS password for node." messages.
• Presence of "null" objects in ASMManager job logs.

The thin‑deployer pod maintains a local cache of the ESXi root password. During the upgrade process the password stored in the PowerFlex PostgreSQL database (os_admin_password) was changed, but the cache in the thin‑deployer pod was not refreshed. Consequently, the pod attempted to authenticate to ESXi hosts with a stale password, resulting in “Invalid login attempt” errors and lockouts.

• Credential cache not synchronized after password change.
• ASMManager and thin‑deployer reported “Failed to update OS password for node.”
• Other pods (asmmanager, alcm) relied on the same stale cache.

• Clear credential caches

  Restart the PowerFlex pods on the affected nodes in quick succession:

  • thin-deployer 

    kubectl delete pod -n powerflex $(kubectl get pods -n powerflex|grep 'thin-deployer'|awk '{print $1}') 

  • asmmanager 

    kubectl delete pod -n powerflex $(kubectl get pods -n powerflex|grep -i asmm|awk '{print $1}')  

  • alcm

    kubectl delete pod -n powerflex $(kubectl get pods -n powerflex|grep -i alcm|awk {'print $1'})

• Synchronize ESXi root credentials

  Use the PowerFlex Resources tab to perform a two‑step password “flip”:

  1. Run Update Password to set a temporary password (e.g., esxi_temp).
  2. Immediately run Update Password again to revert to the original credential (e.g., ESXi_root_original) used by other ESXi nodes within the RG to keep the password uniform across all the hosts

  This two-step flip process forces the PowerFlex database to propagate the correct password to all components.'

• Re‑trigger upgrade jobs

• •  Consider using the Resource tab to do the upgrades, at least with the first node or two before trying from the RG.
  • If it's successful from the resource tab and not the RG then the RG will need to be removed and re-added 

• Repeat for remaining nodes

  Apply the same double‑password update and re‑trigger upgrades for all other nodes in the affected Resource Group.