PowerScale OneFS: Data Unavailability Following Rollback to Versions Before OneFS 9.5.0

OneFS must be upgraded from versions prior to OneFS 9.5.0 to versions 9.5.0 or later.

After rollback, there is authentication-based data unavailability; unable to list or view domain users from the cluster and users are unable to open Shares.

The machine account password is what the cluster uses when communicating with domain controllers.

On OneFS version 9.5.0 and later, the machine account password is stored in Key Manager, previous versions of OneFS store the password in the pstore. 

In OneFS 9.5, if the cluster machine account password is updated based on the Machine Password Lifespan setting on the provider, it is only updated in the Key Manager, and not in the pstore.

If the version is rolled back to a version before OneFS 9.5.0, the machine account password that is presented to the domain controllers is now incorrect as we are forced to use the pstore password on older versions.

lsass logs show the following entries indicating that there are issues with the machine account password:

2023-08-10T11:52:23.435858+00:00 <30.4> CVSISILON1-1(id1) lsass[39396]: [LwKrb5GetTgtImpl /b/mnt/src/isilon/fsp/lwadvapi/threaded/krbtgt.c:247] KRB5 Error code: -1765328360 (Message: Preauthentication failed)
2023-08-10T11:52:23.435899+00:00 <30.3> CVSISILON1-1(id1) lsass[39396]: [lsass] Refresh TGT with new password failed
2023-08-10T11:52:23.435907+00:00 <30.4> CVSISILON1-1(id1) lsass[39396]: [lsass] Old password expired, not using it
2023-08-10T11:52:23.436381+00:00 <30.4> CVSISILON1-1(id1) lsass[39396]: [lsass] Failed to enumerate trusts at domain.com (error 31)
2023-08-10T11:52:23.436452+00:00 <30.3> CVSISILON1-1(id1) lsass[39396]: [lsass] Fatal error enumerating trusts for domain domain.com Error was ERROR_GEN_FAILURE (31)

To see when the machine account password was last updated, check the dword.UnixLastChangeTime.value for the provider in the pstore:

/ifs/.ifsvar/pstore.gc

This can also be viewed on the domain controller side under the properties of the machine account.

To address this issue, remove the cluster from the domain, delete the machine account from the domain controllers, and rejoin. This creates a new machine account password and access is restored.

Before leaving and rejoining the domain, it is a good practice to backup the user mappings.

isi auth mapping dump --file=/ifs/data/Isilon_Support/mapping_bakup