Isilon: Authentication over FQDN fails, error "KRB5KRB_AP_ERR_MODIFIED"

Authentication over FQDN fails with the following error message in lsassd.log:
 
This message indicates that; Isilon is not able to recognize the client requesting for one of the services which Isilon has to offer. This will cause authentication to fail over Kerberos because the client will not a valid ticket which Isilon can recognize and de-crypt. 

The same message can be interpreted over network traces, when you see Isilon [10.17.0.154] sending out "KRB5KRB_AP_ERR_MODIFIED" packet back to the client [10.106.12.162] for a session setup request:

This can happen for several reasons, but the most common are listed below:

• There is an account with the same SPN within the forest. Sometimes the KDC will give an error back of KRB_S_PRINCIPAL_UNKNOWN, but there are instances where it will give a Kerberos ticket that the service [In our case it is the Isilon] cannot de-crypt and thus get a KRB5KRB_AP_ERR_MODIFIED.

• The Service Principal Name is on the wrong Active Directory account (Computer or User). This is again a case of duplicate SPN.

• The Active Directory account that is running the service has updated / changed its password and you are experiencing the problem because of an Active Directory Replication Latency or Active Directory Replication problem.

Here the client is trying to access an Isilon share using the FQDN "mixed.isilon.com".

Now let's take a look at the network trace of this attempt.





1. We see proper name resolution, for "mixed.isilon.com" and the DNS server response back with the IP Address of 10.17.0.156 (frames 5042 & 5049)

2. The machine then gets a TGT from the domain controller (see the AS-REQ and AS-REP) (frames 5082 & 5093)

3. The machine then requests and gets a Service Ticket for "mixed.isilon.com" (frames 5101 & 5104). As you can see below, the machine was asking for a Kerberos ticket of "mixed.isilon.com"


CName stands for Client name and SName stands for Service name




4. The machine then goes back to the Isilon and attempts to authenticate using the Kerberos ticket that it just got from the domain controller (frames 5107 & 5111). During the authentication the Isilon responds back with KRB5KRB_AP_ERR_MODIFIED (frame 5111).


If you see from  step 3, the client is getting the Kerberos ticket from realm "ISILON.COM" while the cluster is joined to its trusted domain in this scenario to "CORP.COM". This means there is a duplicate SPN registered in ISILON.COM and Isilon can't de-crypt the ticket presented by the client since the cluster uses CORP.COM to authenticate; this causes authentication to fail.

Example of the presence of duplicate SPN:

SPN's listed in ISILON.COM and CORP.COM respectively 
 

Check for trusted domains and duplicate SPN's on the domain the Isilon is joined to and the trusted domain:

- To check the trusted domains:

# isi auth ads trusts list --provider-name=<Domain name cluster is joined to>

- To list the SPN's present in a domain:
 
If there are duplicate SPN's present, and only the primary domain is used for authentication, delete the SPN following the KB OneFS: How to find duplicate Service Principal Names (SPNs) in Active Directory preventing SMB clients from authenticating to cluster.

Useful link: Register a Service Principal Name for Kerberos Connections