I have checked the clocks following the troubleshooting guide for authentication issues. They are within microseconds of each other and within milliseconds of the DCs.
I did notice one irregularity when running isi_gather_auth_info script on individual nodes. On node one the script completed without errors. On nodes two and three the command "isi auth ads spn list --domain <the.same.domain>" errored out with "The username or password entered is invalid" after about a minute. Other similar commands in the script that gathered domain information completed without issue for all three domains.
Is it possible this is an indicator of the underlying problem ? Could nodes 2 and 3 be using the wrong credentials ?
Try the following:
isi_for_array -n 2-3 /usr/likewise/bin/lwsm refresh lsass
isi auth ads spn check <provider-name>
and in case there are issues shown, look further how things can be fixed:
isi auth ads spn fix --help
I ran the command and it completed with "refreshing service: lsass".
The issue still remains on nodes 2 & 3. The command "isi auth ads spn list --domain" does return all SPNs for two of the three domains, but still errors out on one domain with "The username or password entered is invalid".
They still do not list any ADS providers in "isi auth status" and they cannot authenticate domain users to their shares.
Node 1 lists all three ADS providers as online and authenticates users from all three domains to their shares. The command "isi auth ads spn list --domain" returns the SPNs for all three domains.
There are no issues reported on node 1 for any providers.
On nodes two and three there are no issues reported for two of the three providers, but the third provider errors out with "The username or password entered is invalid".
The Webui displays the zones differently, depending on whether you are logged into node one or nodes two and three. I will try to insert the screen captures into another reply. Last time I tried it crashed my IE and lost the reply text I had already typed in.
Finally found the fix !
I could not find anything in the cluster configuration that was different from other clusters I have running in multiple zones and untrusted domains so I turned my attentions to the network configuration. We have both GigE interfaces on each node aggregated with ext-1 and ext-2 configured as members of an LACP port channel on the Cisco switch. We compared the configurations of each port channel and found the only difference between node 1 port channel and nodes 2 & 3 port channels was the frame size or MTU. Node 1 port channel was set for jumbo frames with an MTU of 9000. Nodes 2 & 3 port channels were at the default 1500. We changed the MTU on nodes 2 & 3 port channels to 9000 and they began participating in all domains.
All three nodes are now participating in the smartconnect load balancing and can now authenticate domain users to their shares in each zone.
Thanks to Peter, Phil and Alan for all your support while I fought this issue. This is one for the books !
mine was a little different. "isi auth ads list" showed provider name but no status or site. Modified provider and added domain controller and it started working "isi auth ads modify --provider-name=** --domain-controller=**"