Isilon: Using the PowerScale OneFS isi_auth_expert command to manage authentication issues

Not required

Not required

Introduction

Administrators can run the isi_auth_expert command to examine a PowerScale OneFS cluster's authentication environment. This can help ensure that it is properly configured and to identify conditions that could be causing data access latency as a result of authentication issues.

The isi_auth_expert command runs a series of tests, including network and port connectivity and latency, binding, and clock skew. These results can be used to isolate a problematic configuration or network path that is causing data access issues.

Individuals may want to run this tool:

• When existing or new users experience latency while connecting to a share, or are prompted to enter login credentials when accessing data
• When the cluster is reporting events regarding Active Directory or Lightweight Directory Access Protocol (LDAP) offline status
• After modifying the authentication provider setup
• After configuration changes have affected network paths between a cluster and its authentication providers

Instructions

To run the isi_auth_expert command, do the following:

isi_auth_expert

Individuals can also run the command with one or more of the options listed in the table below:
 

Option	Explanation
-h, --help	Show the syntax for this command
-h, --debug	Display debugging messages
-v, --verbose	Enable verbose (more robust) output
--no-color	Disable colored output

Example output:

wcvirt1-1# isi_auth_expert

Checking authentication process health ... done
Checking LDAP provider 'ldaptest' server connectivity ... done
Checking LDAP provider 'ldaptest' base dn ... done
Checking LDAP provider 'ldaptest' object enumeration support ... done
Checking LDAP provider 'ldaptest' group base dn ... done
Checking LDAP provider 'ldaptest' user base dn ... done
  [ERROR] The configured base user dn 'ou=dne,dc=isilon,dc=com' in LDAP provider
  'ldaptest' was not found on LDAP server ldaptest.west.isilon.com.
Checking AD provider 'WMC-ADA.WEST.ISILON.COM' DC connectivity ... done
Checking AD provider 'WMC-ADA.WEST.ISILON.COM' auth related ports ... done
  [ERROR] Failed to establish a connection to the AD domain controller wmc-ada-dc1
            .wmc-ada.west.isilon.com on port 3268.

Implemented tests

When running isi_auth_expert command, the following checks are performed:

Process checks

This test confirms that the authentication-related processes (lsass, lwio, and netlogon) are running. If any of the processes are not running, an error is returned.

Active Directory

The following section describes the tests that the isi_auth_expert command performs for each Active Directory (AD) provider.

• Check Domain Controller connectivity
  Determine whether the cluster has basic network connectivity to at least one domain controller (DC) in the AD domain.

Check DC ports
Verify that for every DC, the cluster can connect to the AD-related ports, and that the ports are accepting connections.
 

Port	Explanation	AD Usage	Traffic Type
88	Port 88 is used for Kerberos authentication traffic.	User and Computer Authentication, Forest Level Trusts	Kerberos
139	Port 139 is used for NetBIOS and NetLogon traffic.	User and Computer Authentication, Replication	DFSN, NetBIOS Session Service, NetLogon
389	Port 389 is used for LDAP queries.	Directory, Replication, User and Computer Authentication, Group Policy, Trusts	LDAP
445	Port 445 is used for replication.	Replication, User and Computer Authentication, Group Policy, Trusts.	SMB, CIFS, SMB2, DFSN, LSARPC, NbtSS, NetLogonR, SamR, SrvSvc
3268	Port 3268 is used for global catalog LDAP queries. (used if the global catalog in the AD provider is enabled)	Directory, Replication, User and Computer Authentication, Group Policy, Trusts	LDAP GC

LDAP

The following section describes the tests that the isi_auth_expert command performs for each LDAP provider.

LDAP connectivity

• Check LDAP server connectivity by making an anonymous LDAP bind and checking the results.

LDAP enumerated objects support

• Confirm that each LDAP server supports enumerated objects by checking the LDAP servers' supported controls. OneFS requires either the paged results controls or both the virtual list view and server-side sorting controls.

Validate configured base-dn

• Perform a test query against the configured base-dn to ensure configuration compatibility with the LDAP server.

Validate configured user-base-dn

• Perform a test query against the configured user-base-dn to ensure configuration compatibility with the LDAP server.

Validate configured group-base-dn

• Perform a test query against the configured group-base-dn to ensure configuration compatibility with the LDAP server.

Other checks and parameters in OneFS 7.2.1.5 and later

The following checks were added in OneFS 7.2.1.5.

• Active Directory

  • Domain controller latency check
  • Clock Skew and latency check
  • Global Catalog service for user (SFU) check
• LDAP - User check
• Kerberos - Service Principal Name (SPN) checks for SmartConnect zones and aliases

The isi_auth_expert command can calculate two types of latencies: Ping latency and LDAP latency for all the domain controllers. If the clock skew is less than five minutes, the command returns: "There is minimal or no skew between the AD provider and your machine."

The following parameters were also added.
 

Option	Explanation
--ldap-user	Checks the LDAP provider for a specified user
--sfu-user	Checks the Active Directory Global Catalog for a specified user
--admin-creds	Supply the credentials that are required when checking the Active Directory Global Catalog.

LDAP user attribute check

To run the LDAP user attribute check, you must run the isi_auth_expert command with the --ldap-user=<user> parameter where <user> is the user you want to check. The username has to be of the form "plain name" for the search to work. The LDAP user attribute check connects to an LDAP server and queries it for the specified user. We can then check the results of the query to ensure that the user has all necessary attributes necessary for authentication in any domain.

Active Directory Global Catalog SFU check

A global catalog server is a domain controller that has information about the domain it is associated with and all the other domains in the forest. Like an LDAP server, the global catalog has a list of data associated with the domain it control in addition to a partial copy of the data it gets from other domain controllers. If it does not have all the data that the domain controllers are sharing, there could be authentication issues.

To run the Active Directory Global Catalog SFU check, you must run the isi_auth_expert command with the --sfu-user=<user> and --admin-creds="[('<Domain>', '<User>', '<password>')]" parameters where <user> is the SFU user you want to check and "[('<Domain>', '<User>', '<password>')]" are the credentials the isi_auth_expert command must provide to perform the Global Catalog lookup in the domain controller. We have the following limitation when checking the global catalog: You must provide administrator credentials.

Server principal name (SPN) check

SPNs may cause authentication failures if they are not present when you join a Kerberos provider, or if you change the name of a SmartConnect zone. The isi_auth_expert command determines if SPNs are missing, stale, or incorrect. This feature runs automatically whenever the isi_auth_expert command is run.

This feature is used to check for missing SPNs in both Kerberos providers and also in SmartConnect zones. The command collects all the SPNs associated with the providers and SmartConnect zones and ensures that the required SPNs are present.

If you are using SmartConnect aliases, it also checks against those aliases. You can use the isi auth ads spn or isi auth krb5 spn commands to list, check, or fix reported missing SPNs.