PowerScale: Troubleshoot AD Offline and AD Join Issues with lw-get-dc commands

Summary: lw-get-dc commands are useful for investigating and troubleshooting AD issues.

This article applies to This article does not apply to This article is not tied to any specific product. Not all product versions are identified in this article.
Check out other resources

Instructions

Approximately 90% of AD offline or AD join issues are down to one of four different issues or a combination of them. 

  1. DNS
  2. Blocked, unresponsive ports
  3. Unresponsive CLDAP (which is usually a blocked port as well but it uses UDP 389 on which NC is unreliable)
  4. Time Skew

Investigation for all the above are covered in this article. 
 

Note: The catch with these commands is they only run against groupnet0. If you are investigating AD issues on groupnets other than groupnet0, other methods should be employed (See additional info section).

lw-get-dc commands are useful for investigating and troubleshooting AD-related issues.

Use the isi auth ads list command to pull domain and site information to plug into the lw-get-dc commands:

# isi auth ads list
Name        Authentication  Status  Site
-----------------------------------------------------------
PRIMARY.COM Yes             online  ovens
-----------------------------------------------------------
Total: 1

lw-get-dc-list:

This command is used to list DCs. It uses DNS and so is a good way to:

  1. Check if there are any potential DNS issues
  2. Get a list of DCs on the domain or site

To get a full list of DCs on the domain use:

# /usr/likewise/bin/lw-get-dc-list <domain>

Example:

# /usr/likewise/bin/lw-get-dc-list primary.com
Got 5 DCs:
===========
DC 1: Name = 'DC1.primary.com', Address = '192.168.1.150'
DC 2: Name = 'DC2.primary.com', Address = '192.168.1.201'
DC 3: Name = 'DC3.primary.com', Address = '192.168.10.202'
DC 4: Name = 'DC4.primary.com', Address = '192.168.10.203'
DC 5: Name = 'DC5.primary.com', Address = '192.168.10.204'

To query DCs on the same site as Isilon, input the site and domain information from isi auth ads list:

# /usr/likewise/bin/lw-get-dc-list <domain> --site <site>

Example:

# /usr/likewise/bin/lw-get-dc-list primary.com --site ovens
Got 2 DCs:
===========
DC 1: Name = 'DC1.primary.com', Address = '192.168.1.150'
DC 2: Name = 'DC2.primary.com', Address = '192.168.1.201'

lw-get-dc-name:

Ensure using the -force flag to avoid caching. Use the --site flag is you want to query DCs on a certain site. This command generates a CLDAP ping and so is useful to (among other things):

  • Check if CLDAP pings (UDP port 389) are responsive, if this command does not complete it may suggest that UDP 389 between PowerScale and DC is not open.
  • Check PowerScale is receiving the correct site info. 

# /usr/likewise/bin/lw-get-dc-name <domain> --force

Example:

# /usr/likewise/bin/lw-get-dc-name primary.com --force
Printing LWNET_DC_INFO fields:
===============================
dwDomainControllerAddressType = 23
dwFlags = 258556
dwVersion = 5
wLMToken = 65535
wNTToken = 65535
pszDomainControllerName = DC2.primary.com
pszDomainControllerAddress = 192.168.1.201
pucDomainGUID(hex) = CF 93 AC EA 33 5E A9 4F B6 B8 D0 F6 EC 06 61 D1
pszNetBIOSDomainName = PRIMARY
pszFullyQualifiedDomainName = primary.com
pszDnsForestName = primary.com
pszDCSiteName = ovens
pszClientSiteName = ovens
pszNetBIOSHostName = DC2
pszUserName = <EMPTY>

lw-get-dc-time:

Useful for comparing node time to dc time (Time skew check). Also uses CLDAP ping (UDP port 389) 

Example:

# /usr/likewise/bin/lw-get-dc-time primary.com
DC TIME: 2023-12-06 12:47:26 GMT

# date; /usr/likewise/bin/lw-get-dc-time primary.com
Wed Dec  6 12:53:33 GMT 2023
DC TIME: 2023-12-06 12:53:33 GMT

lw-lsa get-status:

This command provides extended details on domains and trusts beyond what is found in logs and other isi auth commands. With this command, it is not required to specify the domain or site.

# /usr/likewise/bin/lw-lsa get-status

Additional Information

To check DNS SRV records on a Groupnet other than groupnet0, you can use the Dig command as follows:

Use isi network groupnet list command to retrieve DNS server IPs.

# dig @<DNS_IP> SRV _ldap._tcp.dc._msdcs.<domain>

Example:

# dig @192.168.1.150 SRV _ldap._tcp.dc._msdcs.primary.com

; <<>> DiG 9.10.0-P2 <<>> @192.168.1.150 SRV _ldap._tcp.dc._msdcs.primary.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 43060
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 3

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4000
;; QUESTION SECTION:
;_ldap._tcp.dc._msdcs.primary.com. IN   SRV

;; ANSWER SECTION:
_ldap._tcp.dc._msdcs.primary.com. 600 IN SRV    0 100 389 DC2.primary.com.
_ldap._tcp.dc._msdcs.primary.com. 600 IN SRV    0 100 389 DC1.primary.com.
_ldap._tcp.dc._msdcs.primary.com. 600 IN SRV    0 100 389 DC3.primary.com.
_ldap._tcp.dc._msdcs.primary.com. 600 IN SRV    0 100 389 DC4.primary.com.
_ldap._tcp.dc._msdcs.primary.com. 600 IN SRV    0 100 389 DC5.primary.com.

;; ADDITIONAL SECTION:
DC2.primary.com. 1200 IN    A       192.168.1.201
DC1.primary.com. 1200 IN    A       192.168.1.150
DC3.primary.com. 1200 IN    A       192.168.10.202
DC4.primary.com. 1200 IN    A       192.168.10.203
DC5.primary.com. 1200 IN    A       192.168.10.204

;; Query time: 1 msec
;; SERVER: 192.168.1.150#53(192.168.1.150)
;; WHEN: Wed Dec 06 14:09:34 GMT 2023
;; MSG SIZE  rcvd: 187

To query against a particular site use:

# dig @<DNS_IP> SRV _ldap._tcp.<site_name>._sites.dc._msdcs.<domain>

Example:

# dig @192.168.1.150 SRV _ldap._tcp.ovens._sites.dc._msdcs.primary.com

; <<>> DiG 9.10.0-P2 <<>> @192.168.1.150 SRV _ldap._tcp.ovens._sites.dc._msdcs.primary.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 61529
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 3

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4000
;; QUESTION SECTION:
;_ldap._tcp.ovens._msdcs.primary.com. IN SRV

;; ANSWER SECTION:
_ldap._tcp.ovens._sites.dc._msdcs.primary.com. 600 IN SRV 0 100 389 DC2.primary.com.
_ldap._tcp.ovens._sites.dc._msdcs.primary.com. 600 IN SRV 0 100 389 DC1.primary.com.

;; ADDITIONAL SECTION:
DC2.primary.com. 1200 IN    A       192.168.1.201
DC1.primary.com. 1200 IN    A       192.168.1.150

;; Query time: 0 msec
;; SERVER: 192.168.1.150#53(192.168.1.150)
;; WHEN: Wed Dec 06 14:11:04 GMT 2023
;; MSG SIZE  rcvd: 218

DC Port Check:
A crucial and often used check when AD is offline is to check if the required ports are responding. This is especially true for DCs on the same site as PowerScale, as it is not uncommon for ports on other sites to be blocked. 

The below test is for the three essential (tcp) ports, 88 (Kerberos), 389 (LDAP) and 445. Other ports can be added as needed GC for example.

The below command is for node 1, change the isi_for_array portion of the below command as required.  

# for ip in $(/usr/likewise/bin/lw-get-dc-list <domain> --site <site> | egrep -v "===========|Got" | awk '{ print $8 }' | sed -e "s/^\'//" -e "s/\'$//"); do for port in 88 445 389; do isi_for_array -n 1 nc -w 2 -vz $ip $port; done; done

Example:

# for ip in $(/usr/likewise/bin/lw-get-dc-list primary.com --site ovens | egrep -v "===========|Got" | awk '{ print $8 }' | sed -e "s/^\'//" -e "s/\'$//"); do for port in 88 445 389; do isi_for_array -n 1 nc -w 2 -vz $ip $port; done; done
clusterone-1: Connection to 192.168.1.201 88 port [tcp/kerberos-sec] succeeded!
clusterone-1: Connection to 192.168.1.201 445 port [tcp/microsoft-ds] succeeded!
clusterone-1: Connection to 192.168.1.201 389 port [tcp/ldap] succeeded!
clusterone-1: Connection to 192.168.1.150 88 port [tcp/kerberos-sec] succeeded!
clusterone-1: Connection to 192.168.1.150 445 port [tcp/microsoft-ds] succeeded!
clusterone-1: Connection to 192.168.1.150 389 port [tcp/ldap] succeeded!
clusterone-1#

Affected Products

PowerScale OneFS
Article Properties
Article Number: 000220198
Article Type: How To
Last Modified: 09 شعبان 1447
Version:  3
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.