PowerScale: OneFS: Time on the cluster is not synchronized with Active Directory

• Windows clients are unable to authenticate to the cluster.
• Alerts are sent about time synchronization errors.
• The /var/log/messages file contains entries similar to the following:
   

  2014-04-07T04:15:01-03:00 <4.5> example-1(id1) lsass[60726]: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Clock skew too great)
  2014-04-07T07:41:28-03:00 <30.3> example-3(id3) lsass[59264]: [lsass] Resetting schannel due to status STATUS_TIME_DIFFERENCE_AT_DC (0xc0000133) while authenticating user 'LAB\jdoe'
  2014-04-07T07:41:28-03:00 <30.3> example-3(id3) lsass[59264]: [lsass] AD_NetrlogonOpenSchannel(dc1.example.com) failed with 3221225779 (0xc0000133) (symbol: 'STATUS_TIME_DIFFERENCE_AT_DC')
  2014-04-07T07:41:28-03:00 <30.3> example-3(id3) lsass[59264]: [lsass] Network error connecting to EXAMPLE.COM. Error code: 40134 (symbol: LW_ERROR_RPC_ERROR)
  2014-04-07T07:41:28-03:00 <30.5> example-3(id3) lsass[59264]: [lsass] Domain 'example.com' is now offline

In OneFS, time synchronization is managed in two ways:

• Internal time synchronization (within the cluster nodes)

• External time synchronization (outside of the cluster)

When a cluster joins an AD domain, it uses Server Messenger Block (SMB) for external time sync by default, accurate to the second. Internally, Network Time Protocol (NTP) manages time sync with millisecond precision.

To prevent large time corrections when syncing with an AD domain controller, configure the controller to use NTP instead of SMB for greater precision. Significant adjustments can cause performance issues, such as Kerberos tickets appearing expired and triggering authentication errors.

Verify the time skew between the cluster and the domains:

# isi_for_array -s date
# isi_for_array -s /usr/likewise/bin/lw-get-dc-time <domain_name>

Confirm that time synchronization is set up properly:

Perform all the steps below to configure the AD domain controllers and to ensure that the cluster synchronizes properly with the AD controllers.

Configure NTP on the Active Directory servers:

For Windows 2003 - 2012 Server, configure NTP to reference your NTP servers. For details, refer to the Windows Time Services Tools and Settings page on the Microsoft TechNet website.

NOTE:
Ensure that two AD domain controllers are available for redundancy if one becomes unreachable. Time must follow an accurate hierarchy, either from public NTP servers or an internal source such as a GPS-equipped system.

Verify that the AD domain controllers support NTP and that NTP is not blocked by a firewall or other settings:

1. Open an SSH connection on any node in the cluster and log in using the "root" account.
2. Run the following commands, where <ipaddr1> and <ipaddr2> are the IP addresses of the first and second AD domain controllers:
   

   ntpdate -q -u <ipaddr1>
   ntpdate -q -u <ipaddr2>

If the AD domain controllers support NTP, the output for each command looks similar to the following:

server <ipaddr1>, stratum 1, offset 0.427215, delay 0.04138
11 May 15:45:46 ntpdate[79498]: adjust time server <ipaddr1> offset 0.427215 sec

If the AD domain controllers do not support NTP, the output looks similar to the following, in which case, you should contact the domain administrator for assistance.

b5-10-3: 11 May 15:49:40 ntpdate[79741]: no server suitable for synchronization found

Set the time on the cluster to match the time on the AD domain controller:

1. Open an SSH connection on any node in the cluster and log in using the "root" account.
2. Run the following command where <ippaddr1> is the IP address of the AD domain controller that you want the cluster to synchronize with:

   isi_for_array -s "ntpdate -u -b <ipaddr1>"

The output looks similar to the following:

example-1: 11 May 15:49:48 ntpdate[79756]: step time server <ipaddr1> offset 0.541754 sec
example-2: 11 May 15:49:48 ntpdate[99580]: step time server <ipaddr1> offset 0.541843 sec
example-3: 11 May 15:49:48 ntpdate[63251]: step time server <ipaddr1> offset 0.480573 sec

Delete the ntp.drift file:

The ntp.drift file is a record of the clock drift of the system clock. The system re-creates this file after it accumulates sufficient data.

1. Open an SSH connection on any node in the cluster and log in using the "root" account.

2. Run one of the following commands from the command line, depending on what version of OneFS you are running:

   isi_for_array -sX "rm -fv /var/crash/ntp.drift"

Configure the cluster to use the AD domain controllers as NTP servers:

1. Open the OneFS web administration interface and do one of the following:

• In OneFS WebUI, click Cluster Management > General Settings > NTP tab.

• For each AD domain controller, enter the IP address in the Server IP or hostname box, then click Add.

2. Click Submit.

Alternatively, you can accomplish this task from the command prompt:

1. Open an SSH connection on any node in the cluster and log in using the "root" account.

2. Run the following command, where <ipaddr1> and <ipaddr2> are the IP addresses of the first and second AD domain controllers:

   isi_ntp_config add server <ipaddr1>
   isi_ntp_config add server <ipaddr2>