isi_audit_syslog is not starting. No error messages are found in logs.

The most likely cause is that the AUDIT logs were deleted and not re-created fully after an AUDIT log clean-up process as described in KB OneFS 7.1 and later: How to remove audit log files.

In order to determine why isi_audit_syslog is not starting, DEBUG logging must be enabled on one of the nodes affected.

It is recommended that an SR be opened with DELL EMC Isilon Support, quoting this KB so that support personnel can analyse the DEBUG logs for a cause why isi_audit_syslog has failed to start.

Be aware that syslog forwarding is not necessarily needed. This is dependent on the customer configuration. As the relevant documentation (Administration Guides and White Papers) reveal, the cluster can be configured in a number of ways to forward audit events. One way is to configure CEE and this is done by enabling "config-auditing". This suffices to send events via the CEE framework to CEE servers, and then on to the Auditing Software. In addition, these protocol audit events can also be sent to a remote syslog server. This is described and documented in KB OneFS: How to configure remote logging from a cluster to a remote server (syslog forwarding). But isi_audit_syslog is not required for protocol auditing to a CEE server. These are two different configuration options.

As long as events are being forwarded from the nodes via CEE, isi_audit_syslog is not needed. To check if events are being forwarded, execute the following command on a node where isi_audit_syslog is not running:

isi_audit_progress -t protocol CEE_FWD

Output indicating CEE events are being forwarded will look like the following:
mycluster-2:  Last consumed event time: '2018-02-15 16:24:44'
mycluster-2:  Last logged event time:   '2018-02-15 16:24:45'

If the cluster was configured to forward to a syslog server, then isi_audit_syslog should be running. But it is not necessarily the case, that CEE is failing if that daemon is not started. These points should be reviewed before opening a support case. If remote syslog has not been configured as stated in the KB above, there is no need to have that enabled to support the functionality of CEE protocol auditing.

isi audit settings global modify --config-auditing-enabled yes --config-syslog-enabled yes
cluster-1# isi audit settings global view
Protocol Auditing Enabled: Yes
Audited Zones: System
CEE Server URIs: https://dell.to/4ifuZ6M
Hostname: https://dell.to/41Adk4n
Config Auditing Enabled: Yes
Config Syslog Enabled: Yes

isi audit settings global modify --config-auditing-enabled yes --config-syslog-enabled yes
cluster-1# isi audit settings global view
Protocol Auditing Enabled: Yes
Audited Zones: System
CEE Server URIs: https://dell.to/4ifuZ6M
Hostname: https://dell.to/41Adk4n
Config Auditing Enabled: Yes
Config Syslog Enabled: Yes

Those settings are not needed unless a remote syslog server is actually configured. CEE forwarding of audit events works without enabling this. If there is no remote syslog server configured, these yellow high-lighted options should be configured as "no".

The following command is also helpful to determine the number of events being sent per second:

mycluster-1# isi statistics query current --nodes=all --stats=node.audit.cee.export.rate
   Node  node.audit.cee.export.rate
-----------------------------------
      1                  893.000000
      2                   21.200000
      3                   37.400000
      4                    8.400000
      5                   59.400000
      6                   66.800000
      7                   75.800000
      8                  349.800000
      9                  831.000000
     10                   28.800000
average                  237.160000
-----------------------------------
Total: 11

In the example above, Node 1 is sending 893 events per second to the CEE servers configured. The events are sent one after the other, in a round robin fashion to ALL CEE servers that have been configured (as long as the CEE servers are reachable). Node 2 is sending 21.2 events per second, and so on.

NOTE that in OneFS v8.0.1.x+, events are sent in parallel, resulting in quite a performance improvement for the CEE auditing feature.

In order to determine why isi_audit_syslog is not starting, DEBUG logging must be enabled on one of the nodes affected. To resolve the issue on one node, do the following steps. Be sure to execute the steps quickly, one after the other, as the DEBUG logging will generate a log of logs, especially if CEE forwarding is working.

1. Enable DEBUG logging for the AUDIT subsystem as follows. This will generate a LOT of logs if the cluster is busy sending events via CEE to a server. This will enable DEBUG logging for the /var/log/isi_audit_cee.log file.

Do this on one node where isi_audit_syslog is not starting.

pkill -SIGUSR1 isi_audit_cee

2. Try to manually start the isi_audit_syslog process as follows.

/usr/libexec/isilon/isi_audit_syslog /usr/bin/isi_audit_syslog

3. Stop DEBUG logging as follows. DO NOT FORGET THIS STEP.

pkill -SIGUSR1 isi_audit_cee

4. Review the isi_audit_cee.log files in /var/log. Depending on the activity level, there could be several wrapped logs to review before you find the point where the attempt to start isi_audit_syslog occurred.

The causative log entry will look something like this:
2018-02-15 18:04:33 mycluster-5 isi_audit_syslog[8536][0x817f4fc00]: Opening audit/logs/node005/protocol/00000000: No such file or directory

You will have to hunt for this in the DEBUG logs - there are a lot of logs!

5. Touch that file and restart isi_audit_syslog manually.

touch /ifs/.ifsvar/audit/logs/node005/protocol/00000000

/usr/libexec/isilon/isi_audit_syslog /usr/bin/isi_audit_syslog

6. Control that isi_audit_syslog has started on the node

ps -auwx | grep audit | grep -v grep

7. Repeat for other nodes as needed