PowerFlex: MDM Disconnections And Possible Switch Over When Running A Tanium Plugin

Scenario

MDM (Primary/Secondary and TBs) is disconnecting from each other, simultaneously that we see the failed initiation of the Tanium security auditing service.

Symptoms

From /var/log/messages: 

May 26 05:44:02 na39tp3siob01 systemd: Starting Security Auditing Service...
May 26 05:44:02 na39tp3siob01 auditd[28400]: Started dispatcher: /sbin/audispd pid: 28402
May 26 05:44:02 na39tp3siob01 auditd[28400]: Init complete, auditd 2.8.5 listening for events (startup state enable)
May 26 05:44:02 na39tp3siob01 audispd: audispd initialized with q_depth=32768 and 1 active plugins
May 26 05:44:02 na39tp3siob01 audispd: plugin /opt/Tanium/TaniumClient/TaniumAuditPipe terminated unexpectedly
May 26 05:44:02 na39tp3siob01 audispd: plugin /opt/Tanium/TaniumClient/TaniumAuditPipe was restarted
May 26 05:44:02 na39tp3siob01 audispd: plugin /opt/Tanium/TaniumClient/TaniumAuditPipe terminated unexpectedly
May 26 05:44:02 na39tp3siob01 audispd: plugin /opt/Tanium/TaniumClient/TaniumAuditPipe was restarted
May 26 05:44:02 na39tp3siob01 audispd: plugin /opt/Tanium/TaniumClient/TaniumAuditPipe terminated unexpectedly
May 26 05:44:02 na39tp3siob01 audispd: plugin /opt/Tanium/TaniumClient/TaniumAuditPipe was restarted
May 26 05:44:02 na39tp3siob01 augenrules: /sbin/augenrules: No change
May 26 05:44:02 na39tp3siob01 audispd: plugin /opt/Tanium/TaniumClient/TaniumAuditPipe terminated unexpectedly
May 26 05:44:02 na39tp3siob01 audispd: plugin /opt/Tanium/TaniumClient/TaniumAuditPipe was restarted
May 26 05:44:02 na39tp3siob01 audispd: plugin /opt/Tanium/TaniumClient/TaniumAuditPipe terminated unexpectedly
May 26 05:44:02 na39tp3siob01 audispd: plugin /opt/Tanium/TaniumClient/TaniumAuditPipe was restarted
May 26 05:44:02 na39tp3siob01 audispd: plugin /opt/Tanium/TaniumClient/TaniumAuditPipe terminated unexpectedly
May 26 05:44:02 na39tp3siob01 audispd: plugin /opt/Tanium/TaniumClient/TaniumAuditPipe was restarted
May 26 05:44:02 na39tp3siob01 audispd: plugin /opt/Tanium/TaniumClient/TaniumAuditPipe terminated unexpectedly
May 26 05:44:02 na39tp3siob01 audispd: plugin /opt/Tanium/TaniumClient/TaniumAuditPipe was restarted
May 26 05:44:02 na39tp3siob01 audispd: plugin /opt/Tanium/TaniumClient/TaniumAuditPipe terminated unexpectedly
May 26 05:44:02 na39tp3siob01 audispd: plugin /opt/Tanium/TaniumClient/TaniumAuditPipe was restarted

From MDM events:

na39tp3siob01.nam.nsroot.net:2021-05-26 05:44:01.961000:0101318:MDM_CLUSTER_LOST_CONNECTION      WARNING  The MDM, na39tp3sioc02 (ID 7bc88b3f51ede2c3), has lost connection to the cluster. 
na39tp3siob01.nam.nsroot.net:2021-05-26 05:44:01.963000:0101319:MDM_CLUSTER_LOST_CONNECTION      WARNING  The MDM, na39tp3sioc01 (ID 3a9bc530791fb671), has lost connection to the cluster. 
na39tp3siob01.nam.nsroot.net:2021-05-26 05:44:01.964000:0101320:MDM_CLUSTER_LOST_CONNECTION      WARNING  The MDM, na39tp3sioa01 (ID 4833f0cf19354380), has lost connection to the cluster. 
na39tp3siob01.nam.nsroot.net:2021-05-26 05:44:02.039000:0101321:MDM_CLUSTER_LOST_CONNECTION      WARNING  The MDM, na39tp3siob02 (ID 7ede01055966dd44), has lost connection to the cluster. 
na39tp3siob01.nam.nsroot.net:2021-05-26 05:44:02.090000:0101322:MDM_CLUSTER_CONNECTED            INFO     The MDM, na39tp3sioc02 (ID 7bc88b3f51ede2c3), connected after 360ms 
na39tp3siob01.nam.nsroot.net:2021-05-26 05:44:02.090000:0101323:MDM_CLUSTER_CONNECTED            INFO     The MDM, na39tp3sioc01 (ID 3a9bc530791fb671), connected after 360ms 
na39tp3siob01.nam.nsroot.net:2021-05-26 05:44:02.090000:0101324:MDM_CLUSTER_CONNECTED            INFO     The MDM, na39tp3sioa01 (ID 4833f0cf19354380), connected after 360ms 
na39tp3siob01.nam.nsroot.net:2021-05-26 05:44:02.090000:0101325:MDM_CLUSTER_CONNECTED            INFO     The MDM, na39tp3siob02 (ID 7ede01055966dd44), connected after 360ms

Impact

There should be no impact on the data service, however, the system is at risk of losing the cluster quorum if there is an additional failure for a 3_node cluster or two additional failures for a 5_node cluster.

Root Cause

From a PowerFlex perspective, the root cause is the Tanium process trying to initiate a security check. The issue is being investigated with Tanium and this KB should be updated once an answer will be provided.

For more info about Tanium: Tanium: Autonomous IT, endpoint intelligence and control 

Workaround

Disable the Tanium plugin.

Impacted Versions

All PowerFlex versions

Fixed In Version

Not a PowerFlex issue