Avamar and Embedded Service Enabler (ESE): Kernel error events

The Avamar server is on version 19.10 or later which supports Dell Connectivity with Embedded Service Enabler (ESE).

Kernel error events reported by Avamar may look similar to the following output.

Event Code = 1
Event Date/Time = 10/28-06:42:15.00000
Event Type = ERROR
Event Severity = PROCESS
Event Summary = kernel error: Oct 28 06:42:08 avamarlab ese_launcher[24557]: 24557 2024-10-28 06:42:08,250 App Poller ERROR   DellESE.ese.remote_support.app_polling.app_polling_interface_connection LN: 67 Interface details not found
Software Source = DPN:Unknown
Hardware Source = avamarlab Administrator Server = avamarlab Administrator Server version = 19.10.0-166_HF338862 Avamar Server = avamarlab Avamar Server systemid = 1589815007@00:50:56:8E:1E:BD Avamar Server productId = null Avamar Server version = 19.10.0-166 Backup & Recovery Manager Url = avamarlab Data Domain System = ddlab Data Domain System OS version = 7.7.5.11-1046187 Data Domain System serial number = AUDVSDBC4XDDRW Data Domain System = ddlab Data Domain System OS version = 7.7.5.1-1040473 Data Domain System serial number = AUDVSDBC4XDDRW Event Data = Entry key = date Entry value = 2024/10/28 Entry key = code Entry value = 0001 Entry key = time Entry value = 05:42:15.09665 UTC Entry key = thread Entry value = log-messages:109 Entry key = type Entry value = ERROR Entry key = message
Entry value = kernel error: Oct 28 06:42:08 avamarlab ese_launcher[24557]: 24557 2024-10-28 06:42:08,250 App Poller ERROR   DellESE.ese.remote_support.app_polling.app_polling_interface_connection LN: 67 Interface details not found
Entry key = nodeid
Entry value = 0.0
Entry key = requestor
Entry value = <requestor
 domain="/"
 product="MCS"
 role="Administrator"
 user="root" />

Embedded Service Enabler (ESE) is a product-independent python software which can be configured on Avamar to send telemetry information to Dell Technologies.

ESE can send telemetry data with direct connections from the Avamar server to the Dell backend or by using Secure Connect Gateway (SCG).

Avamar configures ESE in systemd with the "ese_launcher" service, which by default logs to the following file.

/var/log/messages

The configuration which sends the logs there is rsyslog.

/etc/rsyslog.conf

Because of the following line in the rsyslog configuration, it logs mostly everything except mail and news in systemd to the messages file location.

*.*;mail.none;news.none                 -/var/log/messages

Whenever there is an error in the ESE service, it logs to the messages location and the server log scanner configuration marks it as a kernel error.

Marking the ESE errors as kernel errors is not accurate, as ESE is not part of the kernel and has its own logging at the following location.

/opt/ese/var/log/*

Since the ESE service has its own logging implementation, troubleshooting the ESE service should happen from the ESE log folder, not the messages files.

Stop the logging of the ESE service to the messages file and false kernel error reporting for the service by changing the rsyslog configuration.

Take a backup of the rsyslog configuration file.

cp -p /etc/rsyslog.conf /etc/x-rsyslog.conf

As the root user, run the following command:

grep -q "if \$programname == 'ese_launcher' then stop" /etc/rsyslog.conf || sed -i '1i if \$programname == '\''ese_launcher'\'' then stop' /etc/rsyslog.conf

Restart the rsyslog service.

systemctl restart rsyslog