Dell Unity: uDoctor Scheduled Tasks Trigger High CPU Utilization
Summary: A specific Service Data Collection (SDC) plugin used by APEX AIOps may intermittently trigger multiple PostgreSQL (PSQL) queries during uDoctor scheduled tasks, resulting in high CPU utilization on the Primary Storage Processor for approximately 15 minutes. This issue does not affect manually generated Service Data Collections and is resolved in Unity OE version 5.0.3. ...
Symptoms
CPU utilization on the primary Storage Processor reaches 90-100% at the same time each day and remains high for around 15 minutes.
Cause
The APEX AIOps team has identified an issue where a specific Service Data Collection (SDC) plugin may intermittently trigger multiple PostgreSQL (PSQL) commands.
When this occurs, the resulting PSQL queries can cause significant CPU utilization on the Primary Storage Processor (SP), which is responsible for running uDoctor tasks.
Resolution
This issue is resolved in Unity OE version 5.0.3, which includes a code fix to prevent the excessive CPU utilization caused by the affected uDoctor scheduled tasks.
For additional information about Dell Unity OE versions and release schedules, refer to the Dell Unity OE Matrix knowledge article.
Additional Information
/spX/EMC/backend/log_shared/EMCSystemLogFile.log
service@spb:~# tailf 00_emc_backend_log_shared/EMCSystemLogFile.log | grep -i udoctor
" 2020-02-16 T 13:03:12 .155Z" "Array_spb@Serial_Number" "Kittyhawk_safe" "9134" "unix/spb/root" "INFO" "1:115000d" :: " Internal Information only. PSM File "udoctor" Deleted. " :: Category=System Component=psm TimeZone=UTC
"2020-02-17 T 13:03:12 .944Z" "Array_spb@Serial_Number" "Kittyhawk_safe" "9134" "unix/spb/root" "INFO" "1:115000d" :: " Internal Information only. PSM File "udoctor" Deleted. " :: Category=System Component=psm TimeZone=UTC
UEMCLI Metrics (Historical):
service@spb:~# uemcli -no /metrics/value/hist -path sp.*.cpu.summary.utilization show -interval 60 -from "2020-02-16 12:59:00" -to "2020-02-16 13:59:00"
Timestamp |SP |summary
| |CPU
| |Util %
--------------------+----------+-------
2020-02-16 13:02:00 |spa | 13.93
|spb | 66.47
2020-02-16 13:03:00 |spa | 12.49
|spb | 90.96
2020-02-16 13:04:00 |spa | 12.94
|spb | 92.63
2020-02-16 13:05:00 |spa | 11.91
|spb | 93.29
2020-02-16 13:06:00 |spa | 5.072
|spb | 92.28
2020-02-16 13:07:00 |spa | 2.497
|spb | 90.06
2020-02-16 13:08:00 |spa | 2.452
|spb | 92.38
2020-02-16 13:09:00 |spa | 2.527
|spb | 95.68
2020-02-16 13:10:00 |spa | 2.441
|spb | 98.36
2020-02-16 13:11:00 |spa | 2.845
|spb | 97.170
2020-02-16 13:12:00 |spa | 2.466
|spb | 97.096
2020-02-16 13:13:00 |spa | 2.479
|spb | 97.732
2020-02-16 13:14:00 |spa | 2.469
|spb | 98.023
2020-02-16 13:15:00 |spa | 2.362
|spb | 96.391
2020-02-16 13:16:00 |spa | 2.697
|spb | 65.053
service@spb:~# uemcli -no /metrics/value/rt -path sp.*.cpu.summary.utilization show -interval 10
Timestamp |SP |summary
| |CPU
| |Util %
--------------------+----------+-------
2020-02-17 13:03:20 |spa | 0.929
|spb | 90.810
2020-02-17 13:03:30 |spa | 1.928
|spb | 90.980
2020-02-17 13:03:40 |spa | 2.417
|spb | 90.537
2020-02-17 13:03:50 |spa | 1.912
|spb | 91.601
2020-02-17 13:04:00 |spa | 2.124
|spb | 93.625
2020-02-17 13:04:10 |spa | 3.828
|spb | 92.191