CPU load of "base_nas" service in OS10

Hi, thanks for choosing Dell. I'm hoping this may help:

https://dell.to/3jM0KJ4

"util-threshold cpu"

Thanks but I would really prefer to prevent CPU from going high in the first place than to raise alarm thresholds.
In the meantime, did lot of testing and reading and it seems that reload brings CPU load of base_nas service under control (sometimes to 40% sometimes to only 20%) but it looks so random that it is unbelievable.
Also it looks that any connect or disconnect of a port, brings the load back to close to 100%.
At this very moment (after one of the reloads) load is like this on primary VLT node:

1637 root 20 0 2920760 366224 113532 S 60.0 9.1 325:50.28 base_nas

It also seems that while the CPU load is high (95-99%), core switch tends not to let go pings to management of access switches (ping to end devices seems fine) and even breaks totally ssh connections (still over VLAN1) to those (timeout on connection).

Thanks but I do not really want to stop alarms by raising their thresholds but to prevent CPU from going high in the first place.

Anybody knows what is actually this base_nas and how to put it under control?
Read about one firmware bug in earlier versions but this one should have been fixed in ours?

Hi may I advise that you  upgrade the  firmware to the latest 10.5.2.7? Also are you having a performance issue based on this syslog alarms ?

So far noticed random disconnects and re-connects of LAG members (still waiting to see if related), very high latency to access switch management IPs and inability to connect to access switched SSH (login timeout).
Devices are bought used from 3rd party so no update atm...
Can anyone potentially explain what is actually base_nas service and why it would behave like this?

Hi

Updated both nodes to 10.5.2.6 (10.5.2.7 is nowhere to be found).
Unfortunately still the same problem as soon as one touches a port or changes anything:
top - 11:30:14 up 4:07, 1 user, load average: 1.70, 1.96, 1.87
Tasks: 270 total, 1 running, 269 sleeping, 0 stopped, 0 zo...
%Cpu(s): 24.1 us, 16.1 sy, 0.0 ni, 59.3 id, 0.1 wa, 0.0 hi, 0.4 si, 0.0 st
KiB Mem : 4023444 total, 929276 free, 2415148 used, 679020 buff/cache
KiB Swap: 4023440 total, 4023440 free, 0 used. 1354812 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1650 root 20 0 2920904 366596 113128 S 50.0 9.1 107:48.76 base_nas

(started with 23-25% after reload for upgrade and then one port was disconnected...)

Checking other similar cases, I found that 10.5.2.7 contains some fix related to high cpu utilization.

You can check in Digital Locker if you can find this version, or contact support that probably can send you this patch.

Thanks
Marco

I would update to 10.5.2.7 if I could, trust me

What is the issue to retrieve the last firmware version?

I see here that is available in Dell Digital Locker

https://dell.to/3nexuwM

If you don't have access on it, you can contact our technical support to see if they can send you the update.

Thanks

Marco

Hi Marco

As mentioned "Devices are bought used from 3rd party so no update atm..."

BR
Nenad

Sorry without this firmware we cannot help you further. Please check with our technical support.

Thanks

Marco

Hi Marco

NP, thanks anyway!

BR
Nenad

Hi again

We now have  OS10 v10.5.2.7 on both nodes and the problem is still there and easily reproducible:
Normal CPU load on the node is between 20-25% and as soon as one cable is pulled and put back in we have this problem:

<165>1 2021-09-14T09:28:10.407934+02:00 str-sw-core-1 dn_alm 817 - - Node.1-Unit.1:PRI [event], Dell EMC (OS10) %PIM_PURGE_DELAY_STOP_TIMER: PIM detected multicast peer-routing timer stop
<165>1 2021-09-14T09:55:02.545846+02:00 str-sw-core-1 dn_alm 817 - - Node.1-Unit.1:PRI [event], Dell EMC (OS10) %ALM_AUTH_EVENT: Authentication event was raised MESSAGE=pam_unix(sshd:session): session opened for user admin by (uid=0)
<165>1 2021-09-14T09:55:05.046924+02:00 str-sw-core-1 dn_alm 817 - - Node.1-Unit.1:PRI [event], Dell EMC (OS10) %ALM_AUTH_EVENT: Authentication event was raised MESSAGE=pam_unix(sshd:session): session opened for user admin by (uid=0)
<165>1 2021-09-14T10:01:53.010860+02:00 str-sw-core-1 dn_alm 817 - - Node.1-Unit.1:PRI [event], Dell EMC (OS10) %IFM_OSTATE_DN: Interface operational state is down :ethernet1/1/16
<165>1 2021-09-14T10:01:53.156893+02:00 str-sw-core-1 dn_alm 817 - - Node.1-Unit.1:PRI [event], Dell EMC (OS10) %IFM_OSTATE_DN: Interface operational state is down :port-channel16
<165>1 2021-09-14T10:01:53.163327+02:00 str-sw-core-1 dn_alm 817 - - Node.1-Unit.1:PRI [event], Dell EMC (OS10) %LACP_PORT_UNGROUPED: Interface exited port-channel port-channel16 : ethernet1/1/16
<165>1 2021-09-14T10:22:00.483698+02:00 str-sw-core-1 dn_alm 817 - - Node.1-Unit.1:PRI [event], Dell EMC (OS10) %IFM_OSTATE_UP: Interface operational state is up :ethernet1/1/16
<165>1 2021-09-14T10:22:36.172854+02:00 str-sw-core-1 dn_alm 817 - - Node.1-Unit.1:PRI [event], Dell EMC (OS10) %IFM_OSTATE_UP: Interface operational state is up :port-channel16
<165>1 2021-09-14T10:22:36.181728+02:00 str-sw-core-1 dn_alm 817 - - Node.1-Unit.1:PRI [event], Dell EMC (OS10) %LACP_PORT_GROUPED: Interface joined port-channel port-channel16 : ethernet1/1/16
<164>1 2021-09-14T10:24:11.089667+02:00 str-sw-core-1 dn_alm 817 - - Node.1-Unit.1:PRI [alarm], Dell EMC (OS10) %PM_SYS_UTIL_LO: CPU Utilization 1-minute Low Threshold alarm Raised: 78.34 (85-75) %%ACTION=RAISED %%SEQUENCE=7242
<161>1 2021-09-14T10:24:20.409493+02:00 str-sw-core-1 dn_alm 817 - - Node.1-Unit.1:PRI [alarm], Dell EMC (OS10) %PM_SYS_UTIL_HI: CPU Utilization 1-minute High Threshold alarm Raised: 91.32 (85-75) %%ACTION=RAISED %%SEQUENCE=7243
<161>1 2021-09-14T10:24:20.411370+02:00 str-sw-core-1 dn_alm 817 - - Node.1-Unit.1:PRI [event], Dell EMC (OS10) %PM_SYS_UTIL_HI: Top 3 Processes CPU utilization(%): base_nas(1677) - 26.12 , dn_sm(1244) - 3.50 , dn_app_timesync(1253) - 1.40
<164>1 2021-09-14T10:26:27.452252+02:00 str-sw-core-1 dn_alm 817 - - Node.1-Unit.1:PRI [alarm], Dell EMC (OS10) %PM_SYS_UTIL_LO: CPU Utilization 5-minute Low Threshold alarm Raised: 70.89 (80-70) %%ACTION=RAISED %%SEQUENCE=7245
<161>1 2021-09-14T10:27:05.139354+02:00 str-sw-core-1 dn_alm 817 - - Node.1-Unit.1:PRI [alarm], Dell EMC (OS10) %PM_SYS_UTIL_HI: CPU Utilization 5-minute High Threshold alarm Raised: 80.94 (80-70) %%ACTION=RAISED %%SEQUENCE=7246
<161>1 2021-09-14T10:27:05.147474+02:00 str-sw-core-1 dn_alm 817 - - Node.1-Unit.1:PRI [event], Dell EMC (OS10) %PM_SYS_UTIL_HI: Top 3 Processes CPU utilization(%): base_nas(1677) - 27.08 , dn_sm(1244) - 3.50 , dn_app_timesync(1253) - 1.40

show processes node-id 1 pid 1677
top - 10:27:45 up 1:15, 1 user, load average: 4.89, 3.28, 1.80
Tasks: 1 total, 0 running, 1 sleeping, 0 stopped, 0 zom...
%Cpu(s): 19.4 us, 9.2 sy, 0.0 ni, 70.9 id, 0.1 wa, 0.0 hi, 0.4 si, 0.0 st
KiB Mem : 4023444 total, 777572 free, 2600968 used, 644904 buff/cache
KiB Swap: 4023440 total, 4023440 free, 0 used. 1170900 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1677 root 20 0 2920904 371736 113208 S 133.3 9.2 22:22.02 base_nas

show alarms

Sq No Severity Name Timestamp Source
------ ------------- ------------------------------ --------------------------- --------------
7242 warning PM_SYS_UTIL_LO Tue Sep 14 10:24:11 2021 /cpu/1min
7243 critical PM_SYS_UTIL_HI Tue Sep 14 10:24:20 2021 /cpu/1min
7245 warning PM_SYS_UTIL_LO Tue Sep 14 10:26:27 2021 /cpu/5min
7246 critical PM_SYS_UTIL_HI Tue Sep 14 10:27:05 2021 /cpu/5min

I would attribute this to cosmetics but when the CPU is so high it is virtually impossible to get to SSH of PC6248 access switches.

BR

Hello,

let me try to escalate to technical support in order to have feedback.

Can you give me the service tag of the switch in a private message?

Thanks
Marco

Hi

Probably worth of mentioning that we just 1h ago had to reload the nodes (well only one, the other just for getting 1st as primary again).
Very large number of users started to get strange 2-3s stutters/interruptions in connection to their machines (working remote) and after reloading no more issues (cpu is also normal and no alarms):

CPU Utilization 5Sec(%) 1Min(%) 5Min(%)
------------------------------------------------------
UNIT1 23.49 23.26 23.72

BR