ECS: Latency encountered when logging in or authenticating to a specific node

When making an ssh connection to a node, running sudo, or any other command that requires authentication, a delay is encountered. After the delay, the command runs normal.  The node is not heavily loaded, and uptime shows normal load averages. 

If verbosity is enabled for an ssh connection using -vvv, the login delay occurs after "pledge: exec"

Example:

admin:~> ssh 169.254.1.2 -vvv
OpenSSH_7.2p2, OpenSSL 1.0.2p-fips  14 Aug 2018
debug1: Reading configuration data /home/admin/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 25: Applying options for *
debug2: resolving "169.254.1.2" port 22
debug2: ssh_connect_direct: needpriv 0
debug1: Connecting to 169.254.1.2 [169.254.1.2] port 22.
debug1: Connection established.
... <truncated debugging>
debug3: ssh_session2_open: channel_new: 0
debug2: channel 0: send open
debug3: send packet: type 90
debug1: Requesting no-more-sessions@openssh.com
debug3: send packet: type 80
debug1: Entering interactive session.
debug1: pledge: exec

After a pause for 20+ seconds the login will proceed normally.


On the node having the issue if the dbus service status is checked, it reports timeouts:

admin:~> sudo systemctl status dbus
● dbus.service - D-Bus System Message Bus
   Loaded: loaded (/usr/lib/systemd/system/dbus.service; static; vendor preset: disabled)
   Active: active (running) since Fri 2024-01-23 04:56:00 UTC; 1 months 2 days ago
     Docs: man:dbus-daemon(1)
 Main PID: 3060 (dbus-daemon)
    Tasks: 1 (limit: 512)
   Memory: 5.8M
      CPU: 10h 45min 897ms
   CGroup: /system.slice/dbus.service
           └─3060 /bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation

Mar 22 18:28:54 <ecs-node-fqdn> dbus[3060]: [system] Connection has not authenticated soon enough, closing it (auth_timeout=30000ms, elapsed: 30000ms)
Mar 22 18:28:54 <ecs-node-fqdn> dbus[3060]: [system] Connection has not authenticated soon enough, closing it (auth_timeout=30000ms, elapsed: 30000ms)
Mar 22 18:28:54 <ecs-node-fqdn> dbus[3060]: [system] Connection has not authenticated soon enough, closing it (auth_timeout=30000ms, elapsed: 30000ms)
Mar 22 18:28:54 <ecs-node-fqdn> dbus[3060]: [system] Connection has not authenticated soon enough, closing it (auth_timeout=30000ms, elapsed: 30000ms)
Mar 22 18:28:54 <ecs-node-fqdn> dbus[3060]: [system] Connection has not authenticated soon enough, closing it (auth_timeout=30000ms, elapsed: 30000ms)
Mar 22 18:28:54 <ecs-node-fqdn> dbus[3060]: [system] Connection has not authenticated soon enough, closing it (auth_timeout=30000ms, elapsed: 30000ms)
Mar 22 18:29:22 <ecs-node-fqdn> dbus[3060]: [system] Connection has not authenticated soon enough, closing it (auth_timeout=30000ms, elapsed: 30009ms)
Mar 22 18:29:24 <ecs-node-fqdn> dbus[3060]: [system] Connection has not authenticated soon enough, closing it (auth_timeout=30000ms, elapsed: 30002ms)
Mar 22 18:29:54 <ecs-node-fqdn> dbus[3060]: [system] Connection has not authenticated soon enough, closing it (auth_timeout=30000ms, elapsed: 30000ms)
Mar 22 18:30:01 <ecs-node-fqdn> dbus[3060]: [system] Connection has not authenticated soon enough, closing it (auth_timeout=30000ms, elapsed: 30000ms)

Similar timeout messages are found in /var/log/messages

A connection limit for the dbus service is reached which causes a delay for login and or authentication

This issue can be corrected by restarting the dbus service and also, for primary nodes: the dnsmasq service. There is no anticipated impact, however obtain customer approval if needed. 

From the node that is having the issue, restart dbus using the command:  
 

sudo systemctl restart dbus
admin:~> sudo systemctl restart dbus

No output is expected from this command. 

After the command completes, check the status of the dbus service to ensure it is "active (running)" by running the below command:
 

sudo systemctl status dbus
admin ~> sudo systemctl status dbus
● dbus.service - D-Bus System Message Bus
   Loaded: loaded (/usr/lib/systemd/system/dbus.service; static; vendor preset: disabled)
   Active: active (running) since Fri 2024-07-19 18:07:00 UTC; 5s ago
     Docs: man:dbus-daemon(1)
 Main PID: 19340 (dbus-daemon)
    Tasks: 1 (limit: 512)
   Memory: 884.0K
      CPU: 22ms
   CGroup: /system.slice/dbus.service
           └─19340 /bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation

If the node is a primary node (usually node 1) it also necessary to restart dnsmasq with command: 
 

sudo systemctl restart dnsmasq
admin:~> sudo systemctl restart dnsmasq

No output is expected from this command. 

After restarting dnsmasq validate that the service is "active (running)" using command:
 

sudo systemctl status dnsmasq
admin~> sudo systemctl status dnsmasq
● dnsmasq.service - DNS caching server.
   Loaded: loaded (/usr/lib/systemd/system/dnsmasq.service; disabled; vendor preset: disabled)
  Drop-In: /run/systemd/generator/dnsmasq.service.d
           └─50-insserv.conf-$named.conf
   Active: active (running) since Fri 2024-07-19 18:14:47 UTC; 2s ago
  Process: 61272 ExecStartPre=/usr/sbin/dnsmasq --test (code=exited, status=0/SUCCESS)
 Main PID: 61297 (dnsmasq)
    Tasks: 1 (limit: 512)
   Memory: 2.1M
      CPU: 8ms
   CGroup: /system.slice/dnsmasq.service
           └─61297 /usr/sbin/dnsmasq --log-async --enable-dbus --keep-in-foreground

Once the services have been restarted test access to the node by using ssh or running a sudo command to observe that the delay has been resolved. 

ECS, ECS Appliance, ECS Appliance Gen 1, ECS Appliance Gen 2, ECS Appliance Gen 3, ECS Appliance Hardware Gen1 U-Series, ECS Appliance Hardware Gen1 C-Series, ECS Appliance Hardware Gen2 C-Series, ECS Appliance Hardware Gen2 D-Series , ECS Appliance Hardware Gen2 U-Series ... View More about warranties View Less about warranties

ECS Appliance Hardware Gen3 EX5000, ECS Appliance Hardware Gen3 EX300, ECS Appliance Hardware Gen3 EX3000, ECS Appliance Hardware Gen3 EX500, ECS Appliance Hardware Gen3 EXF900, ECS Appliance Hardware Series, ECS Appliance Software with Encryption , ECS Appliance Software without Encryption, ECS Software ... View More View Less