ECS: Node Lockdown Fails due to SSH Rate Limiter on Node

Issue observed post upgrade to ECS 4.0.

Node status shows as "unknown" or "locked" in the ECS Web UI.

CLI lockdown command fails to run.

SSH access to the node remains available despite lockdown attempt.

The presence of an SSH rate limiter applied using a STIG rule (SLES-12-030040) causes this issue. This rule is not compatible with SLES15, and once applied, it cannot be reverted using the SLES12 STIG framework.

Below command displays information about the operating system:

admin@node1:~> cat /etc/os-release
NAME="SLES"
VERSION="15-SP4"
VERSION_ID="15.4"
PRETTY_NAME="SUSE Linux Enterprise Server 15 SP4"
ID="sles"
ID_LIKE="suse"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:suse:sles:15:sp4"
DOCUMENTATION_URL="https://documentation.suse.com/"

To address the issue, remove the SSH rate limiter from the affected node or nodes:

1. Check for the SSH rate limiter: 

admin@node1:~> sudo iptables -L | grep limit  | grep ssh
LOG        tcp  --  anywhere             anywhere             limit: avg 3/min burst 5 tcp dpt:ssh ctstate NEW recent: CHECK seconds: 60 hit_count: 3 name: ssh side: source mask: 255.255.255.255 LOG level warning tcp-options ip-options prefix "SFW2-INext-DROPr "

2. Revert the STIG rule:

admin@node1:~> cd /opt/emc/security/hardening/root/sbin; sudo ./revert-harden -i SLES-12-030040
Changes reverted: FW_SERVICES_ACCEPT_EXT="0/0,tcp,22,,hitcount=3,blockseconds=60,recentname=ssh"
FW_SERVICES_ACCEPT_EXT="0/0,tcp,22,,hitcount=3,blockseconds=60,recentname=ssh"
Changes reverted: TCP="ssh"
Restarting SuSEfirewall2.service ...
SuSEfirewall2.service restarted successfully.
Reverted STIG ID: SLES-12-030040 on /etc/sysconfig/SuSEfirewall2.d/services/sshd successfully

3. Verify changes:

admin@node1:/opt/emc/security/hardening/root/sbin> sudo iptables -L | grep limit  | grep ssh

4. Lock the node using WEB UI or CLI:

admin@node1:~> sudo -i lockdown set lock
[lockdown] :Info: Suspend fabric agent firewall check
INFO: Parsing /opt/emc/nile/etc/conf/nan/emc-firewall-cfg
dev_ext: public - dev_int: private private.4 pslave-0 pslave-1
INFO: Updating /etc/sysconfig/SuSEfirewall2.d/services/emc-ecs-custom
INFO: Updating /etc/sysconfig/SuSEfirewall2
<38>Jun 25 10:34:47 SuSEfirewall2[32514]: Firewall rules unloaded.
<38>Jun 25 10:34:47 SuSEfirewall2[32594]: Setting up rules from /etc/sysconfig/SuSEfirewall2 ...
<38>Jun 25 10:34:47 SuSEfirewall2[32594]: using default zone 'ext' for interface docker0
<38>Jun 25 10:34:47 SuSEfirewall2[32594]: using default zone 'ext' for interface public_mgmt
<38>Jun 25 10:34:47 SuSEfirewall2[32594]: using default zone 'ext' for interface slave_0
<38>Jun 25 10:34:47 SuSEfirewall2[32594]: using default zone 'ext' for interface slave_1
<38>Jun 25 10:34:47 SuSEfirewall2[32594]: Firewall custom rules loaded from /opt/emc/nile/etc/conf/nan/nan_pbr_rules
<38>Jun 25 10:34:48 SuSEfirewall2[32594]: /proc/sys/net/ipv4/conf/all/rp_filter override in /etc/sysctl.d/70-nan-performance.conf, not setting it
<38>Jun 25 10:34:48 SuSEfirewall2[32594]: /proc/sys/net/ipv4/conf/default/rp_filter override in /etc/sysctl.d/70-nan-performance.conf, not setting it
<38>Jun 25 10:34:48 SuSEfirewall2[32594]: /proc/sys/net/ipv4/conf/docker0/rp_filter override in /etc/sysctl.d/70-nan-performance.conf, not setting it
<38>Jun 25 10:34:48 SuSEfirewall2[32594]: /proc/sys/net/ipv4/conf/lo/rp_filter override in /etc/sysctl.d/70-nan-performance.conf, not setting it
<38>Jun 25 10:34:48 SuSEfirewall2[32594]: /proc/sys/net/ipv4/conf/private/rp_filter override in /etc/sysctl.d/70-nan-performance.conf, not setting it
<38>Jun 25 10:34:48 SuSEfirewall2[32594]: /proc/sys/net/ipv4/conf/pslave-0/rp_filter override in /etc/sysctl.d/70-nan-performance.conf, not setting it
<38>Jun 25 10:34:48 SuSEfirewall2[32594]: /proc/sys/net/ipv4/conf/pslave-1/rp_filter override in /etc/sysctl.d/70-nan-performance.conf, not setting it
<38>Jun 25 10:34:48 SuSEfirewall2[32594]: /proc/sys/net/ipv4/conf/public/rp_filter override in /etc/sysctl.d/70-nan-performance.conf, not setting it
<38>Jun 25 10:34:48 SuSEfirewall2[32594]: /proc/sys/net/ipv4/conf/slave-0/rp_filter override in /etc/sysctl.d/70-nan-performance.conf, not setting it
<38>Jun 25 10:34:49 SuSEfirewall2[32594]: /proc/sys/net/ipv4/conf/slave-1/rp_filter override in /etc/sysctl.d/70-nan-performance.conf, not setting it
<38>Jun 25 10:34:49 SuSEfirewall2[32594]: Firewall rules successfully set
INFO: Updating /etc/sysconfig/SuSEfirewall2
<38>Jun 25 10:34:57 SuSEfirewall2[35003]: Firewall rules unloaded.
<38>Jun 25 10:34:57 SuSEfirewall2[35084]: Setting up rules from /etc/sysconfig/SuSEfirewall2 ...
<38>Jun 25 10:34:57 SuSEfirewall2[35084]: using default zone 'ext' for interface docker0
<38>Jun 25 10:34:57 SuSEfirewall2[35084]: using default zone 'ext' for interface public_mgmt
<38>Jun 25 10:34:57 SuSEfirewall2[35084]: using default zone 'ext' for interface slave_0
<38>Jun 25 10:34:57 SuSEfirewall2[35084]: using default zone 'ext' for interface slave_1
<38>Jun 25 10:34:57 SuSEfirewall2[35084]: Firewall custom rules loaded from /opt/emc/nile/etc/conf/nan/nan_pbr_rules
<38>Jun 25 10:34:57 SuSEfirewall2[35084]: /proc/sys/net/ipv4/conf/all/rp_filter override in /etc/sysctl.d/70-nan-performance.conf, not setting it
<38>Jun 25 10:34:57 SuSEfirewall2[35084]: /proc/sys/net/ipv4/conf/default/rp_filter override in /etc/sysctl.d/70-nan-performance.conf, not setting it
<38>Jun 25 10:34:57 SuSEfirewall2[35084]: /proc/sys/net/ipv4/conf/docker0/rp_filter override in /etc/sysctl.d/70-nan-performance.conf, not setting it
<38>Jun 25 10:34:57 SuSEfirewall2[35084]: /proc/sys/net/ipv4/conf/lo/rp_filter override in /etc/sysctl.d/70-nan-performance.conf, not setting it
<38>Jun 25 10:34:57 SuSEfirewall2[35084]: /proc/sys/net/ipv4/conf/private/rp_filter override in /etc/sysctl.d/70-nan-performance.conf, not setting it
<38>Jun 25 10:34:58 SuSEfirewall2[35084]: /proc/sys/net/ipv4/conf/pslave-0/rp_filter override in /etc/sysctl.d/70-nan-performance.conf, not setting it
<38>Jun 25 10:34:58 SuSEfirewall2[35084]: /proc/sys/net/ipv4/conf/pslave-1/rp_filter override in /etc/sysctl.d/70-nan-performance.conf, not setting it
<38>Jun 25 10:34:58 SuSEfirewall2[35084]: /proc/sys/net/ipv4/conf/public/rp_filter override in /etc/sysctl.d/70-nan-performance.conf, not setting it
<38>Jun 25 10:34:58 SuSEfirewall2[35084]: /proc/sys/net/ipv4/conf/slave-0/rp_filter override in /etc/sysctl.d/70-nan-performance.conf, not setting it
<38>Jun 25 10:34:58 SuSEfirewall2[35084]: /proc/sys/net/ipv4/conf/slave-1/rp_filter override in /etc/sysctl.d/70-nan-performance.conf, not setting it
<38>Jun 25 10:34:59 SuSEfirewall2[35084]: Firewall rules successfully set
locked
[lockdown] :Info: Resume fabric agent firewall check

This workaround is applicable only to nodes running SLES15 where the SLES12 STIG framework has been previously applied. Ensure the compatibility before applying or reverting STIG rules.