ECS: xDoctor RAP153: Connection Limit (1000) Reached on ECS Nodes, No New Connection

Summary: Application reading and writing to ECS is unable to establish more socket connections. In some scenarios, this issue may cause potential data unavailability if the application is opening excessive connections greater than the default connection limits. ...

This article applies to This article does not apply to This article is not tied to any specific product. Not all product versions are identified in this article.
Check out other resources

Symptoms

The application is unable to establish new network connections for HTTPs requests.

Cause

ECS nodes have a 1000 connection limit per node. If this limit is exceeded, dataheadsvc shows the following error:

169.xxx.2.9 2021-02-24T21:13:08,792 [qtp85102332-247-acceptor-1@4fda771a-TrafficMetricsNetworkTrafficServerConnector@49de2cc4{HTTP/1.1,[http/1.1]}{10.xxx.xxx.xxx:9020}]  INFO  ConnectionLimit.java (line 177) Connection Limit(1000) reached for [TrafficMetricsNetworkTrafficServerConnector@49de2cc4{HTTP/1.1,[http/1.1]}{10.xxx.xxx.xxx:9020}, TrafficMetricsNetworkTrafficServerConnector@4e293c35{SSL,[ssl, http/1.1]}{10.xxx.xxx.xxx:9021}]

To see more information, parse the logs with svc_log to count the entries for "Connection Limit (1000) reached" in the last 30 minutes.

Commands:

# svc_log dataheadsvc "Connection Limit(1000) cleared" -start "30 minutes ago" -sh -st minute
# svc_log dataheadsvc "Connection Limit(1000) cleared" -start "24 hour ago" -sh -st hour

Example:

admin@node1:~># svc_log dataheadsvc "Connection Limit(1000) cleared" -start "30 minutes ago" -sh -st minute
svc_log v1.0.24 (svc_tools v1.8.6.0)                 Started 2021-02-26 18:47:10

Running on nodes:              <All nodes>
Time range:                    2021-02-24 21:00:00 - 2021-02-24 21:30:59
Filter string(s):              'Connection Limit(1000) cleared'
Show nodename(s):              True
Search reclaim logs (if any):  False

Count of message occurrences per minute:

                       # Log
Time                   Entries
------------------------------

2021-02-24 21:01:xx    0
2021-02-24 21:02:xx    0
2021-02-24 21:03:xx    0
2021-02-24 21:04:xx    41
2021-02-24 21:05:xx    134
2021-02-24 21:06:xx    8
2021-02-24 21:07:xx    10
2021-02-24 21:08:xx    11
2021-02-24 21:09:xx    0
2021-02-24 21:10:xx    0
2021-02-24 21:11:xx    0
2021-02-24 21:12:xx    412
2021-02-24 21:13:xx    650
2021-02-24 21:14:xx    0
2021-02-24 21:15:xx    0
2021-02-24 21:16:xx    0
2021-02-24 21:17:xx    0
2021-02-24 21:18:xx    0
2021-02-24 21:19:xx    0
2021-02-24 21:20:xx    0
2021-02-24 21:21:xx    0
2021-02-24 21:22:xx    0
2021-02-24 21:23:xx    0
2021-02-24 21:24:xx    0
2021-02-24 21:25:xx    0
2021-02-24 21:26:xx    0
2021-02-24 21:27:xx    0
2021-02-24 21:28:xx    0
2021-02-24 21:29:xx    0
2021-02-24 21:30:xx    0

Total:                 1266

Resolution

This issue is generally an application issue with how the application opens and closes network connections. The user's application or load balancer should handle the connections and should close the connections once complete. The application should always try to resolve the issue on their side first before changing ECS as this causes more load on the ECS.

To help identify the issue, use netstat during the event to get the IPs that have established connections.

Command:

# sudo ss | grep 9020

Example:

admin@ecsnode1:~> sudo ss | grep 9020

ESTAB 0 0 [::ffff:10.xx.xxx.71]:9020 [::ffff:10.xxx.xxx.9]:56134

To see the number of connections in current time, use this command:

admin@ecsnode1:~> sudo ss -tn sport = :9020 | wc -l

Affected Products

ECS
Article Properties
Article Number: 000183969
Article Type: Solution
Last Modified: 30 Sept 2025
Version:  14
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.