Unsolved
This post is more than 5 years old
13 Posts
0
911
Watchout - SNMP Collectors eating threads
We have recently found out, that we are getting out of threads on one of our servers, where a lot of SNMP collectors are running. Even one small text-collector using 2 threads got to the point, when it could not start polling, as the threads were unavailable. We were tracking the problem and found out, that when you use Device-Discovery module with SNMP collectors and the DD module pushes new configuration, it restarts only the SNMP collector processess (so not the whole collector-manager) and the SNMP process just takes new set of threads as configured (600 in our case) but do not return the previous ones to the pool. That brought us to the state, where all 65k processes were taken and the collectors started to fail. The collector-managers though run happilly and the only indication to the polling were our Data Availabililty reports, where we found that random polling-groups could not be created.
We have finally created report to monitor the thread consumption on our servers and the report is self-explaining (10:00 is our daily discovery process):
So who is using the SNMP collectors with DD module and do not restart collector-managers regularly watch your threads
SR opened with no 72227894
Happy charting every1
fvompe
3 Posts
1
June 23rd, 2015 06:00
Hi Tomas!
Looks interesting! What W4N SNMP-collector version are you using?
Regards,
Fedor
tomas_nekolny
13 Posts
1
June 23rd, 2015 23:00
Hi Fedor, our version is 6.4.u2 & 3. We are upgrading all to u3 in a week or two, but as I haven't seen this in release notes I do not expect its fixed in 6.4 versions anywhere.
Major version upgrade we plan for Q3 (Sep).
Exact colllector verison is:
* snmp-collector snmp11 : Collecting SNMP-Collector v6.4u1 r50994 linux-x64
If interested I can share the way we measure it and report definition.
Cheers,
T.
EDIT: Just got update from support, that this issue is not in any of the future versions Release Notes. Engineering will follow-up
tomas_nekolny
13 Posts
0
September 15th, 2015 03:00
The issue is resolved in following patches:
6.3u7, 6.4u4, 6.5u4
After application of the patch the threads are not dramatically rising anymore.