Connectrix B-Series: Switch Kernel Panic due to Weblinker Daemon Memory Usage Increasing
Summary: The switch reported a MEMORY_USAGE_STATE issue, and after 30 minutes the switch rebooted due to a kernel panic.
Symptoms
From errdump output:
2025/07/22-22:50:19 (CST), [MAPS-1003], 293324, SLOT 2 | FID 128, WARNING, CC-MA001-SANX7-1, Chassis, Condition=CHASSIS(MEMORY_USAGE_STATE/NONE==WARNING), Current Value:[MEMORY_USAGE_STATE, WARNING], RuleName=defCHASSISMEMORY_USAGE_STATE_WARN, Dashboard Category=Switch Resource, Quiet Time=1 hour. 2025/07/22-22:52:19 (CST), [MAPS-1003], 293331, SLOT 2 | FID 81, WARNING, CC-MA001-SANX7-1-81, Chassis, Condition=CHASSIS(MEMORY_USAGE_STATE/NONE==WARNING), Current Value:[MEMORY_USAGE_STATE, WARNING], RuleName=defCHASSISMEMORY_USAGE_STATE_WARN, Dashboard Category=Switch Resource, Quiet Time=1 hour. 2025/07/22-23:10:09 (CST), [EM-1033], 156895, SLOT 1 | CHASSIS, ERROR, nevccsg31swb01, CP in Slot 2 set to faulty because CP ERROR asserted. 2025/07/22-23:10:09 (CST), [EM-1047], 156896, SLOT 1 | CHASSIS, INFO, nevccsg31swb01, CP in slot 2 not faulty, CP ERROR deasserted. 2025/07/22-23:11:19 (CST), [HAM-1004], 293376, SLOT 2 | CHASSIS, INFO, nevccsg31swb01, Processor rebooted - Software Fault:Kernel Panic
Cause
The switch has encountered FOS-863077 and this defect causing a memory leak.
Symptom:
The weblinker daemon memory usage continues to increase during SANnav monitoring, and activities such as configupload start to fail.
Current switch version Fabric OS 9.2.2 does not have the fix. Customer is using third-party monitoring tool which is also impacted.
The SRM-5011 errors showing its reached 75% of the limit starting on July 21, 2025 so it took 1 day from there to failover:
2025/07/21-18:19:43:390633 (CST), [SRM-5011], 2330474/0, SLOT 2 | FID 128, INFO, CC-MA001-SANX7-1, phy memory utilization meets or exceeds threshold of 75 percent. Current value is 76 percent., osmisc.c, line: 405, comp:osmiscd, ltime:2025/07/21-18:19:43:390406
And seen, there are lots of LDAP login fail messages. Because all the login failures are from HTTPS, they are using the weblinker process. They are draining the available weblinker memory until it depletes entirely and causes a kernel panic to recover.
2025/07/22-23:06:26:207115 (CST), [SEC-1347], 2336526/293364, SLOT 2 | FID 128, INFO, CC-MA001-SANX7-1, LDAP server 10.236.91.139 reachable, but failed to validate or authorize the user account 'svc_bs_brocade_local'., ldap_config.c, line: 1237, comp:weblinker, ltime:2025/07/22-23:06:26:206934 2025/07/22-23:06:26:207828 (CST), [SEC-1193], 2336527/293365, SLOT 2 | FID 128, INFO, CC-MA001-SANX7-1, Security violation: Login failure attempt via HTTPS. IP Addr: 10.236.90.16., public.c, line: 4039, comp:weblinker, ltime:2025/07/22-23:06:26:207430 2025/07/22-23:07:12:879161 (CST), [SEC-1347], 2336528/293366, SLOT 2 | FID 128, INFO, CC-MA001-SANX7-1, LDAP server 10.236.91.139 reachable, but failed to validate or authorize the user account 'svc_bs_brocade_local'., ldap_config.c, line: 1237, comp:weblinker, ltime:2025/07/22-
Resolution
By default, On Gen7 systems, the flow monitor is enabled. However, if any switch running FOS v9.1.0 or later disables flow monitoring, this issue may be exposed. This issue may also be exposed on Gen6 platforms without flow vision license installed.
Workaround:
Activate flow monitor:
flow --activate sys_flow_monitor
On Gen6 systems without flow vision license, upgrade to a FOS version with a fix for this issue.
- FOS9.1.1d6RN-Closed
- FOS9.2.0c3RN-Closed
- FOS9.2.1bRN-Closed
- FOS9.2.2aRN-Closed
Additional Information
Review documentation:
Dell Connectrix B-Series Fabric OS Release Notes