Connectrix Brocade: CP panics with snmpd running out of memory
Summary: CP panics after snmpd runs out of memory.
Symptoms
Unexpected reboot or failover after snmpd runs out of memory.
The following excerpts can be seen under errdump:
2023/08/03-18:58:30, [FSSM-1003], 131524, SLOT 7 | CHASSIS, WARNING, DellBrocadeLab1, HA State out of sync.
2023/08/03-18:58:51, [EM-1033], 131526, SLOT 7 | CHASSIS, ERROR, DellBrocadeLab1, CP in Slot 6 set to faulty because CP ERROR asserted.
2023/08/03-18:59:02, [MAPS-1020], 131531, SLOT 7 | FID 128, WARNING, DellBrocadeLab1, Switch wide status has changed from HEALTHY to MARGINAL.
2023/08/03-19:05:11, [HAMK-1004], 131533, SLOT 7 | CHASSIS, INFO, DellBrocadeLab1, Resetting standby CP (double reset may occur).
2023/08/03-19:05:26, [EM-1047], 131534, SLOT 7 | CHASSIS, INFO, DellBrocadeLab1, CP in slot 6 not faulty, CP ERROR deasserted.
2023/08/03-19:06:08, [TRCE-1001], 131536, SLOT 7 | CHASSIS, WARNING, DellBrocadeLab1, Trace dump available (Slot 6)! (reason: PANIC).
2023/08/03-19:06:08, [TRCE-1004], 131537, SLOT 7 | CHASSIS, WARNING, DellBrocadeLab1, Trace dump (Slot 6) was not transferred because trace auto-FTP disabled.
2023/08/03-19:06:12, [HAM-1004], 131538, SLOT 6 | CHASSIS, INFO, DellBrocadeLab1, Processor rebooted - Reset.
2023/08/03-19:06:12, [TRCE-1001], 131539, SLOT 6 | CHASSIS, WARNING, DellBrocadeLab1, Trace dump available (Slot 6)! (reason: PANIC).Cause
This issue has been identified due to Fabric OS (FOS) defect FOS-841254. This issue occurs when a user uses incorrect credentials to perform an MIB walk. It leaves a large /tmp/snmpd.log file filled with text like "Authentication failed for snmpadmin1" and "security service 3 error parsing ScopedPDU." As a result, the snmpd.log file increases rapidly.
Ten big files in /tmp: 2023-08-03--18:05:08
============================================
-rw-r--r-- 1 root root 322 Aug 3 14:31 ipfpolicy.v4.Active.txt
-rw-rw-rw- 1 root root 1216 Feb 2 2023 debug.0.conf
-rw-r--r-- 1 root root 1591 Aug 3 18:04 snmpdev
-rw-r--r-- 1 root root 1595 Aug 2 01:03 zoneFileData_0
-rw-r--r-- 1 root root 1652 Jan 28 2023 netfilter.rules.ipv6
-rw-r--r-- 1 root root 3329 Aug 2 14:31 netfilter.rules.ipv4
-rw-r--r-- 1 root root 4108 Jan 28 2023 cal_shm_file
-rw-rw-rw- 1 root root 22872 Aug 3 15:04 debug.pid
--wx--Sr-- 1 root root 35376 Jan 29 2023 webd_shared_map.0
-rw-r--r-- 1 root root 860707946 Aug 3 18:05 snmpd.log
Known affected release: FOS v8.2.3a, v8.2.3c1
Resolution
Workaround:
Run the cat /dev/null > /tmp/snmpd.log command weekly to reduce the size of the snmpd.log file.
Resolution:
Upgrade to either of the following fixed FOS releases:
FOS8.2.3a5_cvr_842814
FOS9.0.1f
FOS8.2.3d
FOS7.4.2k
FOS9.1.1a
FOS9.2.0