VxBlock: Cisco MDS9148S running 8.4(2) is unresponsive with "Anon_Resident_Mem 0 KB being killed due to lack of memory"

Summary: MDS9148S running 8.4(2) is unresponsive with "Anon_Resident_Mem 0 KB being killed due to lack of memory"

This article applies to This article does not apply to This article is not tied to any specific product. Not all product versions are identified in this article.
Check out other resources

Instructions

How to collect NSX-T Support Bundles.

Duration: 02:33
Closed captions: English only or Available in multiple languages Use the CC icon to change the closed caption language. 

MDS 9148S switch running with code 8.4(2) was unresponsive due to memory leak.
 

show system reset-reason module 1

----- reset reason for Supervisor-module 1 (from Supervisor in slot 1) ---
1) At 404187 usecs after Tue Dec  8 04:27:58 2020
    Reason: Reset triggered due to HA policy of Reset
    Service: snmpd hap reset
    Version: 8.4(2)

show logging nvram

2020 Dec  8 03:26:55.897 M9A %KERN-0-SYSTEM_MSG: [8296360.653812] [e1687f40] [c0010660] ret_from_syscall+0x0/0x3c - kernel
2020 Dec  8 03:26:55.900 M9A %KERN-0-SYSTEM_MSG: [8296360.722742] --- Exception: c00 at 0x1f45b4bc - kernel
2020 Dec  8 03:26:55.903 M9A %KERN-0-SYSTEM_MSG: [8296360.722742]     LR = 0x1f45b48c - kernel
2020 Dec  8 03:26:55.914 M9A %KERN-1-SYSTEM_MSG: [8296360.834202] Proc swapper/0 (0) with Total_VM 0 KB Resident_Mem 0 KB Anon_Resident_Mem 0 KB being killed due to lack of memory - kernel
2020 Dec  8 03:26:55.885 M9A %SYSMGR-2-SERVICE_CRASHED: Service "snmpd" (PID 28556) hasn't caught signal 9 (no core).
!
2020 Dec  8 04:27:07.306 M9A %SYSMGR-2-SERVICE_CRASHED: Service "snmpd" (PID 32573) hasn't caught signal 9 (no core).
2020 Dec  8 04:27:08.553 M9A %SYSMGR-2-SERVICE_CRASHED: Service "port" (PID 32584) hasn't caught signal 9 (no core).
2020 Dec  8 04:27:09.904 M9A %SYSMGR-2-SERVICE_CRASHED: Service "clis" (PID 32577) hasn't caught signal 9 (no core).
2020 Dec  8 04:27:11.321 M9A %SYSMGR-2-SERVICE_CRASHED: Service "acl" (PID 3622) hasn't caught signal 9 (no core).
2020 Dec  8 04:27:14.498 M9A %SYSMGR-2-SERVICE_CRASHED: Service "snmpd" (PID 32597) hasn't caught signal 9 (no core).
2020 Dec  8 04:27:16.249 M9A %SYSMGR-2-SERVICE_CRASHED: Service "clis" (PID 32603) hasn't caught signal 9 (no core).
2020 Dec  8 04:27:17.652 M9A %SYSMGR-2-SERVICE_CRASHED: Service "aam" (PID 3511) hasn't caught signal 9 (no core).
2020 Dec  8 04:27:28.584 M9A %SYSMGR-2-SERVICE_CRASHED: Service "snmpd" (PID 32611) hasn't caught signal 9 (no core).
2020 Dec  8 04:27:29.190 M9A %KERN-1-SYSTEM_MSG: [8299982.270133] Proc swapper/0 (0) with Total_VM 0 KB Resident_Mem 0 KB Anon_Resident_Mem 0 KB being killed due to lack of memory - kernel


Note: A severe RAM depletion at the time of the reload causes the issue, and the process snmpd in the example above is likely just a victim. You may observe a different process getting stopped for the same reason.

  • Monitor the switch using the following command. Once the LowFree becomes severely depleted (for example, < 20% of LowTotal), the switch can be considered "at risk".
m9148s-2# show system internal kernel meminfo | i Low
LowTotal:         484152 kB
LowFree:          328872 kB  !!!! Danger if this is less than, say, 100000 kB


To recover from this situation, either proactively schedule a reload of the device, or perform an ISSU or ISSD, even an ISSU to a release which does not have the fix, for example, 8.4(2) to 8.4(2a), temporarily recovers lost kernel memory.

  • Remove the switch from Vision or VXBC monitoring temporarily to reduce unnecessary login session, which causes a small kernel memory leak. 
  • The initial fixed version is 8.5(1).
  • Major fixed release is 8.4(2c). 


***Update March 19, 2021***
RCM Engineering included code 8.5(1) in an RCM release in June after NPI review and testing to correct this issue. Recommendation if a customer wants to move forward with correcting this issue, is to file an IA for code 8.5(1).

Affected Products

VxBlock and Vblock Systems, VxBlock and vBlock Systems Series
Article Properties
Article Number: 000191341
Article Type: How To
Last Modified: 19 Nov 2025
Version:  4
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.