This article explains about the list of outputs that need to be collected in the cases where memory leaking is suspected.
Memory leak in most cases is a very slow process and extensive log analysis is required on such cases.
We may require multiple samples of the outputs for our analysis and to understand which part of the memory is affected.
Memory monitoring tool
A memory monitoring tool was incorporated into the S4810, MXL and Z9000 9.1(0.0P6+) releases. This tool captures detailed data allocation usage valuable in troubleshooting potential memory leaks. The memory monitoring tool can be accessed through a console or telnet session.
1: Enable Memory Monitoring Tool
To enable the memory monitoring tool.
Usage e.g.:
remote-exec cp f10mc -i 0x87
Example:
Command:
Z9000#remote-exec cp f10mc -i 0x87
System Response:
f10mc -i 0x87
f10mc: Initialized for malloc, pool, pool_cache, with filter for F10 allocations
SStk-0 #
2: Collecting Data
2a: Collecting Periodic General Memory Data
Collect the data periodically (4 to 6 hour intervals) each day for several days. To collect the output, execute the following command. The following command displays general memory details of the system. The cycle number in the output will increment by one each time the output is collected.
Usage e.g.:
remote-exec cp f10mc -r 0
Example:
Command:
Z9000#remote-exec cp f10mc –r 0
System Response:
f10mc -r 0
{
-----------<cycle 3>-----------
#### This cycle: 46176 bytes held by 56 allocations ####
M: 1888(16); +2329280(18371), -2329280(18371)
P: 0(0); +0(0), -0(0)
PC: 44288(40); +3053056(11226), -3053056(11226)
#### Since MCInit: 708902 bytes held by 2112 allocations ####
M: 52006(1338); +110122134(928732), -110070128(927394)
P: 0(0); +0(0), -0(0)
PC: 656896(774); +148369408(548411), -147712512(547637)
}
SStk-0 #
The following output may be requested in addition to the memory monitoring tool output.
Related Commands:
remote-exec cp top -b -o res all
remote-exec cp netstat –m
remote-exec cp vmstat –s
show process memory
show process memory management
show memory
show clock
2b: Collecting Detailed Memory Data
After the collection period for the "remote-exec cp f10mc –r 0" command has completed, the output of the following command will be requested. The following command will provide precise memory details of the system.
Usage e.g.:
remote-exec cp f10mc -r 1
Example:
Command:
Z9000#remote-exec cp f10mc -r 1
System Response:
f10mc -r 1
{
PC:mbpl:0xC71F5200:256:3
PC:mclpl:0xD7C24800:2048:3
PC:mbpl:0xC71F3500:256:3
{output truncated}
PC:mbpl:0xCAFC7F00:256:2
PC:mclpl:0xD7C4D000:2048:2
-----------<cycle 3>-----------
#### This cycle: 286144 bytes held by 266 allocations ####
M: 3520(16); +21362(417), -20514(415)
P: 0(0); +0(0), -0(0)
PC: 282624(250); +1129472(2200), -1126912(2197)
#### Since MCInit: 599392 bytes held by 542 allocations ####
M: 5216(20); +67838(1335), -62622(1315)
P: 0(0); +0(0), -0(0)
PC: 594176(522); +3513088(6660), -2918912(6138)
}
SStk-0 #
Engineering will evaluate the collected data and provide details on how to collect further information that will assist in identifying source for memory leak. Output requested will be for suspect memory allocations:
Usage e.g.:
remote-exec cp f10mc –c 0x<memory address>
Example:
Command:
Z9000# remote-exec cp f10mc –c 0xCE0DDD00
System Response:
f10mc -c 0xCE0DDD00
PC:mbpl:0xCE0DDD00:256:2> 0xC0100EAD 0xC033DA54 0xC0137ADE 0xC0136DD0 0xC0142682 0xC013FA17 0xC0148C38 0xC037C808 0xC037BE3F 0xC09B805A
call trace:
0xC0100EAD in Xsoftintr
0xC033DA54 in softint_dispatch
0xC0137ADE in ipintr
0xC0136DD0 in ip_input
0xC0142682 in tcp_input
0xC013FA17 in syn_cache_get
0xC0148C38 in tcp_template
0xC037C808 in m_gethdr
0xC037BE3F in m_get
0xC09B805A in f10MC_pool_cache_get_paddr
SStk-0 #
3:Disable Memory Monitoring
To disable the memory monitoring tool after the data review is complete..
Usage e.g.:
remote-exec cp f10mc -i 0x0
Example:
Command:
Z9000#remote-exec cp f10mc -i 0x0
System Response:
f10mc -i 0x0
f10mc: deinitialized
SStk-0 #