OneFS: Memory Utilization
Summary: This article briefly discusses the general memory categories, explains which memory categories can be considered available or "free." Guidelines are provided for what number of memory certain processes and containers can use. ...
Instructions
Running out of memory can cause nodes to reboot, processes to restart and performance issues. This article briefly explains how to evaluate the available memory of a node and references some technical memory guidelines. In OneFS, memory is broadly categorized into five areas or queues: Free, Inactive, Wired, Active, and Buffer
AVAILABLE MEMORY
Free memory is immediately available for use and is not allocated.
Inactive (Inact) memory has not been used recently and is available to be allocated.
UNAVAILABLE MEMORY
Wired memory is in use by the kernel and unavailable for other allocation.
Active memory has been used recently and is allocated to user space.
Buffer (Buf) memory is used for disk caching.
CALCULATING AVAILABLE MEMORY
These memory categories can be seen from the command top.
Notice it is expected to not see a Cache category and not see anything in the Swap section:
# top last pid: 98143; load averages: 0.00, 0.02, 0.00 up 48+04:45:17 16:01:09 643 processes: 1 running, 642 sleeping CPU: 0.0% user, 0.0% nice, 0.2% system, 0.0% interrupt, 99.8% idle Mem: 512M Active, 28G Inact, 11G Wired, 12G Buf, 7530M Free Swap:
If you view output from vmstat -H, you see available memory pages (AVM) and Free memory (FRE) in bytes.
# vmstat -H procs memory page disks faults cpu r b w avm fre flt re pi po fr sr ad4 ad7 in sy cs us sy id 0 14 0 9333548 7713640 863 0 0 0 564 15 0 0 428 903 200 0 0 100
Converting AVM pages to bytes shows available memory (inactive + free):9333548 x 4096 = 38230212608 bytes
Convert to GB 38230212608/(1024^3)= 35.6 GB Available Memory
If you add Inact and Free values from the top output in this example, you have:28GB + 7.35GB = 35.35GB
Notice top output was already rounded so it may not match exactly the value from vmstat.
PROCESS SPECIFIC MEMORY UTILIZATION
Virtual Size or vsz (SIZE) refers to memory allocated to this process.
From the manual page of top, "SIZE is the total size of the process (text, data, and stack)"
Resident Set Size (RES) refers to the amount of physical memory in use by this process.
From the manual page of top, "RES is the current amount of resident memory (both SIZE and RES are given in kilobytes)"top command showing the six processes using the most virtual memory:
# top -o size -n 6 last pid: 26745; load averages: 0.15, 0.20, 0.17 up 3+19:06:14 20:03:26 103 processes: 1 running, 102 sleeping Mem: 152M Active, 23G Inact, 14G Wired, 12G Buf, 10G Free Swap:
PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 10244 root 15 60 r112 592M 310M kqread 3 6:35 0.00% nfs 2698 root 11 20 0 585M 125M select 7 34:41 0.00% isi_celog_monitor 2544 root 26 60 r112 506M 70184K kqread 4 0:05 0.00% isi_papi_d 3217 root 2 26 0 439M 48336K kqread 3 0:13 0.00% isi_celog_capture 2676 root 2 52 0 399M 41540K kqread 2 1:46 0.00% isi_celog_capture 26740 root 2 20 0 399M 40168K kqread 6 0:00 0.00% isi_celog_capture
MAXIMUM VALUES FOR LW-CONTAINER MEMORY CONSUMPTION AND OPEN FILES PER NODE
There are some published maximum values for memory depending on the OneFS version. Items such as open files per node and SMB and NFS within the LW-container are explained.
See the OneFS Technical Specifications guide
- LW-container: Page 9~23
In OneFS, three lwio process containers exist one container for each SMB, NFS, and Swift process. The SMB container can be up to 20% of the total RAM, but it is at least 1 GB and at most 32 GB. The NFS container can be up to 25% of the total RAM, but it is at least 1 GB and at most 8 GB. The Swift container is 512 MB.
- Open files limit: Page 12
The maximum number of open files per node is 90% of the maximum number of vnodes on that node, as expressed in the following formula: kern.maxfiles = kern.maxvnodes * 0.9 The OneFS protocol daemons, such as the input/output daemon (lwio), might impose additional constraints on the number of files that a node can have open. The protocol daemons typically impose such constraints because the kernel places limits on per-process memory consumption.
CHECK MEMORY UTILIZATION AGAINST MAXIMUMS
Viewing memory of an lsass process using ps:
# ps auwx | egrep "USER|lsass" | grep -v grep USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND root 14689 0.0 0.3 162112 20404 ?? I 20Feb17 0:33.30 lw-container lsass (lsass)
Checking using top command:
top -n 100 | egrep "SIZE|lsass" PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 14689 root 26 20 0 158M 20764K ucond 2 0:00 0.00% lsass