Isilon: The /var/log Messages on Some Nodes Show Out of Memory Messages From Time to Time

Summary: The /var/log messages on some nodes show these Out of Memory (OOM) messages from time to time: OOM: v_wire_count: 2832982, v_active_count: 516

This article applies to This article does not apply to This article is not tied to any specific product. Not all product versions are identified in this article.
Check out other resources

Symptoms

 A complete message set looks like this:  
2021-08-17T14:40:50.703082+12:00 <0.4> cluster-1(id1) /boot/kernel.amd64/kernel: OOM: v_wire_count: 2843879, v_active_count: 871
2021-08-17T14:40:50.703246+12:00 <0.4> cluster-1(id1) /boot/kernel.amd64/kernel: Malloc Pigs:
2021-08-17T14:40:50.703277+12:00 <0.4> cluster-1(id1) /boot/kernel.amd64/kernel: Type                   InUse   MemUse   Requests
2021-08-17T14:40:50.703303+12:00 <0.4> cluster-1(id1) /boot/kernel.amd64/kernel: 8kB dinodes           840507  647775K 13330759531
2021-08-17T14:40:50.703325+12:00 <0.4> cluster-1(id1) /boot/kernel.amd64/kernel: isi_hash               31424   88002K 1548035744
2021-08-17T14:40:50.703344+12:00 <0.4> cluster-1(id1) /boot/kernel.amd64/kernel: lbm super              17535   84150K    1435677
2021-08-17T14:40:50.703362+12:00 <0.4> cluster-1(id1) /boot/kernel.amd64/kernel: layout_hints          183473   57336K   22320273
2021-08-17T14:40:50.703379+12:00 <0.4> cluster-1(id1) /boot/kernel.amd64/kernel: iaddr_set             886694   55419K 5188138062
2021-08-17T14:40:50.703397+12:00 <0.4> cluster-1(id1) /boot/kernel.amd64/kernel: devbuf                 17379   37840K     184744
2021-08-17T14:40:50.703414+12:00 <0.4> cluster-1(id1) /boot/kernel.amd64/kernel: newblk                    26   32774K    2364548
2021-08-17T14:40:50.703432+12:00 <0.4> cluster-1(id1) /boot/kernel.amd64/kernel: bar_owner_vec259         261   32288K    2220148
2021-08-17T14:40:50.703451+12:00 <0.4> cluster-1(id1) /boot/kernel.amd64/kernel: parent_vec153         218013   27538K    4567646
2021-08-17T14:40:50.703475+12:00 <0.4> cluster-1(id1) /boot/kernel.amd64/kernel: ddvec84               183471   22840K  177809736
2021-08-17T14:40:50.703496+12:00 <0.4> cluster-1(id1) /boot/kernel.amd64/kernel: inodedep                  22   16390K    1677902
2021-08-17T14:40:50.703515+12:00 <0.4> cluster-1(id1) /boot/kernel.amd64/kernel: sysctloid             201426   10989K     201598
2021-08-17T14:40:50.703535+12:00 <0.4> cluster-1(id1) /boot/kernel.amd64/kernel: vfscache                   4    8241K          4
2021-08-17T14:40:50.703555+12:00 <0.4> cluster-1(id1) /boot/kernel.amd64/kernel: KTR                        2    7168K          2
2021-08-17T14:40:50.703574+12:00 <0.4> cluster-1(id1) /boot/kernel.amd64/kernel: ptr_llcb_map           56126    7017K  116151979
2021-08-17T14:40:50.703592+12:00 <0.4> cluster-1(id1) /boot/kernel.amd64/kernel: Unshown bins account for 84039K
2021-08-17T14:40:50.703611+12:00 <0.4> cluster-1(id1) /boot/kernel.amd64/kernel: Total: 1219801K
2021-08-17T14:40:50.703630+12:00 <0.4> cluster-1(id1) /boot/kernel.amd64/kernel: UMA Zalloc Pigs:
2021-08-17T14:40:50.703648+12:00 <0.4> cluster-1(id1) /boot/kernel.amd64/kernel: ZONE NAME       SIZE     LIMIT     COUNT   MEM USED
2021-08-17T14:40:50.703668+12:00 <0.4> cluster-1(id1) /boot/kernel.amd64/kernel: mbuf          000256, 06012268, 00017191, 00004297K
2021-08-17T14:40:50.703687+12:00 <0.4> cluster-1(id1) /boot/kernel.amd64/kernel: socket        000872, 00765000, 00002327, 00001981K
2021-08-17T14:40:50.703704+12:00 <0.4> cluster-1(id1) /boot/kernel.amd64/kernel: tcpcb         001080, 00765000, 00001652, 00001742K
2021-08-17T14:40:50.703722+12:00 <0.4> cluster-1(id1) /boot/kernel.amd64/kernel: tcp_inpcb     000488, 00765000, 00001665, 00000793K
2021-08-17T14:40:50.703740+12:00 <0.4> cluster-1(id1) /boot/kernel.amd64/kernel: udp_inpcb     000488, 00765000, 00000270, 00000128K
2021-08-17T14:40:50.703758+12:00 <0.4> cluster-1(id1) /boot/kernel.amd64/kernel: unpcb         000192, 00765000, 00000391, 00000073K
2021-08-17T14:40:50.703776+12:00 <0.4> cluster-1(id1) /boot/kernel.amd64/kernel: hostcache     000096, 00015360, 00000191, 00000017K
2021-08-17T14:40:50.703794+12:00 <0.4> cluster-1(id1) /boot/kernel.amd64/kernel: udpcb         000032, 00765000, 00000270, 00000008K
2021-08-17T14:40:50.703812+12:00 <0.4> cluster-1(id1) /boot/kernel.amd64/kernel: clpbuf        001016, 00015872, 00000006, 00000005K
2021-08-17T14:40:50.703832+12:00 <0.4> cluster-1(id1) /boot/kernel.amd64/kernel: tcptw         000096, 00027767, 00000013, 00000001K
2021-08-17T14:40:50.703853+12:00 <0.4> cluster-1(id1) /boot/kernel.amd64/kernel: ripcb         000488, 00765000, 00000001, 00000000K
2021-08-17T14:40:50.703872+12:00 <0.4> cluster-1(id1) /boot/kernel.amd64/kernel: Unshown zones account for 0K
2021-08-17T14:40:50.703890+12:00 <0.4> cluster-1(id1) /boot/kernel.amd64/kernel: Total: 9051K

Cause

A new OneFS function introduced in OneFS 9.2 adds these messages. 
This new function logs memory usage information every time a pageout function is called by pagedaemon when it is unable to bring the free memory above free_target.
The operating system is proactively calling the function above to log information in case memory goes further low, and the node stop responding or panic.

Resolution

Important:
If the node only has 16 GB RAM, advise the user to upgrade to 64 GB.

If the node has 24 GB RAM or more, these messages can safely be ignored if all these conditions are met:

  • The /var/log/vmlog shows that the free memory did not frequently go below 50% free target
  • No nodes in the cluster panic reboot with BUF_TIMELOCK or other OOM-related timeout messages in the panic string
  • The freevnodes sysctl shows enabled (value of 1) across all nodes:
# isi_for_array 'sysctl vfs.vnlru_reuse_freevnodes'

Note:
Remember to change sysctl vfs.vnlru_reuse_freevnodes back to the default value (0 - zero) if all the following conditions are met:
  • The node has 24 GB or more physical memory
  • The vnlru_reuse_freevnodes=1 setting was originally added to tackle the high 8KB Dinodes issue
  • The OneFS version is upgraded to one of these levels or later:
    • OneFS 9.2.1.25_GA-RUP_2023-12
    • OneFS 9.4.0.17_GA-RUP_2024-02
    • OneFS 9.5.0.7_LTS2023_GA-RUP (January 2024)
Article Properties
Article Number: 000191515
Article Type: Solution
Last Modified: 03 Apr 2024
Version:  5
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.