OneFS 8.2 and later: Allow Giving up Free Vnodes for New Vnodes When the Cache is Full

Summary: Allow giving up free virtual nodes (Vnodes) for new Vnodes when the cache is full.

This article applies to This article does not apply to This article is not tied to any specific product. Not all product versions are identified in this article.
Check out other resources

Symptoms

Running on low Vnodes is identified by seeing multiple vlruwk threads running: 
/var/log/messages 2022-03-11T17:17:01-08:00 <0.4> Isilon-4(id4) /boot/kernel.amd64/kernel: [rbm_worker.c:350](pid 4738="kt: dxt1")(tid=102982) There are now 505 DXT threads, msg stats: READ_INODE: 1, callback stats: kernel:lk_lock_async_callback2+0: 502, wait stats: biord: 3, vlruwk: 507
From the vmlog, notice that VNODE is taking the highest memory: 
vm.uma_zalloc_pigs: UMA Zalloc Pigs:
ZONE NAME       SIZE     LIMIT     COUNT   MEM USED
VNODE         000560, 00000000, 00830165, 00453996K
IFSINODE      000568, 00000000, 00777624, 00431338K
VM OBJECT     000240, 00000000, 00813154, 00190582K
lki_lin_ent   000208, 00000000, 00934988, 00189919K
lkc_gen_ent   000064, 00000000, 02235094, 00139693K
stat_cache    000176, 00000000, 00752869, 00129399K
lki_ref_ent   000105, 00000000, 00936379, 00096015K
lkc_lin_idds  000076, 00000000, 01012017, 00075110K
lkc_ref_idds  000068, 00000000, 01008047, 00066940K
mbuf_jumbo_p  004096, 00000000, 00016394, 00065576K
mbuf_cluster  002048, 00000000, 00026630, 00053260K
RADIX NODE    000144, 00000000, 00308743, 00043416K
ifm_attr_cac  000048, 00000000, 00770256, 00036105K
lki_mds_ent   000144, 00000000, 00255978, 00035996K
UMA Slabs     000080, 00000000, 00239085, 00018678K
Unshown zones account for 174014K
Total: 2200045K

Cause

Nodes can run Out Of Memory (OOM) due to not reclaiming free Vnodes. In OneFS 8.2 and later, a new systcl was implemented (vfs.vnlru_reuse_freevnodes) to allow giving up free Vnodes for new Vnodes when the cache is full.

By default, vnlru_reuse_freevnodes is off because Isilon prefers to defer Vnodes recycling to vnlru_proc. Turning on vnlru_reuse_freevnodes allows getvnewvnode() to recycle available free Vnodes to make room for new ones when the cache is full (but not over size). This prevents Vnode allocators from sleeping and avoids waking up vnlru_proc.

Resolution

Caution: Get Engineering Approval before disabling or enabling vfs.vnlru_reuse_freevnode. All clusters should be independently evaluated, even if they are a peer cluster or Disaster Recovery cluster.


Process:

  1. Check if the value is set with the following command (the default is 0):
# isi_for_array sysctl vfs.vnlru_reuse_freevnodes
  1. Set vfs.vnlru_reuse_freevnodes to 1:
# isi_sysctl_cluster vfs.vnlru_reuse_freevnodes=1
  1. Verify that the value is set by re-running the command:
# isi_for_array sysctl vfs.vnlru_reuse_freevnodes

To monitor after change, use the below commands to monitor usage of vnode cache and vlurwk threads:

# isi_for_array -s 'sysctl vfs.numvnodes kern.maxvnodes| xargs'; sleep 300 

# isi_for_array -s sysctl kern.malloc_pigs|grep vlurwk

 

Additional Information

Related article: 
Isilon: The /var/log Messages on Some Nodes Show Out of Memory Messages From Time to Time

Article Properties
Article Number: 000173892
Article Type: Solution
Last Modified: 18 Jun 2025
Version:  17
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.