OneFS: Event Every Hour: Node pool -pool_name- (node pool ID: -ID-) is at or over capacity
Summary:
OneFS generates an event every hour "Node pool (node pool ID: ) is at or over capacity."
Symptoms
Node pool <pool_name> (node pool ID: <ID>) is at or over capacity.This message occurs on OneFS v8.2.2.x and later.
Cause
This event is generated when a node pool is at or over 70% utilization. This is working as designed and intended.
By default, a cluster generates Informational events when pool utilization exceeds 70%. Warning events are posted when pool utilization exceeds 80%, Critical events at 90%, and Emergency events at 97%.
Use the following command in OneFS v9.1 or later to view current event threshold settings:
cluster-2# isi event thresholds view 100010015
ID: 100010015
ID Name: SYS_DISK_POOLFULL
Description: Percentage at which a nodepool is near capacity
Defaults: info (70%), warn (80%), crit (90%), emerg (97%)
Thresholds: info (70%), warn (80%), crit (90%), emerg (97%)
Resolution
Verify the health of the node pool with the following command:
# isi storagepool nodepools list -v
OneFS versions v8.2.2 to v9.0 cannot be changed and the only solution for those clusters is to get the pool utilization below 70% capacity. Manually resetting CELOG does not address the issue.
Beginning with OneFS v9.1 the thresholds for the event can be adjusted. The following example adjusts the Informational Alert Threshold from 70% to 75%.
View the current Diskpool Threshold settings. Note that the Info Threshold defaults to 70%.
isilon-2# isi event thresholds view 100010015
ID: 100010015
ID Name: SYS_DISK_POOLFULL
Description: Percentage at which a nodepool is near capacity
Defaults: info (70%), warn (80%), crit (90%), emerg (97%)
Thresholds: info (70%), warn (80%), crit (90%), emerg (97%)
This example changes the Info Threshold to 75%
isilon-2# isi event thresholds modify 100010015 --info=75
isilon-2# isi event thresholds view 100010015
ID: 100010015
ID Name: SYS_DISK_POOLFULL
Description: Percentage at which a nodepool is near capacity
Defaults: info (70%), warn (80%), crit (90%), emerg (97%)
Thresholds: info (75%), warn (80%), crit (90%), emerg (97%)