Openshift Event Code: 1030NODE0005

A node's file system becoming full can have a widespread negative impact. This issue can cause any or all of the applications scheduled to that node to experience anything from degraded performance to becoming fully inoperable.

Depending on the node and file system involved, this issue can pose a critical threat to the stability of the cluster.

The NodeFilesystemAlmostOutOfSpace alert is similar to the NodeFilesystemSpaceFillingUp alert, but rather than being based on a prediction that a file system will become full in a certain amount of time, it uses simple static thresholds.

This alert triggers at a warning level when 5% of space remains in the file system, and at a critical level when 3% of space remains.

Diagnosis:
Study recent trends of file system usage on a dashboard. Sometimes, a periodic pattern of writing and cleaning up in the file system can cause the linear prediction algorithm to trigger a false alert.

Use the Linux operating system tools and utilities to investigate what directories are using the most space in the file system. Is the issue an irregular condition, such as a process failing to clean up behind itself and using a large amount of space? Or does the issue seem to be related to organic growth?

To assist in your diagnosis, watch the following metric in PromQL (execute the query on the OCP web console: Observe → Metrics → Run queries):

node_filesystem_free_bytes

Then, check the mountpoint label for the alert.

Mitigation:

If the mountpoint label is /, /sysroot or /var, remove unused images to resolve the issue:

1.Debug the node by accessing the node file system:
$ NODE_NAME=<instance label from alert>
$ oc -n default debug node/$NODE_NAME
$ chroot /host

2.Remove dangling images:
$ podman images -q -f dangling=true | xargs --no-run-if-empty podman rmi

3.Remove unused images:
$ podman images | grep -v -e registry.redhat.io -e "quay.io/openshift" -e registry.access.redhat.com -e docker-registry.usersys.redhat.com -e docker-registry.ops.rhcloud.com -e rhmap | xargs --no-run-if-empty podman rmi 2>/dev/null

4.Exit debug:
$ exit
$ exit

Support:
If all the above steps cannot resolve the issue, contact the Dell EMC technical support for further investigation.