Openshift Event Code: 1038NODE0008

Zusammenfassung: System Memory Exceeds Reservation

Dieser Artikel gilt für Dieser Artikel gilt nicht für Dieser Artikel ist nicht an ein bestimmtes Produkt gebunden. In diesem Artikel werden nicht alle Produktversionen aufgeführt.
Andere Ressourcen ansehen

Symptome

System memory usage on certain node exceeds 95% of the reservation.

Ursache

Reserved memory ensures system processes can function even when the node is fully allocated and protects against workload out of memory events impacting the proper functioning of the node.

The default reservation is expected to be sufficient for most configurations and should be increased (https://docs.openshift.com/container-platform/latest/nodes/nodes/nodes-nodes-managing.html) when running nodes with high numbers of pods (either due to rate of change or at steady state).

Lösung

Diagnostic

Check the systemReserved configuration applied to the nodes: 

$ oc get nodes
[...]
$ oc debug node/[node_name] -- cat /host/etc/kubernetes/kubelet.conf
[...]
systemReserved:
 cpu: 500m
 memory: 1Gi
 ephemeral-storage: 1Gi
[...]

Check the resources used in the nodes (check the values used by kubelet and runtime if you
want to compare with the configured systemReserved values):

$ oc get --raw /api/v1/nodes/<node>/proxy/stats/summary
[...]
 {
    "node": {
        "nodeName": "cluster.node22",
        "systemContainers": [
            {
                "cpu": {
                    "usageCoreNanoSeconds": 929684480915,
                    "usageNanoCores": 190998084
                },
                "memory": {
                    "rssBytes": 176726016,
                    "usageBytes": 1397895168,
                    "workingSetBytes": 1050509312
                },
                "name": "kubelet"
            },
            {
                "cpu": {
                    "usageCoreNanoSeconds": 128521955903,
                    "usageNanoCores": 5928600
                },
                "memory": {
                    "rssBytes": 35958784,
                    "usageBytes": 129671168,
                    "workingSetBytes": 102416384
                },
                "name": "runtime"
            }
    [...
            ]
        ]
    }
} 
[...]

Mitigation

Increasing the memory reservations to resolve the issue. 

For example, to set resource allocation for the default worker MachineConfigPool , create the following CR inside project openshift-config-operator:

WARNING: This will trigger a serial restart of nodes using this MachineConfigPool: 

apiVersion: machineconfiguration.openshift.io/v1
kind: KubeletConfig
metadata:
 name: set-allocatable 
spec:
 machineConfigPoolSelector:
  matchLabels:
    pools.operator.machineconfiguration.openshift.io/worker: "" 
 kubeletConfig:
  systemReserved:
    cpu: 500m  	                 # change the CPU if needed
    memory: 1512Mi               # change the memory if needed

Support

If all the above steps cannot resolve the issue, contact the Dell EMC technical support for further investigation.

Betroffene Produkte

APEX Cloud Platform for Red Hat OpenShift
Artikeleigenschaften
Artikelnummer: 000217423
Artikeltyp: Solution
Zuletzt geändert: 20 Feb. 2026
Version:  3
Antworten auf Ihre Fragen erhalten Sie von anderen Dell NutzerInnen
Support Services
Prüfen Sie, ob Ihr Gerät durch Support Services abgedeckt ist.