Openshift Event Code: 1038NODE0008
Summary: System Memory Exceeds Reservation
Symptoms
Cause
Reserved memory ensures system processes can function even when the node is fully allocated and protects against workload out of memory events impacting the proper functioning of the node.
The default reservation is expected to be sufficient for most configurations and should be increased (https://docs.openshift.com/container-platform/latest/nodes/nodes/nodes-nodes-managing.html) when running nodes with high numbers of pods (either due to rate of change or at steady state).
Resolution
Diagnostic
Check the systemReserved configuration applied to the nodes:
$ oc get nodes [...] $ oc debug node/[node_name] -- cat /host/etc/kubernetes/kubelet.conf [...] systemReserved: cpu: 500m memory: 1Gi ephemeral-storage: 1Gi [...] |
Check the resources used in the nodes (check the values used by kubelet and runtime if you
want to compare with the configured systemReserved values):
$ oc get --raw /api/v1/nodes/<node>/proxy/stats/summary
[...]
{
"node": {
"nodeName": "cluster.node22",
"systemContainers": [
{
"cpu": {
"usageCoreNanoSeconds": 929684480915,
"usageNanoCores": 190998084
},
"memory": {
"rssBytes": 176726016,
"usageBytes": 1397895168,
"workingSetBytes": 1050509312
},
"name": "kubelet"
},
{
"cpu": {
"usageCoreNanoSeconds": 128521955903,
"usageNanoCores": 5928600
},
"memory": {
"rssBytes": 35958784,
"usageBytes": 129671168,
"workingSetBytes": 102416384
},
"name": "runtime"
}
[...
]
]
}
}
[...]
|
Mitigation
Increasing the memory reservations to resolve the issue.
For example, to set resource allocation for the default worker MachineConfigPool , create the following CR inside project openshift-config-operator:
WARNING: This will trigger a serial restart of nodes using this MachineConfigPool:
apiVersion: machineconfiguration.openshift.io/v1
kind: KubeletConfig
metadata:
name: set-allocatable
spec:
machineConfigPoolSelector:
matchLabels:
pools.operator.machineconfiguration.openshift.io/worker: ""
kubeletConfig:
systemReserved:
cpu: 500m # change the CPU if needed
memory: 1512Mi # change the memory if needed |
Support
If all the above steps cannot resolve the issue, contact the Dell EMC technical support for further investigation.