PowerStore: NAS Out of Memory Condition Leads to Node Reboot
Summary: Node reboot or panic due to NAS out of memory (OOM) condition
Symptoms
The SDNAS container is killed due to an out of memory condition leading to a node panic.
This article does not apply for panics during a PowerStoreOS upgrade or NAS server move.
This issue affects PowerStore clusters running versions between 3.5.x and 3.6.1.1.
Cause
The issue is a result of a new design for SDNAS interaction with backend block devices that started with PowerStore version 3.5.
Due to an issue in the heartbeat implementation, memory that is consumed may not be freed, which leads to an OOM condition and consequent panic.
Resolution
The fix is available in PowerStoreOS 3.6.1.2 and PowerStoreOS 4.0.0.0 and later code.
Contact Dell Technical Support or your Authorized Service Representative, and quote this article ID if any assistance is required.
Additional Information
Memory usage for the NAS container is checked in real time over SSH using the below command (login with the service user):
egrep "new snap|^CONT.*MEM|cyc_nas_docker.*GiB" /cyc_var/resources/memory-node-?.log | tail -3
Use svc_ssh_peer to change to the peer node and run the same command.
Example:
[SVC:service@1234567-B user]$ egrep "new snap|^CONT.*MEM|cyc_nas_docker.*GiB" /cyc_var/resources/memory-node-?.log | tail -3 ------------------- new snapshot at Thu May 2 12:40:49 UTC 2024 ------------------- CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS xxxxxxxxxxxx cyc_nas_docker 0.57% 8.854GiB / 28GiB 31.62% 0B / 0B 1.44TB / 41.3GB x [SVC:service@1234567-B user]$ svc_ssh_peer [SVC:service@1234567-A user]$ egrep "new snap|^CONT.*MEM|cyc_nas_docker.*GiB" /cyc_var/resources/memory-node-?.log | tail -3 ------------------- new snapshot at Thu May 2 12:41:54 UTC 2024 ------------------- CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS yyyyyyyyyyyy cyc_nas_docker 0.44% 7.08GiB / 28GiB 95.28% 0B / 0B 328GB / 4.29GB x [SVC:service@1234567-A user]$
To see the memory usage progression over time, remove the | tail -3 portion of the command.
If the memory usage is higher than 80%, the affected node may be proactively rebooted outside of peak hours to minimize impact. Alternatively, contact Dell Technical Support to restart only the NAS container instead of the node.