PowerFlex Manager 3.8: Out-OfMemory Java Heap Errors Lead To Poor Performance

The PFxM UI exhibits poor performance and can be slow to log in and slow to pull up different sections within the UI. Various tasks, such as resource inventory, deploying a new service, or adding an existing service, may unexpectedly fail.

The asmui and asmManager logs shows Out-OfMemory errors:

2024-06-01 01:17:24 [http-nio-9030-exec-4] (DeviceController.java:626) [ERROR] getDeviceList() - Exception from service call
java.lang.OutOfMemoryError: Java heap space
2024-06-01 01:17:24 [http-nio-9030-exec-4] (BaseController.java:304) [ERROR] Found generic exception in Controller
java.lang.OutOfMemoryError: Java heap space
2024-06-01 01:17:24 [http-nio-9030-exec-9] (DeviceController.java:626) [ERROR] getDeviceList() - Exception from service call
java.lang.OutOfMemoryError: Java heap space
2024-06-01 01:17:24 [http-nio-9030-exec-9] (BaseController.java:304) [ERROR] Found generic exception in Controller
java.lang.OutOfMemoryError: Java heap space
2024-06-01 01:17:26 [http-nio-9030-exec-6] (DeviceController.java:626) [ERROR] getDeviceList() - Exception from service call
java.lang.OutOfMemoryError: Java heap space
2024-06-01 01:17:26 [http-nio-9030-exec-6] (BaseController.java:304) [ERROR] Found generic exception in Controller
java.lang.OutOfMemoryError: Java heap space
...
2024-07-10 00:40:50 [http-nio-9030-exec-11] (RestTemplateResponseErrorHandler.java:47) [ERROR] REST Error: {
  "detailMessage" : "Handler dispatch failed; nested exception is java.lang.OutOfMemoryError: Java heap space",
  "cause" : {
    "cause" : {
      "cause" : null,
      "stackTrace" : [ ],
      "message" : "Java heap space",
      "localizedMessage" : "Java heap space",
      "suppressed" : [ ]
...
"stackTrace" : [ ],
  "suppressedExceptions" : [ ],
  "status" : 500,
  "timestamp" : "2024-07-10T04:40:50.986+00:00",
  "error" : "Internal Server Error",
  "path" : "/AsmManager/job/",
  "messages" : [ {
    "messageBundle" : null,
    "messageCode" : null,
    "severity" : null,
    "category" : null,
    "displayMessage" : "Runtime error",
    "responseAction" : null,
    "detailedMessage" : "Handler dispatch failed; nested exception is java.lang.OutOfMemoryError: Java heap space",
    "agentId" : null,
    "correlationId" : null,
    "timeStamp" : "2024-07-10T04:40:50.987+00:00",
    "sequenceNumber" : 0
  } ]
}
2024-07-10 00:40:50 [http-nio-9030-exec-11] (JobsController.java:83) [ERROR] getJobs() - Exception from service call
com.dell.asm.rest.common.exception.LocalizedWebApplicationException: HTTP 500 Internal Server Error
    at com.dell.pfxm.springboot.helpers.error.RestTemplateResponseErrorHandler.handleError(RestTemplateResponseErrorHandler.java:64) ~[RestCommon-0.7.0-SNAPSHOT.jar!/:?]
...

Impact

Java heap memory exhaustion leads to poor PFxM UI performance and random unexpected failures against various tasks.

The asmui and asmManager components have a default Java heap memory range of 128M-512M and 512M-6144M respectively. In larger environments, this memory range may not be adequate to allow the various processes to correctly run, leading to heap memory exhaustion.

Environments with a large number of objects in PFxM (e.g. 100+ resources, 3+ services with 32+ nodes, 10+ total services) will benefit from a larger Java heap memory pool.

The Java heap size can be increased for the asumui and asmManager components. In the past, this was commonly seen as the 'large-scale patch' because a patch had to be applied to the current PFxM instance and did not persist across PFxM upgrades.

A patch is no longer required as this update is applied to configuration files that persist across the PFxM upgrade.

1. From PFxM CLI, back up the current asmui and asmManager Java configuration files:

 sudo cp /opt/Dell/ASM/conf/asmui-jvm-props.conf asmui-jvm-props.conf.bak
 
sudo cp /opt/Dell/ASM/conf/asmManager-jvm-props.conf asmManager-jvm-props.conf.bak

2. Take a snapshot of the PFxM VM in Center. 

3. Increase the asmui heap upper range to 2048M:

 echo '-Xloggc:/opt/Dell/ASM/logs/gc_asmui.log -XX:+PrintHeapAtGC -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=10M -XX:-HeapDumpOnOutOfMemoryError -XX:HeapDumpPath="/opt/Dell/ASM/logs/" -Xms128m -Xmx2048m -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -Dhttp.nonProxyHosts="localhost.localdomain|localhost|127.0.0.1" -Djava.io.tmpdir=/opt/Dell/ASM/temp -Dspring.profiles.active=main' > /opt/Dell/ASM/conf/asmui-jvm-props.conf

4. Increase the asmManager heap upper range to 18432M:

 echo '-Xloggc:/opt/Dell/ASM/logs/gc_asmManager.log -XX:+PrintHeapAtGC -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=10M -XX:-HeapDumpOnOutOfMemoryError -XX:HeapDumpPath="/opt/Dell/ASM/logs/" -Xms512m -Xmx18432m -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -Dhttp.nonProxyHosts="localhost.localdomain|localhost|127.0.0.1" -Djava.io.tmpdir=/opt/Dell/ASM/temp -Dspring.profiles.active=main' > /opt/Dell/ASM/conf/asmManager-jvm-props.conf

5. Restart the PFxM VM.

6. Verify that the configuration changes are applied:

 cat /opt/Dell/ASM/conf/asmui-jvm-props.conf | grep -E 'Xms|Xmx'
 
cat /opt/Dell/ASM/conf/asmManager-jvm-props.conf | grep -E 'Xms|Xmx'

7. After the changes have been implemented and the issue is resolved, delete the PFxM VM snapshot in Center. 

Impacted

 Versions 3.8.6 and newer