PowerFlex 管理平台：由于 Java 堆空间，PowerFlex Gateway API 调用失败

Summary: 由于 PowerFlex Gateway Pod 中的 Java 堆空间耗尽，来自 OpenStack 的 API 调用以及与 PowerFlex 管理平台 （PFMP） 的其他类似集成失败。重新启动受影响的数据块网关 Pod 可暂时解决此问题。

This article applies to This article does not apply to This article is not tied to any specific product. Not all product versions are identified in this article.
Check out other resources

Symptoms

  • 对 PowerFlex Manager （PFMP） 的 API 调用间歇性失败，从而影响 OpenStack 和其他集成。
  • block-gateway-0 Pod 报告 OutOfMemoryError，请求将重定向到 block-gateway-1。 
    06:30:21.635 [|https-jsse-nio-443-exec-4-ReceiveLoop-254285513] ERROR c.e.e.n.c.LayeredProtocolConnection - LPC-254285513 #e1a663711aca639e [xxx.xxx.xxx.xxx:8611]: IOException in receive loop: message=null, e=java.io.EOFException, cause=null
06:30:21.635 [|https-jsse-nio-443-exec-5-ReceiveLoop-838015790] ERROR c.e.e.n.c.LayeredProtocolConnection - LPC-838015790 #ed300a36c5228121 [xxx.xxx.xxx.xxx:8611]: IOException in receive loop: message=null, e=java.io.EOFException, cause=null
06:30:23.475 [|https-jsse-nio-443-exec-11] ERROR c.e.s.s.w.c.SpecialImplController - Got an exception in handleException
org.springframework.web.util.NestedServletException: Handler dispatch failed; nested exception is java.lang.OutOfMemoryError: Java heap space
  • 最终 block-gateway-1 Pod 还报告 OutOfMemoryError。 
    07:03:30.663 [|https-jsse-nio-443-exec-15] WARN  o.s.w.s.m.m.a.ExceptionHandlerExceptionResolver - Failure in @ExceptionHandler com.emc.s3g.scaleio.web.controller.RestApiController#handleException(Exception, HttpServletRequest, HttpServletResponse)
java.lang.OutOfMemoryError: Java heap space
07:03:42.234 [|https-jsse-nio-443-exec-5-ReceiveLoop-793498724] ERROR c.e.e.n.c.LayeredProtocolConnection - LPC-793498724 #7f7d737257ceb2e0 [10.41.209.52:8611]: Problem with message deserialization
java.lang.OutOfMemoryError: Java heap space
07:03:42.234 [|https-jsse-nio-443-exec-12] ERROR c.e.s.s.s.ConnectionServiceImpl - Got rc MESSAGE_DESERIALIZATION_FAILED for command QueryPoll
  • catalina.out 日志指示 中的故障 HostConfig.checkResources()，表明资源验证期间内存耗尽。 
    16-Jan-2025 06:31:21.628 SEVERE [Catalina-utility-2] org.apache.catalina.core.ContainerBase.threadStart Exception processing background thread
        Caused by: java.lang.OutOfMemoryError: Java heap space
                at org.apache.catalina.startup.HostConfig.checkResources(HostConfig.java:1298)
  • 资源指标显示受影响的网关 Pod 没有显著的 CPU 或内存消耗。 
    Namespace                   Name                                                     CPU Requests  CPU Limits  Memory Requests  Memory Limits  Age
---------                   ----                                                     ------------  ----------  ---------------  -------------  ---
powerflex                   block-legacy-gateway-1                                   0 (0%)        0 (0%)      0 (0%)           0 (0%)         56d
powerflex                   ciam-service-8c98989cb-5prsg                             100m (0%)     0 (0%)      128Mi (0%)       0 (0%)         56d
powerflex                   block-legacy-gateway-0                                   0 (0%)        0 (0%)      0 (0%)           0 (0%)         56d

影响

  • 对 PFMP 的 API 调用间歇性失败，可能会影响自动化和管理作。
  • 由于 Java 堆空间耗尽，网关 Pod 变得无响应。
  • 恢复服务需要手动干预。

 

Cause

This issue is triggered by block-legacy-gateway POST requests and the following API requests: "/api/gatewayLogin", "/api/logout", and "/api/gatewayLogout". The pod handling these requests does not fully release the associated memory upon completion, leading to a memory leak.

 

Resolution

重新启动受影响的 block-gateway pod，使用以下命令：

kubectl -n powerflex rollout restart statefulset block-legacy-gateway

 

 

Additional Information

受影响的版本

PFMP v4.6.0.x 和 v4.6.1

已修复问题的版本

PFMP v4.6.2 及更高版本

 

Article Properties
Article Number: 000297144
Article Type: Solution
Last Modified: 24 Mar 2025
Version:  1
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.