PowerFlex 3.x: The Presentation Server stops logging alerts in the UI.

In this scenario, the Presentation Server (mgmt-server) is installed on an SDS node.

Events from the Primary Meta Data Manager (MDM) indicate that the SDS on which the mgmt-server is installed is being placed in Maintenance Mode and rebooted.
 

2023-07-19 23:39:45.351000:0237719:MDM_CLI_CONF_COMMAND_RECEIVED    INFO     Command enter_maintenance_mode received, User: ': user-a'. [3246236] SDS: ID: 7fcf39320000000c;
2023-07-19 23:40:02.906000:0237721:SDS_DECOUPLED                    ERROR    SDS: sds-03 (id: 7fcf39320000000c) decoupled.
2023-07-19 23:58:58.776000:0237726:SDS_RECONNECTED                  INFO     SDS: sds-03 (ID 7fcf39320000000c) reconnected.

Impact:
Alerts are not shown in the UI.

/var/log/messages show the node being rebooted.
 

Jul 19 23:47:27 sds-03 systemd-shutdownd: Shutting down at Wed 2023-07-19 23:48:27 BST (reboot)...

When the node boots up, the mgmt-server service initiates but fails when attempting to start the Alert Manager service. 

The scaleio.log from the mgmt-server may show the following errors.
 

2023-07-19 23:58:39,662 [main|INFO |com.emc.vxflexos.webui.backend.Server.main] Starting PowerFlex MGMT server...
2023-07-19 23:58:44,796 [main|INFO |com.emc.vxflexos.webui.backend.services.ServicesModule.lambda$null$1] Registering service AlertsManagerService to ServiceManager
2023-07-19 23:58:45,804 [AlertsManagerService STARTING|INFO |com.emc.vxflexos.webui.backend.services.alerts.AlertsManagerService.lambda$new$2] Alerts observable got subscribed
2023-07-19 23:58:45,811 [RxSingleScheduler-1|ERROR|com.emc.vxflexos.webui.backend.services.alerts.AlertsManagerService.onError] Error occurred:
java.util.ConcurrentModificationException: null
    at java.util.HashMap.computeIfAbsent(HashMap.java:1135) ~[?:?]
    at com.slimgears.util.autovalue.annotations.PropertyMeta$1.getAnnotation(PropertyMeta.java:130) ~[auto-value-utils-annotations-0.7.46.jar:?]
    at com.slimgears.util.autovalue.annotations.PropertyMeta.hasAnnotation(PropertyMeta.java:76) ~[auto-value-utils-annotations-0.7.46.jar:?]
    at com.slimgears.rxrepo.util.PropertyExpressions.lambda$mandatoryProperties$13(PropertyExpressions.java:267) ~[rxrepo-core-0.7.130.jar:?]
...

Following a node reboot, the Presentation Server may not display alerts. This sporadic issue appears to be associated with timing or a race condition during the node's startup.
 

 

To work around the issue, restart the mgmt-server service to get the alerts back.

systemctl restart mgmt-server.service

Impacted Versions

PowerFlex 3.6.3 and below

Fixed In Version

PowerFlex 3.6.4

The Presentation Server stops logging alerts in the UI - PowerFlex Escalation Engineering - ISG Confluence (internal)