PowerFlex 4.x Unable To Receive MDM Events Alert

• The PowerFlex System is on version 4.x.
• ActiveMQ is up and running on the MDM Manager servers.
• An alert in the PFxI UI shows the following:

UNABLE_TO_RECEIVE_MDM_EVENTS

• Logs from the pod mdm-events-extractor:

svc_collect-<timestamp>/svc_collect-<timestamp>/logs/powerflex/mdm-events-extractor/unnamed_component/mdm-events-extractor-<UID>/mdm-events-extractor.stdout.log

2024-01-25T16:40:47.953291127-06:00 stdout F 2024-01-25 22:40:47,953 [main] (MdmEventsExtractorService.java:178) [DEBUG] ActiveMq  URL is ssl://10.10.10.12:61714?socket.verifyHostName=false
 
2024-01-25T16:41:17.956484522-06:00 stdout F 2024-01-25 22:41:17,956 [main] (MdmEventsExtractorService.java:131) [ERROR] Unable to connect to MDM MdmInfo(id=0ead75972a038400, name=so1, ips=[10.10.10.12])
 
2024-01-25T16:41:17.956519661-06:00 stdout F javax.jms.JMSException: Could not connect to broker URL: ssl://10.10.10.12:61714?socket.verifyHostName=false. Reason: java.net.SocketException: Socket closed

• Port connectivity test shows that the IP or Port is unreachable (The test can be performed from any PowerFlex Manager Platform (PFMP) server to the MDM server with ActiveMQ):

rackpfmp1:~ # kubectl exec -n powerflex -it $(kubectl get pods -n powerflex |grep -i active) -- curl -v https://10.10.10.12:61714
*   Trying 10.10.10.12:61714...
* connect to 10.10.10.12 port 61714 failed: No route to host
* Failed to connect to 10.10.10.12 port 61714 after 0 ms: Couldn't connect to server

Impact

Events from the MDM servers are not viewable in the PFxM UI.

ActiveMQ serves as a message broker agent facilitating communication between various servers. It plays a crucial role in fetching MDM events for display on the PFxM UI. For operation of ActiveMQ, it is imperative that PFMP servers establish effective communication with MDM servers equipped with the ActiveMQ component. To ensure smooth communication, it is essential to have port 61714 opened for external access.

Validate communication through port 61714 on all servers that are Manager MDMs and servers that may potentially be Manager MDMs at some point. If not available, ensure opening the port for external access. Validate that the activemq service is running. If it is running, but an alert is still triggered. Restart the activemq service.

Steps to allow port on the firewall.

1. SSH to the MDM that it complains about
2. Verify that the port is missing under the public (active) section

firewall-cmd --list-all-zones

3. Add the missing port, in this case 61714/tcp

firewall-cmd --permanent --zone=public --add-port=61714/tcp

4. Reload the firewall

firewall-cmd --reload

5. Verify that the alert is gone using PFxM UI or using curl from MVM

kubectl exec -n powerflex -it $(kubectl get pods -n powerflex |grep -i active) -- curl -v https://10.10.10.12:61714

If the issue is not related to the FW policy. The following can be performed to clear the alert.

1. SSH to the IP of the host that is reported in the alert.

Run the following command. 

systemctl restart activemq

       2. Check the Alerts in PFMP to see if the alert has cleared.  

Impacted Versions

PowerFlex Manager 4.x

Fixed In Version

N/A - Working as designed