ViPR SRM 3.7.1: Primary Backend Server intermittently drops connectivity in Centralized Management
Summary: After applying an upgrade to a 7 VM ViPR SRM instance the Primary Backend will not stay connected and errors are seen in logs, such as Catalina as well as the Centralized Management >> Servers UI showing a handshake error ...
This article applies to
This article does not apply to
This article is not tied to any specific product.
Not all product versions are identified in this article.
Symptoms
Power-off the VM that was pirating the Primary Backend's IP.
Symptoms:
1) Inconsistent access to Centralized-Management pages from the Reporting Frontend
2) Alerting-frontend was not accessible, errors: The version of the JMX objects in the frontend is not equal to the version of the manager...
3) Once into the Centralized Management interface: Configuration >> Servers showed the PBE being accessible (green checkmark) and 1 min later inaccessible (HTTP SSL errors)
4) Physical Overview graphs were showing no data, even though, they had been all filled previously
5) On the Primary Backend could not use the manage-resources.sh list command - returned errors as found in Catalina regarding MasterDatasourceManager (see below)
Catalina Errors:
WARNING: Unable to retrieve host ID for server 'fqdn.hostname.emc.com - Primary Backend'!
com.sun.xml.internal.ws.client.ClientTransportException: HTTP transport error: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
Caused by: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
Caused by: java.io.EOFException: SSL peer shut down incorrectly
at sun.security.ssl.InputRecord.read(InputRecord.java:505)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
One minute later Catalina shows additional errors:
Nov 27, 2016 10:09:25 AM com.watch4net.apg.v2.gui.resource.ExternalResourceManager refreshResources
WARNING: Cannot load resource definitions from MasterDatasourceManager, ignoring....
com.watch4net.apg.gui.master.accessor.MasterDataAccessException: Resources cannot be loaded from master database!
Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Could not create connection to database server. Attempted reconnect 3 times. Giving up.
Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Too many connections
When attempting to log into the Centralized Management from the Reporting Frontend the Centralized-Managment page could not be served, throwing a 500 Server error ... stating to review the logs which showed the following information:
WARNING: Access to web application '/centralized-management' has been denied to user 'admin' (Reason: User 'admin' not found!).
Nov 27, 2016 10:26:22 AM com.opensymphony.xwork2.util.logging.commons.CommonsLogger error
SEVERE: Exception occurred during processing request: User 'admin' not found!
java.lang.IllegalStateException: User 'admin' not found!
This was intermittent in nature and the connectivity continued to fluctuate, working for a brief time and then connectivity failing again.
Symptoms:
1) Inconsistent access to Centralized-Management pages from the Reporting Frontend
2) Alerting-frontend was not accessible, errors: The version of the JMX objects in the frontend is not equal to the version of the manager...
3) Once into the Centralized Management interface: Configuration >> Servers showed the PBE being accessible (green checkmark) and 1 min later inaccessible (HTTP SSL errors)
4) Physical Overview graphs were showing no data, even though, they had been all filled previously
5) On the Primary Backend could not use the manage-resources.sh list command - returned errors as found in Catalina regarding MasterDatasourceManager (see below)
Catalina Errors:
WARNING: Unable to retrieve host ID for server 'fqdn.hostname.emc.com - Primary Backend'!
com.sun.xml.internal.ws.client.ClientTransportException: HTTP transport error: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
Caused by: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
Caused by: java.io.EOFException: SSL peer shut down incorrectly
at sun.security.ssl.InputRecord.read(InputRecord.java:505)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
One minute later Catalina shows additional errors:
Nov 27, 2016 10:09:25 AM com.watch4net.apg.v2.gui.resource.ExternalResourceManager refreshResources
WARNING: Cannot load resource definitions from MasterDatasourceManager, ignoring....
com.watch4net.apg.gui.master.accessor.MasterDataAccessException: Resources cannot be loaded from master database!
Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Could not create connection to database server. Attempted reconnect 3 times. Giving up.
Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Too many connections
When attempting to log into the Centralized Management from the Reporting Frontend the Centralized-Managment page could not be served, throwing a 500 Server error ... stating to review the logs which showed the following information:
WARNING: Access to web application '/centralized-management' has been denied to user 'admin' (Reason: User 'admin' not found!).
Nov 27, 2016 10:26:22 AM com.opensymphony.xwork2.util.logging.commons.CommonsLogger error
SEVERE: Exception occurred during processing request: User 'admin' not found!
java.lang.IllegalStateException: User 'admin' not found!
This was intermittent in nature and the connectivity continued to fluctuate, working for a brief time and then connectivity failing again.
Cause
Two separately deployed ViPR SRM instances had VM's that were deployed using the same IP (different hostnames).
Resolution
The cause of these errors and being unable to consistently retain communication with the ViPR SRM Primary Backend server was found to be an issue with IP duplication. Once the secondary VM host was powered off the issues with the webservice-gateway, database resources, etc. corrected themselves.
Affected Products
Storage SoftwareArticle Properties
Article Number: 000065789
Article Type: Solution
Last Modified: 31 Jan 2025
Version: 4
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.