Reporting and Analytics (DPA) show Application service and Application Server Admin UI as red in ACM after upgrade to IDPA 2.7 or 2.7.1
Summary: This KB covers how to work around DPA Application services not starting after IDPA 2.7 or 2.7.1 upgrade.
Symptoms
After upgrading to IDPA 2.7 or 2.7.1, Reporting and Analytics (DPA) application services does not start and show red marks in ACM UI.
Examining the DPA Application server log located in <dpa_install_path>/services/logs/server.log the following messages will be seen.
2021-12-27 14:14:07,113 INFO [com.emc.apollo.serverstart.pre.PreApolloStartupBean] (ServerService Thread Pool -- 62) Server starting
2021-12-27 14:14:07,114 INFO [com.emc.apollo.services.container.ApolloClusterServicesJbossImpl] (ServerService Thread Pool -- 62) initIsMaster set to true
2021-12-27 14:14:07,115 INFO [com.emc.apollo.serverstart.pre.PreApolloStartupBean] (ServerService Thread Pool -- 62) Apollo Version: 19.5.0.1634796815
2021-12-27 14:14:07,117 WARN [org.jboss.jca.core.connectionmanager.pool.strategy.PoolBySubject] (ServerService Thread Pool -- 62) IJ000604: Throwable while attempting to get a new connection: null: javax.resource.ResourceException: Could not create connection
...
Caused by: org.postgresql.util.PSQLException: Connection to <ip_address_of_DS>:9003 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.
...
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.8.0_301]
And
2021-12-27 14:14:07,120 ERROR [com.emc.apollo.serverstart.pre.PreApolloStartupBean] (ServerService Thread Pool -- 62) exception trying to obtain a connection to the database: com.emc.apollo.jdbc.ApolloSQLException: Could not obtain connection: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:jboss/datasources/ApolloDS
Lastly to confirm the issue, log in to the DPA DS server using PUTTY/SSH as root and run the following command:
dpa.sh ds con
If the DPA APP service is not starting, the output is similar to below:
dpa.sh ds con
EMC Data Protection Advisor
[INFO] Service Bind Address(es) : 127.0.0.1
Client(s) Address(es) : 127.0.0.1/32
::1/128
127.0.0.1/32
::1/128
<DS_ip_addresss>/32
Command completed successfully.
Where <DS_ip_addresss> is the IP Address of the DPA Datastore, example 10.0.0.20.
The identification of the issue has either one or both of the following components:
- In the Service Bind Address (es), the IP Address of the DPA Datastore is not listed.
- In the Client (s) Address (es), the IP Address of the DPA Application is not listed.
In this example, both issues are present.
A proper output from this command should be as below:
dpa.sh ds con
EMC Data Protection Advisor
[INFO] Service Bind Address(es) : <DS_ip_addresss>
Client(s) Address(es) : 127.0.0.1/32
::1/128
127.0.0.1/32
::1/128
<App_ip_addresss>/32
<DS_ip_addresss>/32
Command completed successfully.
Where <DS_ip_addresss> is the IP Address of the DPA Datastore, example 10.0.0.20.
Where <App_ip_addresss> is the IP Address of the DPA Application, example 10.0.0.21.Cause
Resolution
There is no resolution to this issue.
A workaround is available for this issue. Follow the steps below.
Either one of both of the following items must be performed. In some cases only the DPA Datastore bind address must be fixed and in others both the DPA Datastore bind address and the Client Address for the DPA Application must be fixed.
Fixing the DPA Datastore "bind" address.
1. Run the following command on the DPA Datastore server from /opt/emc/epa/services/bin directory:
dpa.sh ds con -b <ip_address_of_the_datastore>
2. Restart the DPA Datastore services.
opt/emc/epa/services/bin directory:
dpa.sh ds restart
3. Restart the DPA Application services.
/opt/emc/epa/services/bin directory:
dpa.sh app restart
Fixing the DPA Application Client Address
1. Run the following command on the DPA Datastore server from /opt/emc/epa/services/bin directory:
dpa.sh ds con -a <ip_address_of_the_dpa_application>
2. Restart the DPA Datastore services.
/opt/emc/epa/services/bin directory:
dpa.sh ds restart
3. Restart the DPA Application services.
/opt/emc/epa/services/bin directory:
dpa.sh app restart
Removing configuration IP if present
If in the output of dpa.sh ds con you see the following IP listed, it must be removed:
10.63.62.242/32
To remove this entry, run the following command on the DPA Datastore server from /opt/emc/epa/services/bin directory:
dpa.sh ds con -d 10.63.62.242
Contact Dell Technical Support for further details or information.