Avamar: DPE vcpsrv service is down due to failed connection with external rabbitmq.

vcpsrv service on DPE fails to start with the following error:
 

2023-01-25 09:56:48,601 [main] INFO  (StartupInfoLogger.java:48) - Starting Application v19.7.0.90 on  with PID 6 (/opt/vcp/vcpsrv/vcp-server.jar started by root in /opt/vcp/vcpsrv)
2023-01-25 09:56:48,610 [main] INFO  (SpringApplication.java:593) - No active profile set, falling back to default profiles: default
2023-01-25 09:56:51,109 [main] INFO  (CstServiceManager.java:147) - Creating Service Manager
2023-01-25 09:56:52,045 [main] INFO  (CstServiceManager.java:161) - Done: Created Service manager
2023-01-25 09:56:52,046 [main] INFO  (VcpCstService.java:529) - Created the CstServiceManager successfully
2023-01-25 09:56:52,049 [main] INFO  (CstLBStore.java:292) - Creating LockBox Store
2023-01-25 09:56:52,053 [main] INFO  (CstLBStore.java:298) - Done: Created LockBox Store
2023-01-25 09:56:52,054 [main] INFO  (VcpCstService.java:542) - Created the LBStore successfully
2023-01-25 09:56:53,160 [main] INFO  (CredentialedDataSource.java:122) - Postgres Version: PostgreSQL 9.6.13 on x86_64-pc-linux-gnu, compiled by gcc (SUSE Linux) 4.8.5, 64-bit
2023-01-25 09:56:53,227 [main] INFO  (Migration.java:44) - Starting Database Validation and Migration...
2023-01-25 09:56:53,240 [main] INFO  (Migration.java:53) - Found table 'schema_version', no need to initialize.
2023-01-25 09:56:54,637 [main] WARN  (SequenceGenerator.java:87) - HHH90000014: Found use of deprecated [org.hibernate.id.SequenceGenerator] sequence-based id generator; use org.hibernate.id.enhanced.SequenceStyleGenerator instead.
 See Hibernate Domain Model Mapping Guide for details.
2023-01-25 09:56:54,646 [main] WARN  (SequenceGenerator.java:87) - HHH90000014: Found use of deprecated [org.hibernate.id.SequenceGenerator] sequence-based id generator; use org.hibernate.id.enhanced.SequenceStyleGenerator instead.
 See Hibernate Domain Model Mapping Guide for details.
2023-01-25 09:56:57,631 [main] INFO  (VcdClient.java:715) - Watchdog: vCloud session created: 1674640617631
2023-01-25 09:56:57,633 [main] INFO  (VcdClient.java:282) - VCP-Service Version: 19.7.0.90
2023-01-25 09:56:57,714 [main] WARN  (AbstractApplicationContext.java:551) - Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'amqpConfiguration': Invocation of init method failed; nested exception is org.springframework.amqp.AmqpIOException: javax.net.ssl.SSLException: readHandshakeRecord
2023-01-25 09:56:57,715 [main] INFO  (VcdClient.java:287) - Destroying VcdClient...
2023-01-25 09:56:57,728 [main] ERROR (SpringApplication.java:771) - Application startup failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'amqpConfiguration': Invocation of init method failed; nested exception is org.springframework.amqp.AmqpIOException: javax.net.ssl.SSLException: readHandshakeRecord
        at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:137) ~[spring-beans-4.3.17.RELEASE.jar!/:4.3.17.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:409) ~[spring-beans-4.3.17.RELEASE.jar!/:4.3.17.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1626) ~[spring-beans-4.3.17.RELEASE.jar!/:4.3.17.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:555) ~[spring-beans-4.3.17.RELEASE.jar!/:4.3.17.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483) ~[spring-beans-4.3.17.RELEASE.jar!/:4.3.17.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:312) ~[spring-beans-4.3.17.RELEASE.jar!/:4.3.17.RELEASE]
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230) ~[spring-beans-4.3.17.RELEASE.jar!/:4.3.17.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:308) ~[spring-beans-4.3.17.RELEASE.jar!/:4.3.17.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197) ~[spring-beans-4.3.17.RELEASE.jar!/:4.3.17.RELEASE]
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:761) ~[spring-beans-4.3.17.RELEASE.jar!/:4.3.17.RELEASE]
        at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:867) ~[spring-context-4.3.17.RELEASE.jar!/:4.3.17.RELEASE]
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:543) ~[spring-context-4.3.17.RELEASE.jar!/:4.3.17.RELEASE]
        at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:693) ~[spring-boot-1.5.13.RELEASE.jar!/:1.5.13.RELEASE]
        at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:360) ~[spring-boot-1.5.13.RELEASE.jar!/:1.5.13.RELEASE]
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:303) ~[spring-boot-1.5.13.RELEASE.jar!/:1.5.13.RELEASE]
        at com.emc.vcp.Application.main(Application.java:51) ~[classes!/:19.7.0.90]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_301]
        at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[?:1.8.0_301]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:1.8.0_301]
        at java.lang.reflect.Method.invoke(Unknown Source) ~[?:1.8.0_301]
        at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48) ~[vcp-server.jar:19.7.0.90]
        at org.springframework.boot.loader.Launcher.launch(Launcher.java:87) ~[vcp-server.jar:19.7.0.90]
        at org.springframework.boot.loader.Launcher.launch(Launcher.java:50) ~[vcp-server.jar:19.7.0.90]
        at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:51) ~[vcp-server.jar:19.7.0.90]
Caused by: org.springframework.amqp.AmqpIOException: javax.net.ssl.SSLException: readHandshakeRecord
        at org.springframework.amqp.rabbit.support.RabbitExceptionTranslator.convertRabbitAccessException(RabbitExceptionTranslator.java:71) ~[spring-rabbit-1.7.7.RELEASE.jar!/:?]
        at org.springframework.amqp.rabbit.connection.AbstractConnectionFactory.createBareConnection(AbstractConnectionFactory.java:376) ~[spring-rabbit-1.7.7.RELEASE.jar!/:?]
.
.
.
.
Caused by: java.net.SocketException: Connection reset
        at java.net.SocketOutputStream.socketWrite(Unknown Source) ~[?:1.8.0_301]
        at java.net.SocketOutputStream.write(Unknown Source) ~[?:1.8.0_301]
        at sun.security.ssl.SSLSocketOutputRecord.flush(Unknown Source) ~[?:1.8.0_301]
.
.
.
.
.       at sun.security.ssl.HandshakeOutStream.flush(Unknown Source) ~[?:1.8.0_301]
        at sun.security.ssl.ECDHClientKeyExchange$ECDHEClientKeyExchangeProducer.produce(Unknown Source) ~[?:1.8.0_301]
        at sun.security.ssl.ClientKeyExchange$ClientKeyExchangeProducer.produce(Unknown Source) ~[?:1.8.0_301]
        at sun.security.ssl.SSLHandshake.produce(Unknown Source) ~[?:1.8.0_301]
        at sun.security.ssl.ServerHelloDone$ServerHelloDoneConsumer.consume(Unknown Source) ~[?:1.8.0_301]
        at sun.security.ssl.SSLHandshake.consume(Unknown Source) ~[?:1.8.0_301]
        at sun.security.ssl.HandshakeContext.dispatch(Unknown Source) ~[?:1.8.0_301]
        at sun.security.ssl.HandshakeContext.dispatch(Unknown Source) ~[?:1.8.0_301]
        at sun.security.ssl.TransportContext.dispatch(Unknown Source) ~[?:1.8.0_301]
        at sun.security.ssl.SSLTransport.decode(Unknown Source) ~[?:1.8.0_301]
        at sun.security.ssl.SSLSocketImpl.decode(Unknown Source) ~[?:1.8.0_301]
        at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(Unknown Source) ~[?:1.8.0_301]
        at sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) ~[?:1.8.0_301]
        at sun.security.ssl.SSLSocketImpl.ensureNegotiated(Unknown Source) ~[?:1.8.0_301]
        at sun.security.ssl.SSLSocketImpl.access$200(Unknown Source) ~[?:1.8.0_301]
        at sun.security.ssl.SSLSocketImpl$AppOutputStream.write(Unknown Source) ~[?:1.8.0_301]
        at java.io.BufferedOutputStream.flushBuffer(Unknown Source) ~[?:1.8.0_301]
        at java.io.BufferedOutputStream.flush(Unknown Source) ~[?:1.8.0_301]
        at java.io.DataOutputStream.flush(Unknown Source) ~[?:1.8.0_301]
        at com.rabbitmq.client.impl.SocketFrameHandler.sendHeader(SocketFrameHandler.java:147) ~[amqp-client-4.0.3.jar!/:4.0.3]
.
.
.
.
2023-01-25 09:58:11,558 [main] INFO  (VcdClient.java:282) - VCP-Service Version: 19.7.0.90
2023-01-25 09:58:11,632 [main] WARN  (AbstractApplicationContext.java:551) - Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'amqpConfiguration': Invocation of init method failed; nested exception is org.springframework.amqp.AmqpIOException: javax.net.ssl.SSLException: readHandshakeRecord
2023-01-25 09:58:11,633 [main] INFO  (VcdClient.java:287) - Destroying VcdClient...
2023-01-25 09:58:11,644 [main] ERROR (SpringApplication.java:771) - Application startup failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'amqpConfiguration': Invocation of init method failed; nested exception is org.springframework.amqp.AmqpIOException: javax.net.ssl.SSLException: readHandshakeRecord
        at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:137) ~[spring-beans-4.3.17.RELEASE.jar!/:4.3.17.RELEASE]

The VCP RabbitMQ TLS handshake is trying to do a mutual tls authentication and its handshake is being reset by RabbitMQ  after vcpsrv sends the DPE certificate. 

The external RabbitMQ server needs to be told to trust the DPE root certificate used to sign vcpsrv (and vcpbg ) certificates to allow RabbitMQ traffic to be mutual tls authenticated.

Follow the steps below to achieve this:

1- Use keytool to fetch the DPE root CA (and dpem cert).

On DPE node: 

keytool -printcert -sslserver localhost:9000 -rfc | sed -n '/-----BEGIN CERTIFICATE-----/,/-----END CERTIFICATE-----/p' | awk '/-----BEGIN CERTIFICATE-----/{i++}{print > "cert"i".pem"}'

This should create a cert1.pem and cert2.pem file. 
cert2.pem should be DPE root ca.

2- Add another CAfile from step A to the customer's rabbitmq.config on 1st rabbitmq node.

On Rabbitmq node:
Edit the file at /etc/rabbitmq/ssl/cacert.pem and add the contents of /path/to/cert2.pem at the end of the file:

{cacertfile, "/etc/rabbitmq/ssl/cacert.pem"}

After this change, restart RabbitMQ for the changes to take effect.

3- Repeat step 2 for other rabbitmq nodes.

In case of a DPE purge because of lockbox or other issue, the following steps are needed.
The DPE purge will result in new rootca certificates being generated on the DPE side.
 
This can then cause an issue when the SRV service is starting and trying to connect to the customers external rabbitMQ server.
 
In the vcpserver.log:

WARN (AbstractApplicationContext.java:551) - Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'amqpConfiguration': Invocation of init method failed; nested exception is org.springframework.amqp.AmqpIOException: javax.net.ssl.SSLException: readHandshakeRecord
2024-04-29 13:35:16,654 [main] INFO (VcdClient.java:287) - Destroying VcdClient...
2024-04-29 13:35:16,665 [main] ERROR (SpringApplication.java:771) - Application startup failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'amqpConfiguration': Invocation of init method failed; nested exception is org.springframework.amqp.AmqpIOException: javax.net.ssl.SSLException: readHandshakeRecord

The above error is caused by the customer external rabbitMQ server rejecting our DPE cert.
 
To resolve, follow the steps below:

1) The vcp-purge would have generated a new DPE root CA in the following folder: /root/certs/ca.pem
2) Copy the contents of this file, and append the new ca.pem to external rabbitmq server (change on all RMQ in cluster) "/etc/rabbitmq/ca/rootca.pem" .
4) There is no need to restart the services on the rabbitmq side.