Avamar：DPE vcpsrv 服務因與外接式 rabbitmq 的連線失敗而中斷。

DPE 上的 vcpsrv 服務無法啟動，並出現下列錯誤：
 

2023-01-25 09:56:48,601 [main] INFO  (StartupInfoLogger.java:48) - Starting Application v19.7.0.90 on  with PID 6 (/opt/vcp/vcpsrv/vcp-server.jar started by root in /opt/vcp/vcpsrv)
2023-01-25 09:56:48,610 [main] INFO  (SpringApplication.java:593) - No active profile set, falling back to default profiles: default
2023-01-25 09:56:51,109 [main] INFO  (CstServiceManager.java:147) - Creating Service Manager
2023-01-25 09:56:52,045 [main] INFO  (CstServiceManager.java:161) - Done: Created Service manager
2023-01-25 09:56:52,046 [main] INFO  (VcpCstService.java:529) - Created the CstServiceManager successfully
2023-01-25 09:56:52,049 [main] INFO  (CstLBStore.java:292) - Creating LockBox Store
2023-01-25 09:56:52,053 [main] INFO  (CstLBStore.java:298) - Done: Created LockBox Store
2023-01-25 09:56:52,054 [main] INFO  (VcpCstService.java:542) - Created the LBStore successfully
2023-01-25 09:56:53,160 [main] INFO  (CredentialedDataSource.java:122) - Postgres Version: PostgreSQL 9.6.13 on x86_64-pc-linux-gnu, compiled by gcc (SUSE Linux) 4.8.5, 64-bit
2023-01-25 09:56:53,227 [main] INFO  (Migration.java:44) - Starting Database Validation and Migration...
2023-01-25 09:56:53,240 [main] INFO  (Migration.java:53) - Found table 'schema_version', no need to initialize.
2023-01-25 09:56:54,637 [main] WARN  (SequenceGenerator.java:87) - HHH90000014: Found use of deprecated [org.hibernate.id.SequenceGenerator] sequence-based id generator; use org.hibernate.id.enhanced.SequenceStyleGenerator instead.
 See Hibernate Domain Model Mapping Guide for details.
2023-01-25 09:56:54,646 [main] WARN  (SequenceGenerator.java:87) - HHH90000014: Found use of deprecated [org.hibernate.id.SequenceGenerator] sequence-based id generator; use org.hibernate.id.enhanced.SequenceStyleGenerator instead.
 See Hibernate Domain Model Mapping Guide for details.
2023-01-25 09:56:57,631 [main] INFO  (VcdClient.java:715) - Watchdog: vCloud session created: 1674640617631
2023-01-25 09:56:57,633 [main] INFO  (VcdClient.java:282) - VCP-Service Version: 19.7.0.90
2023-01-25 09:56:57,714 [main] WARN  (AbstractApplicationContext.java:551) - Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'amqpConfiguration': Invocation of init method failed; nested exception is org.springframework.amqp.AmqpIOException: javax.net.ssl.SSLException: readHandshakeRecord
2023-01-25 09:56:57,715 [main] INFO  (VcdClient.java:287) - Destroying VcdClient...
2023-01-25 09:56:57,728 [main] ERROR (SpringApplication.java:771) - Application startup failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'amqpConfiguration': Invocation of init method failed; nested exception is org.springframework.amqp.AmqpIOException: javax.net.ssl.SSLException: readHandshakeRecord
        at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:137) ~[spring-beans-4.3.17.RELEASE.jar!/:4.3.17.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:409) ~[spring-beans-4.3.17.RELEASE.jar!/:4.3.17.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1626) ~[spring-beans-4.3.17.RELEASE.jar!/:4.3.17.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:555) ~[spring-beans-4.3.17.RELEASE.jar!/:4.3.17.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483) ~[spring-beans-4.3.17.RELEASE.jar!/:4.3.17.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:312) ~[spring-beans-4.3.17.RELEASE.jar!/:4.3.17.RELEASE]
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230) ~[spring-beans-4.3.17.RELEASE.jar!/:4.3.17.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:308) ~[spring-beans-4.3.17.RELEASE.jar!/:4.3.17.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197) ~[spring-beans-4.3.17.RELEASE.jar!/:4.3.17.RELEASE]
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:761) ~[spring-beans-4.3.17.RELEASE.jar!/:4.3.17.RELEASE]
        at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:867) ~[spring-context-4.3.17.RELEASE.jar!/:4.3.17.RELEASE]
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:543) ~[spring-context-4.3.17.RELEASE.jar!/:4.3.17.RELEASE]
        at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:693) ~[spring-boot-1.5.13.RELEASE.jar!/:1.5.13.RELEASE]
        at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:360) ~[spring-boot-1.5.13.RELEASE.jar!/:1.5.13.RELEASE]
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:303) ~[spring-boot-1.5.13.RELEASE.jar!/:1.5.13.RELEASE]
        at com.emc.vcp.Application.main(Application.java:51) ~[classes!/:19.7.0.90]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_301]
        at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[?:1.8.0_301]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:1.8.0_301]
        at java.lang.reflect.Method.invoke(Unknown Source) ~[?:1.8.0_301]
        at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48) ~[vcp-server.jar:19.7.0.90]
        at org.springframework.boot.loader.Launcher.launch(Launcher.java:87) ~[vcp-server.jar:19.7.0.90]
        at org.springframework.boot.loader.Launcher.launch(Launcher.java:50) ~[vcp-server.jar:19.7.0.90]
        at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:51) ~[vcp-server.jar:19.7.0.90]
Caused by: org.springframework.amqp.AmqpIOException: javax.net.ssl.SSLException: readHandshakeRecord
        at org.springframework.amqp.rabbit.support.RabbitExceptionTranslator.convertRabbitAccessException(RabbitExceptionTranslator.java:71) ~[spring-rabbit-1.7.7.RELEASE.jar!/:?]
        at org.springframework.amqp.rabbit.connection.AbstractConnectionFactory.createBareConnection(AbstractConnectionFactory.java:376) ~[spring-rabbit-1.7.7.RELEASE.jar!/:?]
.
.
.
.
Caused by: java.net.SocketException: Connection reset
        at java.net.SocketOutputStream.socketWrite(Unknown Source) ~[?:1.8.0_301]
        at java.net.SocketOutputStream.write(Unknown Source) ~[?:1.8.0_301]
        at sun.security.ssl.SSLSocketOutputRecord.flush(Unknown Source) ~[?:1.8.0_301]
.
.
.
.
.       at sun.security.ssl.HandshakeOutStream.flush(Unknown Source) ~[?:1.8.0_301]
        at sun.security.ssl.ECDHClientKeyExchange$ECDHEClientKeyExchangeProducer.produce(Unknown Source) ~[?:1.8.0_301]
        at sun.security.ssl.ClientKeyExchange$ClientKeyExchangeProducer.produce(Unknown Source) ~[?:1.8.0_301]
        at sun.security.ssl.SSLHandshake.produce(Unknown Source) ~[?:1.8.0_301]
        at sun.security.ssl.ServerHelloDone$ServerHelloDoneConsumer.consume(Unknown Source) ~[?:1.8.0_301]
        at sun.security.ssl.SSLHandshake.consume(Unknown Source) ~[?:1.8.0_301]
        at sun.security.ssl.HandshakeContext.dispatch(Unknown Source) ~[?:1.8.0_301]
        at sun.security.ssl.HandshakeContext.dispatch(Unknown Source) ~[?:1.8.0_301]
        at sun.security.ssl.TransportContext.dispatch(Unknown Source) ~[?:1.8.0_301]
        at sun.security.ssl.SSLTransport.decode(Unknown Source) ~[?:1.8.0_301]
        at sun.security.ssl.SSLSocketImpl.decode(Unknown Source) ~[?:1.8.0_301]
        at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(Unknown Source) ~[?:1.8.0_301]
        at sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) ~[?:1.8.0_301]
        at sun.security.ssl.SSLSocketImpl.ensureNegotiated(Unknown Source) ~[?:1.8.0_301]
        at sun.security.ssl.SSLSocketImpl.access$200(Unknown Source) ~[?:1.8.0_301]
        at sun.security.ssl.SSLSocketImpl$AppOutputStream.write(Unknown Source) ~[?:1.8.0_301]
        at java.io.BufferedOutputStream.flushBuffer(Unknown Source) ~[?:1.8.0_301]
        at java.io.BufferedOutputStream.flush(Unknown Source) ~[?:1.8.0_301]
        at java.io.DataOutputStream.flush(Unknown Source) ~[?:1.8.0_301]
        at com.rabbitmq.client.impl.SocketFrameHandler.sendHeader(SocketFrameHandler.java:147) ~[amqp-client-4.0.3.jar!/:4.0.3]
.
.
.
.
2023-01-25 09:58:11,558 [main] INFO  (VcdClient.java:282) - VCP-Service Version: 19.7.0.90
2023-01-25 09:58:11,632 [main] WARN  (AbstractApplicationContext.java:551) - Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'amqpConfiguration': Invocation of init method failed; nested exception is org.springframework.amqp.AmqpIOException: javax.net.ssl.SSLException: readHandshakeRecord
2023-01-25 09:58:11,633 [main] INFO  (VcdClient.java:287) - Destroying VcdClient...
2023-01-25 09:58:11,644 [main] ERROR (SpringApplication.java:771) - Application startup failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'amqpConfiguration': Invocation of init method failed; nested exception is org.springframework.amqp.AmqpIOException: javax.net.ssl.SSLException: readHandshakeRecord
        at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:137) ~[spring-beans-4.3.17.RELEASE.jar!/:4.3.17.RELEASE]

VCP RabbitMQ TLS 交握正在嘗試進行相互驗證，而在 vcpsrv 傳送 DPE 憑證後，RabbitMQ 會重設其交握。 

需要告知外部 RabbitMQ 伺服器信任用於簽署 vcpsrv （和 vcpbg） 憑證的 DPE 根憑證，以便對 RabbitMQ 流量進行相互驗證。

請遵循以下步驟，以達成此：

1- 使用 Keytool 擷取 DPE root CA （和 dpem 憑證）。

在 DPE 節點上： 

keytool -printcert -sslserver localhost:9000 -rfc | sed -n '/-----BEGIN CERTIFICATE-----/,/-----END CERTIFICATE-----/p' | awk '/-----BEGIN CERTIFICATE-----/{i++}{print > "cert"i".pem"}'

這應該會建立 cert1.pem 和 cert2.pem 檔案。
cert2.pem 應為 DPE root ca。

2- 將步驟 A 中的另一個 CAfile 新增至第 1 個 rabbitmq 節點上的客戶 rabbitmq.config。

在 Rabbitmq 節點上：
在 /etc/rabbitmq/ssl/cacert.pem 編輯檔案，並在檔案結尾新增 /path/to/cert2.pem 的內容：

{cacertfile, "/etc/rabbitmq/ssl/cacert.pem"}

變更完成後，請重新開機 RabbitMQ，讓變更生效。

3- 針對其他 rabbitmq 節點重複步驟 2。