Avamar : Unable to add remote Avamar server to Client Manager

Error

When trying to add remote Avamar server in Client Manager web application, the following error arises.
 

Alert: Server authentication failed.

Logs

The following exception can be captured in the /usr/local/avamar-tomcat/logs/ACMTrace.log
 

2023-01-17 17:09:24,819 ERROR [ajp-nio-0:0:0:0:0:0:0:1-8509-exec-2]-sdk.ServiceConnection: Unable to connect to web service for grid remote_avamar.dell.com
org.apache.axis2.AxisFault: Socket is closed
        at org.apache.axis2.AxisFault.makeFault(AxisFault.java:430)
        at org.apache.axis2.transport.http.SOAPMessageFormatter.writeTo(SOAPMessageFormatter.java:83)
        at org.apache.axis2.transport.http.AxisRequestEntity.writeRequest(AxisRequestEntity.java:84)
        at org.apache.commons.httpclient.methods.EntityEnclosingMethod.writeRequestBody(EntityEnclosingMethod.java:499)
        at org.apache.commons.httpclient.HttpMethodBase.writeRequest(HttpMethodBase.java:2114)
        at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1096)
        at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398)
        at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
        at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398)
        at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
        at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
        at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:346)
        at org.apache.axis2.transport.http.AbstractHTTPSender.executeMethod(AbstractHTTPSender.java:557)
        at org.apache.axis2.transport.http.HTTPSender.sendViaPost(HTTPSender.java:199)
        at org.apache.axis2.transport.http.HTTPSender.send(HTTPSender.java:76)
        at org.apache.axis2.transport.http.CommonsHTTPTransportSender.writeMessageWithCommons(CommonsHTTPTransportSender.java:400)
        at org.apache.axis2.transport.http.CommonsHTTPTransportSender.invoke(CommonsHTTPTransportSender.java:225)
        at org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:438)
        at org.apache.axis2.description.OutInAxisOperationClient.send(OutInAxisOperation.java:402)
        at org.apache.axis2.description.OutInAxisOperationClient.executeImpl(OutInAxisOperation.java:229)
        at org.apache.axis2.client.OperationClient.execute(OperationClient.java:165)
        at com.avamar.mc.sdk.client.McServiceStub.login(McServiceStub.java:6088)
        at com.avamar.mc.sdk.client.McBaseServiceClient.login(McBaseServiceClient.java:120)
        at com.avamar.mc.sdk.client.McBaseServiceClient.connect(McBaseServiceClient.java:93)
        at com.emc.avamar.acm.core.mc.sdk.ServiceConnection.connect(ServiceConnection.java:67)
        at com.emc.avamar.acm.core.impl.WebServiceProxy.<init>(WebServiceProxy.java:152)
        at com.emc.avamar.acm.core.mapper.AvamarGridMapper.create(AvamarGridMapper.java:99)
        at com.emc.avamar.acm.core.entities.impl.DomainObjectImpl.create(DomainObjectImpl.java:86)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)
        at com.emc.avamar.acm.core.entities.impl.DomainObjectImpl.invoke(DomainObjectImpl.java:50)
        at com.sun.proxy.$Proxy161.create(Unknown Source)
        at com.emc.avamar.acm.service.impl.GridServiceImpl.addGrid(GridServiceImpl.java:164)
        at com.emc.avamar.acm.action.common.AddGridAction.execute0(AddGridAction.java:68)
        at com.emc.avamar.acm.action.common.BaseServiceAction.execute(BaseServiceAction.java:56)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)
        at com.opensymphony.xwork.DefaultActionInvocation.invokeAction(DefaultActionInvocation.java:358)
        at com.opensymphony.xwork.DefaultActionInvocation.invokeActionOnly(DefaultActionInvocation.java:218)
        at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:192)
        at com.opensymphony.xwork.interceptor.DefaultWorkflowInterceptor.doIntercept(DefaultWorkflowInterceptor.java:175)
        at com.opensymphony.xwork.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:86)
        at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:190)
        at com.opensymphony.xwork.validator.ValidationInterceptor.doIntercept(ValidationInterceptor.java:115)
        at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:190)
        at com.opensymphony.xwork.validator.ValidationInterceptor.doIntercept(ValidationInterceptor.java:115)
        at com.opensymphony.xwork.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:86)
        at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:190)
        at com.opensymphony.xwork.interceptor.AroundInterceptor.intercept(AroundInterceptor.java:31)
        at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:190)
        at com.opensymphony.xwork.interceptor.AroundInterceptor.intercept(AroundInterceptor.java:31)
        at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:190)
        at com.opensymphony.xwork.interceptor.AroundInterceptor.intercept(AroundInterceptor.java:31)
        at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:190)
        at com.opensymphony.webwork.interceptor.FileUploadInterceptor.intercept(FileUploadInterceptor.java:174)
        at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:190)
        at com.opensymphony.xwork.interceptor.AroundInterceptor.intercept(AroundInterceptor.java:31)
        at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:190)
        at com.opensymphony.webwork.interceptor.debugging.DebuggingInterceptor.intercept(DebuggingInterceptor.java:169)
        at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:190)
        at com.opensymphony.xwork.interceptor.AroundInterceptor.intercept(AroundInterceptor.java:31)
        at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:190)
        at com.opensymphony.xwork.interceptor.I18nInterceptor.intercept(I18nInterceptor.java:151)
        at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:190)
        at com.opensymphony.xwork.interceptor.AroundInterceptor.intercept(AroundInterceptor.java:31)
        at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:190)
        at com.opensymphony.xwork.interceptor.AroundInterceptor.intercept(AroundInterceptor.java:31)
        at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:190)
        at com.opensymphony.xwork.interceptor.AroundInterceptor.intercept(AroundInterceptor.java:31)
        at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:190)
        at com.opensymphony.xwork.interceptor.ExceptionMappingInterceptor.intercept(ExceptionMappingInterceptor.java:186)
        at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:190)
        at com.opensymphony.xwork.DefaultActionProxy.execute(DefaultActionProxy.java:116)
        at com.opensymphony.webwork.dispatcher.DispatcherUtils.serviceAction(DispatcherUtils.java:273)
        at com.opensymphony.webwork.dispatcher.FilterDispatcher.doFilter(FilterDispatcher.java:202)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at com.emc.avamar.filters.SecurityFilter.doFilter(SecurityFilter.java:35)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at com.emc.avamar.filters.AuthenticationFilter.doFilter(AuthenticationFilter.java:74)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at com.emc.avamar.filters.RedirectSecurityFilter.doFilter(RedirectSecurityFilter.java:18)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at com.emc.avamar.filters.RedirectSecurityFilter.doFilter(RedirectSecurityFilter.java:18)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:197)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:662)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
        at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:698)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:367)
        at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:526)
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:882)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1647)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
        at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Unknown Source)
Caused by: com.ctc.wstx.exc.WstxIOException: Socket is closed
        at com.ctc.wstx.sw.BaseStreamWriter.finishDocument(BaseStreamWriter.java:1692)
        at com.ctc.wstx.sw.BaseStreamWriter.close(BaseStreamWriter.java:288)
        at org.apache.axiom.util.stax.wrapper.XMLStreamWriterWrapper.close(XMLStreamWriterWrapper.java:46)
        at org.apache.axiom.om.impl.MTOMXMLStreamWriter.close(MTOMXMLStreamWriter.java:188)
        at org.apache.axiom.om.impl.llom.OMSerializableImpl.serializeAndConsume(OMSerializableImpl.java:197)
        at org.apache.axis2.transport.http.SOAPMessageFormatter.writeTo(SOAPMessageFormatter.java:79)
        ... 101 more
Caused by: java.net.SocketException: Socket is closed
        at com.rsa.sslj.x.ap.l(Unknown Source)
        at com.rsa.sslj.x.ap.a(Unknown Source)
        at com.rsa.sslj.x.ap.a(Unknown Source)
        at com.rsa.sslj.x.am.write(Unknown Source)
        at java.io.BufferedOutputStream.flushBuffer(Unknown Source)
        at java.io.BufferedOutputStream.flush(Unknown Source)
        at org.apache.commons.httpclient.ChunkedOutputStream.flush(ChunkedOutputStream.java:191)
        at com.ctc.wstx.io.UTF8Writer.flush(UTF8Writer.java:99)
        at com.ctc.wstx.sw.BufferingXmlWriter.flush(BufferingXmlWriter.java:214)
        at com.ctc.wstx.sw.BufferingXmlWriter.close(BufferingXmlWriter.java:194)
        at com.ctc.wstx.sw.BaseStreamWriter.finishDocument(BaseStreamWriter.java:1690)
        ... 106 more

Tcpdump/Wireshark

Before clicking save in source Client Manager to attempt to add a server, take a tcpdump on source Avamar's primary interface only capturing packets with a source or destination IP of the remote Avamar we are trying to add.

Example:
tcpdump -i <primary_interface> host <remote_avamar> -w <capture_file_output>
tcpdump -i bond0 host remote_avamar.dell.com -w /home/admin/client_manager.pcapng

Run the tcpdump dump command to start capturing network traffic.
After starting the command, click save in the Client Manager page to try to add remote Avamar.
Once the authentication error happens, CTRL C the tcpdump to stop capturing packets.

Now view the tcpdump in a tool like Wireshark to investigate.

Example:
Connection flow explained:
- TCP 3 way handshake in first 3 packets to remote Avamar on port 9443 (MCSDK web service)
- 4th packet TLS handshake kicks off with Client Hello where the source Avamar sends its capabilities to the destination.
- 5th packet remote Avamar acknowledges packet 4
- 6th packet remote Avamar sends Server Hello with its certificate chain, the certificate chain is the mcssl alias from its /usr/local/avamar/lib/rmi_ssl_keystore
- 7th packet the source Avamar acknowledges packet 6
- 8th packet the source Avamar throws a TLS Fatal Alert 42 (Bad Certificate)



Must Confirm

Before continuing with this KB you must confirm a few things:

- Verify forward and reverse DNS lookup between source and remote Avamar servers
- Verify using correct credential for remote Avamar MCUser account
- Verify no issues on source Avamar server with local accounts by investigating /usr/local/avamar/var/mc/server_log/mcserver.out
- Verify port 9443 is open between source and remote Avamar, this is used for mcsdk

Source Avamar server is unable to validate remote Avamar server's mcssl certificate on port 9443.
During this connection, the source Avamar will attempt to validate the remote Avamar server's mcssl certificate using its own trust store /usr/local/avamar/lib/rmi_ssl_keystore.
If the remote Avamar's mcssl is not in the source Avamar's rmi keystore, the socket will be closed following TLS handshake failure (see tcpdump analysis in symptoms)

There are two scenarios:

1. Remote Avamar server being added has self-signed web certificates (mcssl is self-signed in rmi keystore)
2. Remote Avamar server being added has CA signed web certificates (mcssl is a full chained signed certificate)

How to tell if self-signed or CA signed mcssl?
- if you have installed signed certs in the AUI by replacing the private key in system tab then mcssl will be signed
- if not, the certs are likely self-signed

Command to view mcssl certificate in rmi keystore from an Avamar server:

keytool -list -v -keystore /usr/local/avamar/lib/rmi_ssl_keystore -storepass $(avlockbox.sh -r keystore_passphrase) -alias mcssl

Example of self-signed mcssl:
We can see below that there is only one certificate in the chain length, and that the owner and issuer are the same entity (Administrator).


Example of CA signed mcssl:
We can see below that mcssl has a chain length of 3.
The owner and issuer of each certificate in the chain links up to the trusted root ca (Dell Tech Root CA 2018)



Scenario 1

Steps:

1. SSH to the source Avamar server

2. Get the remote mcssl certificate using openssl

echo | openssl s_client -connect <remote_avamar_ip_addr>:9443 | openssl x509 > /home/admin/remote_mcssl.crt

3. Import the remote mcssl certificate to rmi keystore

keytool -import -file /home/admin/remote_mcssl.crt -keystore /usr/local/avamar/lib/rmi_ssl_keystore -storepass $(avlockbox.sh -r keystore_passphrase) -alias remotemcssl

4. Restart mcs as admin user

mcserver.sh --restart --force

5. Restart tomcat as root user

emwebapp.sh --restart

6. Start backup scheduler again if needed

dpnctl start sched

Scenario 2

Steps:

1. SSH to the source Avamar server

2. Get the CA certs in the chain for the remote mcssl certificate

- if you have access to grab the intermediate CA(s) and root CA from your CA provider then place the individual files on the Avamar server in order to import them to rmi keystore
- this KB covers how to use openssl to grab the full cert chain from remote Avamar mcssl
 

echo | openssl s_client -connect <remote_avamar_ip_addr>:9443 -showcerts

Example Output:

root@source_avamar:/home/admin/#: echo | openssl s_client -connect <remote_avamar_ip_addr>:9443 -showcerts
CONNECTED(00000003)
depth=2 C = US, ST = Texas, L = Round Rock, O = Dell Technologies, OU = Cybersecurity, CN = Dell Technologies Root Certificate Authority 2018
verify error:num=19:self signed certificate in certificate chain
---
Certificate chain
 0 s:/C=US/ST=Raleigh/L=North Carolina/O=Dell/OU=ISG Support/CN=remote_avamar.dell.com
   i:/C=US/ST=Texas/L=Round Rock/O=Dell Technologies/CN=Dell Technologies Issuing CA 101
-----BEGIN CERTIFICATE-----
MIIFhjCCBG6gAwIBAgITYQABIryZXAGxuzqHpgAAAAEivDANBgkqhkiG9w0BAQsF
ADB5MQswCQYDVQQGEwJVUzEOMAwGA1UECBMFVGV4YXMxEzARBgNVBAcTClJvdW5k
IFJvY2sxGjAYBgNVBAoTEURlbGwgVGVjaG5vbG9naWVzMSkwJwYDVQQDEyBEZWxs
IFRlY2hub2xvZ2llcyBJc3N1aW5nIENBIDEwMTAeFw0yMjEyMjkxNTUyMzRaFw0y
NDEyMjkxNjAyMzRaMIGNMQswCQYDVQQGEwJVUzEQMA4GA1UECBMHUmFsZWlnaDEX
MBUGA1UEBxMOTm9ydGggQ2Fyb2xpbmExDTALBgNVBAoTBERlbGwxFDASBgNVBAsT
C0lTRyBTdXBwb3J0MS4wLAYDVQQDEyVhdm1yLTQ0MDAtcnRwLmRwYWQuZ3NsYWJz
LmxhYi5lbWMuY29tMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAxwou
w833CQTH+BVANqyS4pPBjz6/0jU0vGHu4Hz5L9/hPen0Nk8H6hmc2h0nMCfs5Q27
9I1YdP0XyY+fNt2qmXTFQ+pQ5e23QKLnRRo3NLxlPt0J8L9ju8WtHIcgmPnxd7zp
/9h5CDkdhJ73d0+bkXU+nj9TbLtGrX1xbxgEfGuth4ofTZuYar9uyDMZSsTnklI9
8/sInptiHMS9+u55RDCRt68PfkLY0MJooPbTsH08HVcI6Y3OHxivdEHf2tGDcHLM
x5onF6pw6e0l7Z1b0OXf6wdwrhDUoinfIFNDMOWFU3K0gWqToTu+TtL90ZaZy0zW
SHVtP5yKDzBthjBpwwIDAQABo4IB8DCCAewwPwYDVR0RBDgwNoIlYXZtci00NDAw
LXJ0cC5kcGFkLmdzbGFicy5sYWIuZW1jLmNvbYINYXZtci00NDAwLXJ0cDAdBgNV
HQ4EFgQU7Gix38z9pC5bfb/kIZD/Dv1xMOAwHwYDVR0jBBgwFoAUUwwDUHFz+sTs
UVOMRSttymBWk7YwXQYDVR0fBFYwVDBSoFCgToZMaHR0cDovL2NybC5kZWxsLmNv
bS9pbnQvaXNzdWluZy9EZWxsJTIwVGVjaG5vbG9naWVzJTIwSXNzdWluZyUyMENB
JTIwMTAxLmNybDBoBggrBgEFBQcBAQRcMFowWAYIKwYBBQUHMAKGTGh0dHA6Ly9h
aWEuZGVsbC5jb20vaW50L2lzc3VpbmcvRGVsbCUyMFRlY2hub2xvZ2llcyUyMElz
c3VpbmclMjBDQSUyMDEwMS5jcnQwDAYDVR0TAQH/BAIwADALBgNVHQ8EBAMCBaAw
PQYJKwYBBAGCNxUHBDAwLgYmKwYBBAGCNxUIh7jZbIGE+AiGvY0PhfOANYKzmVoz
g9fvaoHFyVgCAWQCARwwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMCcG
CSsGAQQBgjcVCgQaMBgwCgYIKwYBBQUHAwEwCgYIKwYBBQUHAwIwDQYJKoZIhvcN
AQELBQADggEBAFW4PGDJT4KwLfEKVv1xRapUrFjIdWpZUjnuBEYk7RXlQ7o0OBM6
wKmwJHuiM5kLnlCMeQgPHyDtrzQVWMAwzw9RP0sFm+OdQklarVc5CqNoj0n+agN1
jbknUwyN2av2XBM7GF+CSXFDILteyVQmEsb3QD11khEU4pXGasIm/3R7Tm4uY7qq
dtARJeSr0m/uFeaT2AONBZn2uy/EIomzJanB3FI2sLu7UcaoQ1VqsH0c9PZbWAkv
tUYVBHxOD79PUiEQWgajHFPwanmUuzwlZpWxr09z0Yqjx623ZpdkMwP7jVV+tJXF
yfJoP0EuL3NvyROwvwu4CWhUIiJ26Q6+ACI=
-----END CERTIFICATE-----
 1 s:/C=US/ST=Texas/L=Round Rock/O=Dell Technologies/CN=Dell Technologies Issuing CA 101
   i:/C=US/ST=Texas/L=Round Rock/O=Dell Technologies/OU=Cybersecurity/CN=Dell Technologies Root Certificate Authority 2018
-----BEGIN CERTIFICATE-----
MIIFNjCCBB6gAwIBAgITEwAAAAJT0lQwmqXBWgAAAAAAAjANBgkqhkiG9w0BAQsF
ADCBojELMAkGA1UEBhMCVVMxDjAMBgNVBAgTBVRleGFzMRMwEQYDVQQHEwpSb3Vu
ZCBSb2NrMRowGAYDVQQKExFEZWxsIFRlY2hub2xvZ2llczEWMBQGA1UECxMNQ3li
ZXJzZWN1cml0eTE6MDgGA1UEAxMxRGVsbCBUZWNobm9sb2dpZXMgUm9vdCBDZXJ0
aWZpY2F0ZSBBdXRob3JpdHkgMjAxODAeFw0xODExMTQyMTI2MzVaFw0yNTExMTQy
MTM2MzVaMHkxCzAJBgNVBAYTAlVTMQ4wDAYDVQQIEwVUZXhhczETMBEGA1UEBxMK
Um91bmQgUm9jazEaMBgGA1UEChMRRGVsbCBUZWNobm9sb2dpZXMxKTAnBgNVBAMT
IERlbGwgVGVjaG5vbG9naWVzIElzc3VpbmcgQ0EgMTAxMIIBIjANBgkqhkiG9w0B
AQEFAAOCAQ8AMIIBCgKCAQEAs1ijUZedCGDMDz4vpkxEO2JRi7Q3f5WE8m8emZFX
0393V+ekbyJZnEB3RLLPJiIy+YDgctJMLU51GenChujMJklk7gk+SKwqL+2p30vJ
Dpwz7Lsl7Cjn3v1vQ5AxEHkkUKL0H6NsRZ7uJKrPsH1cVvQtcpgZCg4ghdGB92wE
AIlSVYRPN07T/QkmD3EFSEdNEISFNk4KljIlRmpxprEN7hKyUv002COvN5S6zXMs
9GFPdnbTEaiZpFPtRF+ZZn0ub0i16Us0IRE60xHEllWyKWJYNhiW+srztLR/eP6a
1J4EIImvNz32pxrKDbIeQZxQAH9FVQqJpssSgN8LJzNXnwIDAQABo4IBizCCAYcw
EAYJKwYBBAGCNxUBBAMCAQAwHQYDVR0OBBYEFFMMA1Bxc/rE7FFTjEUrbcpgVpO2
MBEGA1UdIAQKMAgwBgYEVR0gADAZBgkrBgEEAYI3FAIEDB4KAFMAdQBiAEMAQTAL
BgNVHQ8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zAfBgNVHSMEGDAWgBTbAOslX5H7
mca2SBzQY61f4qe0VjBtBgNVHR8EZjBkMGKgYKBehlxodHRwOi8vY3JsLmRlbGwu
Y29tL2ludC9yb290L0RlbGwlMjBUZWNobm9sb2dpZXMlMjBSb290JTIwQ2VydGlm
aWNhdGUlMjBBdXRob3JpdHklMjAyMDE4LmNybDB4BggrBgEFBQcBAQRsMGowaAYI
KwYBBQUHMAKGXGh0dHA6Ly9haWEuZGVsbC5jb20vaW50L3Jvb3QvRGVsbCUyMFRl
Y2hub2xvZ2llcyUyMFJvb3QlMjBDZXJ0aWZpY2F0ZSUyMEF1dGhvcml0eSUyMDIw
MTguY3J0MA0GCSqGSIb3DQEBCwUAA4IBAQCAJoPC4DTuxgC/wAHcHHvstGfiP/sB
/moVVdyOabZ7jSk+Pr2UjBPsALcfnwEsqxeL25J5Hn112baDTnVvrcymdqo9gD/5
tkrWUOWwrvzuwCuB082qHcbWdaOJtoLyuYdobPDktShKc43Oprv7G19TL3ccCmMJ
CPWGh4uv0D5vZnOh0T3wA591Gok6wJAKlXU3GCl5hiPII0j84DXE1kVDDSvDSF82
syl1wzha6gYe8s0S64NiB9eWofwJEHhZlam86+N6WQm1qBHdJ4kix5ndaa9sUJYG
ldHZZdACf5Cg9+/f/5sBiSpua0Id87TmKHu2Cm51LwAiLLbJYvexZbXz
-----END CERTIFICATE-----
 2 s:/C=US/ST=Texas/L=Round Rock/O=Dell Technologies/OU=Cybersecurity/CN=Dell Technologies Root Certificate Authority 2018
   i:/C=US/ST=Texas/L=Round Rock/O=Dell Technologies/OU=Cybersecurity/CN=Dell Technologies Root Certificate Authority 2018
-----BEGIN CERTIFICATE-----
MIIEITCCAwmgAwIBAgIQGRwh4YLNUKpNzFg7nT7QgjANBgkqhkiG9w0BAQsFADCB
ojELMAkGA1UEBhMCVVMxDjAMBgNVBAgTBVRleGFzMRMwEQYDVQQHEwpSb3VuZCBS
b2NrMRowGAYDVQQKExFEZWxsIFRlY2hub2xvZ2llczEWMBQGA1UECxMNQ3liZXJz
ZWN1cml0eTE6MDgGA1UEAxMxRGVsbCBUZWNobm9sb2dpZXMgUm9vdCBDZXJ0aWZp
Y2F0ZSBBdXRob3JpdHkgMjAxODAeFw0xODA3MjMxNzA3NDVaFw00MzA3MjMxNzE3
NDRaMIGiMQswCQYDVQQGEwJVUzEOMAwGA1UECBMFVGV4YXMxEzARBgNVBAcTClJv
dW5kIFJvY2sxGjAYBgNVBAoTEURlbGwgVGVjaG5vbG9naWVzMRYwFAYDVQQLEw1D
eWJlcnNlY3VyaXR5MTowOAYDVQQDEzFEZWxsIFRlY2hub2xvZ2llcyBSb290IENl
cnRpZmljYXRlIEF1dGhvcml0eSAyMDE4MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8A
MIIBCgKCAQEAqlQlbFbW7/Iy6wVtPEmfxbRMliJj4S/0vboVuZySAYI7TtMBAaDu
gBBeKPuKmRD1nUP+T4L0i5KBUFRWq5OvgtDYk8Ig382douSbWe4dwGOvcjNKvbHf
Iki9cVSvgTjaeOtORYXzMlCLw59JT6FtXFKjlW1GrM+TPbb2rjDLNir0IzDYvPUx
INXXnxyBSaBvkTipChTnIqXJJG53xEZIb/tK4NE3vcuP5RTFE0AEZmdY9Ka06x8Y
3X83ifbFZ2ZNTjRYggWdVpXB1sdfhjskdUDHunlvN9xBdur4QIUhfsefpe8sVE0K
RAv7IFBKwo5X+zFrfFM9KZGRGQsCBhKWgwIDAQABo1EwTzALBgNVHQ8EBAMCAYYw
DwYDVR0TAQH/BAUwAwEB/zAdBgNVHQ4EFgQU2wDrJV+R+5nGtkgc0GOtX+KntFYw
EAYJKwYBBAGCNxUBBAMCAQAwDQYJKoZIhvcNAQELBQADggEBAKZ5Su5LLbOXtito
8OFAkOs/AxuzRPP16E1ZO8VWN8Q4y0RsCbJ7V+R1QAlhd793X1LnEMHYpnXQLPbg
tWtWcPfoZbVK5zcDCIt4/7XgpCmL6yGGk9SVDkJrZIv+bvlB6P64vR/Fs2BYHaQx
G2ITb2GN4CrU+UxmieBPMvj12epD23R8HIcarKD/ZRCAR3XAd80v2A5mncxSqdHI
PzNT/fmlMgiesEl+IjJqc98HCtVAVa2hLsXkiZiLGKZPmS0cbQTsW/sNBE+s5WlG
Yo++cDfTgqdLiYcTXmx9PPyMwbmzkUr8Mr8dbDBodWSEpVgv6oCej2kGOF6saUAr
/4YKU4g=
-----END CERTIFICATE-----
---
Server certificate
subject=/C=US/ST=Raleigh/L=North Carolina/O=Dell/OU=ISG Support/CN=remote_avamar.dell.com
issuer=/C=US/ST=Texas/L=Round Rock/O=Dell Technologies/CN=Dell Technologies Issuing CA 101
---
No client certificate CA names sent
Peer signing digest: SHA256
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 4353 bytes and written 459 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES128-SHA256
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES128-SHA256
    Session-ID: 63D41D4F774A930BF695B94A53099134EF9DBC7B62B05595E1F82F4CC7CEA4AE
    Session-ID-ctx:
    Master-Key: 9CF0572465C0B5F72F2C4D19DF3EA5E5B5D27A37EA8A5C8EAD6656B7E362B80B1FF1F71E0BB07BE4CC9EAD393FCE2D30
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1674845514
    Timeout   : 300 (sec)
    Verify return code: 19 (self signed certificate in certificate chain)
---
DONE

From the output we see that there are 3 certificates in the chain.

In order:

Server Cert
0 s:/C=US/ST=Raleigh/L=North Carolina/O=Dell/OU=ISG Support/CN=remote_avamar.dell.com

Intermediate CA
1 s:/C=US/ST=Texas/L=Round Rock/O=Dell Technologies/CN=Dell Technologies Issuing CA 101

Root CA
2 s:/C=US/ST=Texas/L=Round Rock/O=Dell Technologies/OU=Cybersecurity/CN=Dell Technologies Root Certificate Authority 2018

The contents of a certificate starts with -----BEGIN CERTIFICATE----- and ends with -----END CERTIFICATE----- inclusive

Take the second certificate in the output (Intermediate CA at depth 1) and save its contents to a file like remote_mcssl_int.crt

Take the third certificate in the output (Root CA at depth 2) and save its contents to a file like remote_mcssl_root.crt

You would do this process for any length chain for signed mcssl, saving all the intermediate CAs and root CA to individual files.


3. Import each CA to the rmi keystore
 

keytool -import -file /home/admin/remote_mcssl_int.crt -keystore /usr/local/avamar/lib/rmi_ssl_keystore -storepass $(avlockbox.sh -r keystore_passphrase) -alias remoteMcsslInt

keytool -import -file /home/admin/remote_mcssl_root.crt -keystore /usr/local/avamar/lib/rmi_ssl_keystore -storepass $(avlockbox.sh -r keystore_passphrase) -alias remoteMcsslRoot

4. Restart mcs as admin user

mcserver.sh --restart --force

5. Restart tomcat as root user

emwebapp.sh --restart

6. Start backup scheduler again if needed

dpnctl start sched