Avamar and Data Domain Integration: Unable to Sync Certificates with Session Security Enabled

Summary: When Session Security is enabled on Avamar, certificates must sync between Avamar and Data Domain. This requires the SCP protocol to be enabled on Data Domain.

This article applies to This article does not apply to This article is not tied to any specific product. Not all product versions are identified in this article.
Check out other resources

Symptoms

Backups may fail with the following errors: 
DDR result code: 5049, desc: file not found
DDR result code: 5341, desc: SSL library error "failed to import host or ca certificate automatically"
DDR result code: 5008, desc: invalid argument
When following Dell article 197106, Avamar and Data Domain Integration: DD Showing Red in Avamar AUI and or user interface Resolution Path, the certificates are not generated.

Upon checking the MCS logs, there is an exception related to the SCP protocol. 
09/29-16:29:13.00727 [RMI TCP Connection(27)-192.x.x.x#965] com.avamar.mc.datadomain.DdrCache.firsttimeToAddEx
FINE: Importing host certificate and ca certificates...
09/29-16:29:13.00743 [RMI TCP Connection(27)-192.x.x.x#965] com.avamar.mc.datadomain.DdrSsh.executeDdrCommand
FINE: Executing ddr command. host: idpa-lab.dell.com cmd: adminaccess certificate cert-signing-request show ...
09/29-16:29:14.00095 [RMI TCP Connection(27)-192.x.x.x#965] com.avamar.mc.kc.PrefsCertRsa.<init>
FINE: RSA certificate:
09/29-16:29:14.00095 [RMI TCP Connection(27)-192.x.x.x#965] com.avamar.mc.kc.PrefsCertRsa.<init>
FINE: Message digest algorithm: sha512
09/29-16:29:14.00095 [RMI TCP Connection(27)-192.x.x.x#965] com.avamar.mc.datadomain.PrefsDdrCert.<init>
INFO: DD RSA certificate:
09/29-16:29:14.00095 [RMI TCP Connection(27)-192.x.x.x#965] com.avamar.mc.datadomain.PrefsDdrCert.<init>
INFO: Number bits(key strength): 3072bit
09/29-16:29:14.00095 [RMI TCP Connection(27)-192.x.x.x#965] com.avamar.mc.datadomain.PrefsDdrCert.<init>
INFO: Message digest algorithm: sha512
09/29-16:29:14.00137 [RMI TCP Connection(27)-192.x.x.x#965] com.avamar.mc.datadomain.DdrSsh.executeDdrCommand
FINE: Executing ddr command. host: idpa-lab.dell.com cmd: adminaccess certificate cert-signing-request generate  key-strength 3072bit country 'US' state 'California' city 'Irvine' org-name 'EMC Corp' org-unit 'BRS Division'...
09/29-16:29:14.00721 [RMI TCP Connection(27)-192.x.x.x#965] com.avamar.mc.datadomain.DdrSsh.copyFile
FINE: Copying file from host: idpa-lab.dell.com...
09/29-16:29:15.00619 [RMI TCP Connection(27)-192.x.x.x#965] com.avamar.mc.datadomain.DdrSsh.copyFile
WARNING: Failed to copy file from host: idpa-lab.dell.com.
09/29-16:29:15.00619 [RMI TCP Connection(27)-192.x.x.x#965] com.avamar.mc.util.MCException.logException
WARNING: com.maverick.ssh.SshException: java.io.IOException
  at com.maverick.scp.ScpClientIO.get(ScpClientIO.java:151)
  at com.maverick.scp.ScpClientIO.get(ScpClientIO.java:124)
  at com.avamar.mc.datadomain.DdrSsh.copyFile(DdrSsh.java:940)
  at com.avamar.mc.datadomain.DdrSsh.copyFileEx(DdrSsh.java:961)
  at com.avamar.mc.datadomain.DdrSshCertificateCmd.getcertificateSigningRequest(DdrSshCertificateCmd.java:200)
  at com.avamar.mc.datadomain.DataDomainService.generateAndImportDdrHostCert(DataDomainService.java:5520)
  at com.avamar.mc.datadomain.DataDomainService.firsttimeToAdd(DataDomainService.java:5183)
  at com.avamar.mc.datadomain.DataDomainService.firsttimeToAdd(DataDomainService.java:6041)
  at com.avamar.mc.datadomain.DdrCache.firsttimeToAdd(DdrCache.java:1599)
  at com.avamar.mc.datadomain.DdrCache.firsttimeToAddEx(DdrCache.java:1645)
  at com.avamar.mc.datadomain.DdrCache.ConfigCerts(DdrCache.java:1454)
  at com.avamar.mc.datadomain.DdrCache.checkAndConfigCerts(DdrCache.java:1251)
  at com.avamar.mc.datadomain.DdrCache.update(DdrCache.java:402)
  at com.avamar.mc.datadomain.DdrCache.update(DdrCache.java:676)
  at com.avamar.mc.datadomain.DataDomainService.rewriteDdrCloudInfo(DataDomainService.java:6457)
  at com.avamar.mc.datadomain.DataDomainService.disableCloudTier(DataDomainService.java:6486)
  at com.avamar.mc.datadomain.DataDomainService._updateDdr(DataDomainService.java:1271)
  at com.avamar.mc.datadomain.DataDomainService.updateDdr(DataDomainService.java:1036)
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
  at java.lang.reflect.Method.invoke(Unknown Source)
  at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:318)
  at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
  at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:318)
  at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
  at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
  at org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor.invoke(MethodSecurityInterceptor.java:61)
  at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
  at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
  at com.sun.proxy.$Proxy37.updateDdr(Unknown Source)
  at com.avamar.mc.datadomain.DataDomainServiceContext.updateDdr(DataDomainServiceContext.java:223)
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
  at java.lang.reflect.Method.invoke(Unknown Source)
  at sun.rmi.server.UnicastServerRef.dispatch(Unknown Source)
  at sun.rmi.transport.Transport$1.run(Unknown Source)
  at sun.rmi.transport.Transport$1.run(Unknown Source)
  at java.security.AccessController.doPrivileged(Native Method)
  at sun.rmi.transport.Transport.serviceCall(Unknown Source)
  at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source)
  at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Source)
  at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(Unknown Source)
  at java.security.AccessController.doPrivileged(Native Method)
  at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Source)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
  at java.lang.Thread.run(Unknown Source)
Caused by: java.io.IOException: SCP unexpected cmd: Scp is disabled. Access denied.
  at com.maverick.scp.ScpClientIO$ScpEngineIO.readStreamFromRemote(ScpClientIO.java:305)
  at com.maverick.scp.ScpClientIO.get(ScpClientIO.java:148)
  at com.maverick.scp.ScpClientIO.get(ScpClientIO.java:124)
  at com.avamar.mc.datadomain.DdrSsh.copyFile(DdrSsh.java:940)
  at com.avamar.mc.datadomain.DdrSsh.copyFileEx(DdrSsh.java:961)
  at com.avamar.mc.datadomain.DdrSshCertificateCmd.getcertificateSigningRequest(DdrSshCertificateCmd.java:200)
  at com.avamar.mc.datadomain.DataDomainService.generateAndImportDdrHostCert(DataDomainService.java:5520)
  at com.avamar.mc.datadomain.DataDomainService.firsttimeToAdd(DataDomainService.java:5183)
  at com.avamar.mc.datadomain.DataDomainService.firsttimeToAdd(DataDomainService.java:6041)
  at com.avamar.mc.datadomain.DdrCache.firsttimeToAdd(DdrCache.java:1599)
  at com.avamar.mc.datadomain.DdrCache.firsttimeToAddEx(DdrCache.java:1645)
  at com.avamar.mc.datadomain.DdrCache.ConfigCerts(DdrCache.java:1454)
  at com.avamar.mc.datadomain.DdrCache.checkAndConfigCerts(DdrCache.java:1251)
  at com.avamar.mc.datadomain.DdrCache.update(DdrCache.java:402)
  at com.avamar.mc.datadomain.DdrCache.update(DdrCache.java:676)
  at com.avamar.mc.datadomain.DataDomainService.rewriteDdrCloudInfo(DataDomainService.java:6457)
  at com.avamar.mc.datadomain.DataDomainService.disableCloudTier(DataDomainService.java:6486)
  at com.avamar.mc.datadomain.DataDomainService._updateDdr(DataDomainService.java:1271)
  at com.avamar.mc.datadomain.DataDomainService.updateDdr(DataDomainService.java:1036)
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
  at java.lang.reflect.Method.invoke(Unknown Source)
  at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:318)
  at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
  at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
  at org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor.invoke(MethodSecurityInterceptor.java:61)
  at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
  at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
  at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
  at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
  at com.sun.proxy.$Proxy37.updateDdr(Unknown Source)
  at com.avamar.mc.datadomain.DataDomainServiceContext.updateDdr(DataDomainServiceContext.java:223)
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
  at java.lang.reflect.Method.invoke(Unknown Source)
  at sun.rmi.server.UnicastServerRef.dispatch(Unknown Source)
  at sun.rmi.transport.Transport$1.run(Unknown Source)
  at sun.rmi.transport.Transport$1.run(Unknown Source)
  at java.security.AccessController.doPrivileged(Native Method)
  at sun.rmi.transport.Transport.serviceCall(Unknown Source)
  at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source)
  at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Source)
  at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(Unknown Source)
  at java.security.AccessController.doPrivileged(Native Method)
  at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Source)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
  at java.lang.Thread.run(Unknown Source)

Cause

Review the mcserver log: 
/usr/local/avamar/var/mc/server_log/mcserver.log.0

Caused by: java.io.IOException: SCP unexpected cmd: Scp is disabled. Access denied.
This shows that the SCP protocol is disabled on Data Domain.

The flow of syncing certificates between Avamar and Data Domain requires SCP to be enabled as demonstrated below:
  1. Avamar runs a command on Data Domain using the Data Domain public key for password-less authentication. The first command is to generate a Certificate Signing Request (CSR) on Data Domain.
  2. The Avamar then attempts to copy the CSR from Data Domain using SCP, but fails to do so when SCP is disabled on Data Domain.
  3. Avamar would use the CSR to sign a certificate issued to Data Domain by the Avamar root certificate authority. On Data Domain it is called the 'imported-host ddboost" certificate.
Checking the Data Domain user interface shows that SCP is disabled:
 
Data Domain user interface shows that SCP is disabled
Figure 1: Data Domain user interface shows that SCP is disabled
 

Resolution

Enable SCP in the Data Domain web interface

From Administration > Access > Services > Check SCP > Configure > Check Allow SCP.
 
Enable SCP in the Data Domain web interface
Figure 2: Enable SCP in the Data Domain web interface
 

Affected Products

Avamar
Article Properties
Article Number: 000218137
Article Type: Solution
Last Modified: 23 Nov 2023
Version:  4
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.