PowerProtect: VM backups fail with ABV0006 "An internal error occurred"

Summary: The PowerProtect Data Manager appliance is used to protect the vSphere environment. The virtual machine (VM) backups fail with ABV0006 indicating an internal error occurred.

This article applies to This article does not apply to This article is not tied to any specific product. Not all product versions are identified in this article.
Check out other resources

Symptoms

The backup job shows:

 

Internal error screenshot

The backup export log and /var/log/brs/vmdm/vmdm.log reference an exception with the copy-catalog daemon:

YYYY-MM-DD HH:MM:SS INFO [] [https-jsse-nio-0.0.0.0-8447-exec-8] [00000000-0000-4000-a000-000000000000][][][TRACE_ID:a1e39e453b18b247;JOB_ID:93a1af84fd05235f][] [c.e.d.e.c.c.ProtectionCopySetServiceCcsAdapter.getByQuery(199)] -  Query PCSs by query Query[filter=(((((protectionLifeCycle.id eq "c9c86438-9264-4f26-83dc-d01b73a52e82" and topDataSourceId eq "4e3b763f-99e7-56db-9509-9e6d32a7829c") and replica ne true) and attributes.vmBackup.dataMoverType eq "VADP") and not (((state eq "DELETED" or state eq "SOFT_DELETED") or state eq "DELETING") or state eq "DELETE_FAILED")) and dataTargetIds eq "695d92ea-0c00-4a8f-9232-5e95fe4fcd7b") and backupLevel eq "FULL",genericPageRequest=PageRequest[pageNumber=1,pageSize=10,sort=time: DESC],nestedQueries={},projection=<null>,queryVerifier=<null>,refresh=true] 
YYYY-MM-DD HH:MM:SS WARN [] [https-jsse-nio-0.0.0.0-8447-exec-8] [00000000-0000-4000-a000-000000000000][][][TRACE_ID:a1e39e453b18b247;JOB_ID:93a1af84fd05235f][] [c.e.d.e.cs.sdk.FactoryBase.lambda$initialize$3(118)] -  2025-07-21T14:28:20.640847897-04:00[America/New_York]: Retry 'retry-for-io-exception', waiting PT4.421S until attempt '1'. Last attempt failed with exception 'com.emc.dpsg.ecdm.cs.sdk.exception.ClientExecutionTimeoutException: connection timed out after 60000 ms: copy-catalog/172.24.0.253:8458'. 
YYYY-MM-DD HH:MM:SS WARN [] [https-jsse-nio-0.0.0.0-8447-exec-8] [00000000-0000-4000-a000-000000000000][][][TRACE_ID:a1e39e453b18b247;JOB_ID:93a1af84fd05235f][] [c.e.b.v.v.a.VPEProtectOp.logErrorStack(64)] -  Unable to create backup
com.emc.dpsg.ecdm.baseresourceservice.exception.EcdmServiceException: An internal error occurred.
	at com.emc.dpsg.ecdm.commonresourceservice.common.CopyServiceExceptionMapper.lambda$bootstrapMappings$1(CopyServiceExceptionMapper.java:104)
	at com.emc.dpsg.ecdm.commonresourceservice.common.CopyServiceExceptionMapper.toEcdmServiceException(CopyServiceExceptionMapper.java:79)
	at com.emc.dpsg.ecdm.commonresourceservice.copyadapter.ProtectionCopySetServiceCcsAdapter.getByQuery(ProtectionCopySetServiceCcsAdapter.java:247)
	at com.emc.dpsg.ecdm.commonresourceservice.copyadapter.ProtectionCopySetServiceCcsAdapter.getByQuery(ProtectionCopySetServiceCcsAdapter.java:194)
	at jdk.internal.reflect.GeneratedMethodAccessor588.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:234)
	at jdk.proxy3/jdk.proxy3.$Proxy226.getByQuery(Unknown Source)
	at com.emc.dpsg.ecdm.commonresourceservice.sp.protectioncopyset.ProtectionCopySetServiceRouter.getByQuery(ProtectionCopySetServiceRouter.java:140)
	at com.emc.brs.vmdm.vpe.actions.VPEProtectOp.getBackupCopies(VPEProtectOp.java:1440)
	at com.emc.brs.vmdm.vpe.actions.VPEProtectOp.getProtectionCopySetResources(VPEProtectOp.java:1686)
	at com.emc.brs.vmdm.vpe.actions.VPEProtectOp.prepareConfig(VPEProtectOp.java:1598)
	at com.emc.brs.vmdm.vpe.actions.VPEProtectOp.execute(VPEProtectOp.java:678)
	at com.emc.brs.vmdm.vpe.actions.VPEProtectOp.initialize(VPEProtectOp.java:582)
	at com.emc.brs.vmdm.vpe.service.VPEActionDispatcher.submit(VPEActionDispatcher.java:283)
	at com.emc.brs.vmdm.vpe.service.VPEActionDispatcher.submitAsync(VPEActionDispatcher.java:316)
	at com.emc.brs.vmdm.controller.VmdmWorkflowController.executeRequest(VmdmWorkflowController.java:236)
	at com.emc.brs.vmdm.controller.VmdmWorkflowController.vmdProtect_aroundBody2(VmdmWorkflowController.java:141)
	at com.emc.brs.vmdm.controller.VmdmWorkflowController$AjcClosure3.run(VmdmWorkflowController.java:1)
	at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:179)
	at com.emc.brs.aaa.client.wizardry.PermissionAspect.ensurePermissions(PermissionAspect.java:67)
	at com.emc.brs.vmdm.controller.VmdmWorkflowController.vmdProtect(VmdmWorkflowController.java:141)
	at com.emc.brs.vmdm.controller.VmdmWorkflowController$$FastClassBySpringCGLIB$$5316496a.invoke(<generated>)
	at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:792)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762)
	at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:89)
	at com.emc.brs.feature.aspect.FeatureAspect.featureToggler(FeatureAspect.java:109)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:634)
	at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:624)
	at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:72)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762)
	at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:89)
	at com.emc.brs.common.idempotent.aspect.BaseIdempotentAspect.processNonReactiveIdempotentRequest(BaseIdempotentAspect.java:113)
	at com.emc.brs.common.idempotent.aspect.BaseIdempotentAspect.processIdempotentRequest(BaseIdempotentAspect.java:106)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:634)
	at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:624)
	at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:72)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762)
	at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762)
	at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:707)
	at com.emc.brs.vmdm.controller.VmdmWorkflowController$$EnhancerBySpringCGLIB$$2b773ac8.vmdProtect(<generated>)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
	at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:150)
	at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:117)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:903)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:809)
	at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
	at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1072)
	at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:965)
	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
	at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:665)
	at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:750)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:209)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153)
	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:51)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153)
	at com.emc.brs.common.rest.RestCORSFilter.doFilter(RestCORSFilter.java:46)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153)
	at com.emc.brs.feature.filter.BodyReaderFilter.doFilter(BodyReaderFilter.java:20)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:352)
	at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:117)
	at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:83)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361)
	at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:126)
	at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:120)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361)
	at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:131)
	at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:85)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361)
	at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:164)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361)
	at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361)
	at com.emc.brs.aaa.client.AccessTokenRedeemingRequestFilter.doFilterInternal(AccessTokenRedeemingRequestFilter.java:54)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361)
	at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:107)
	at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:93)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361)
	at org.springframework.security.web.header.HeaderWriterFilter.doHeadersAfter(HeaderWriterFilter.java:90)
	at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:75)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361)
	at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:117)
	at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361)
	at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:62)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361)
	at org.springframework.security.web.session.DisableEncodeUrlFilter.doFilterInternal(DisableEncodeUrlFilter.java:42)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361)
	at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:225)
	at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:190)
	at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:354)
	at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:267)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153)
	at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153)
	at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153)
	at com.atlassian.oai.validator.springmvc.OpenApiValidationFilter.doFilterInternal(OpenApiValidationFilter.java:61)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
	at com.dell.ppdm.rest.validation.webmvc.EnhancedOpenApiValidationFilter.doFilter(EnhancedOpenApiValidationFilter.java:36)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153)
	at com.emc.brs.vmdm.filter.RestLogFilter.doFilter(RestLogFilter.java:54)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153)
	at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153)
	at org.springframework.web.filter.ForwardedHeaderFilter.doFilterInternal(ForwardedHeaderFilter.java:156)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153)
	at com.emc.brs.tracing.api.CorrelationIDFilter.doFilter(CorrelationIDFilter.java:41)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:168)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90)
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:481)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:130)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
	at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:390)
	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63)
	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:928)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1794)
	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52)
	at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
	at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: com.emc.dpsg.ppdm.common.exception.PpdmException: null: copy-catalog/172.24.0.253:8458; nested exception is io.netty.channel.AbstractChannel$AnnotatedNoRouteToHostException: null: copy-catalog/172.24.0.253:8458
	at com.dell.ppdm.copycatalog.codegen.nonReactiveSdk.CopyMetadataWebClient.lambda$translateToPpdmException$0(CopyMetadataWebClient.java:197)
	at reactor.core.publisher.Mono.lambda$onErrorMap$31(Mono.java:3811)
	at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:94)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.onError(MonoFlatMap.java:172)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.onError(Operators.java:2065)
	at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.onError(FluxOnAssembly.java:544)
	at reactor.core.publisher.FluxMap$MapSubscriber.onError(FluxMap.java:134)
	at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:106)
	at reactor.core.publisher.Operators.error(Operators.java:198)
	at reactor.core.publisher.MonoErrorSupplied.subscribe(MonoErrorSupplied.java:56)
	at reactor.core.publisher.Mono.subscribe(Mono.java:4490)
	at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:103)
	at reactor.core.publisher.FluxPeek$PeekSubscriber.onError(FluxPeek.java:222)
	at reactor.core.publisher.FluxPeek$PeekSubscriber.onError(FluxPeek.java:222)
	at reactor.core.publisher.FluxPeek$PeekSubscriber.onError(FluxPeek.java:222)
	at reactor.core.publisher.MonoNext$NextSubscriber.onError(MonoNext.java:93)
	at reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain.onError(MonoFlatMapMany.java:204)
	at reactor.core.publisher.SerializedSubscriber.onError(SerializedSubscriber.java:124)
	at reactor.core.publisher.FluxRetryWhen$RetryWhenMainSubscriber.whenError(FluxRetryWhen.java:225)
	at reactor.core.publisher.FluxRetryWhen$RetryWhenOtherSubscriber.onError(FluxRetryWhen.java:274)
	at reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onError(FluxContextWrite.java:121)
	at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.drain(FluxConcatMap.java:415)
	at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.onNext(FluxConcatMap.java:251)
	at reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onNext(FluxContextWrite.java:107)
	at reactor.core.publisher.EmitterProcessor.drain(EmitterProcessor.java:537)
	at reactor.core.publisher.EmitterProcessor.tryEmitNext(EmitterProcessor.java:343)
	at reactor.core.publisher.SinkManySerialized.tryEmitNext(SinkManySerialized.java:100)
	at reactor.core.publisher.InternalManySink.emitNext(InternalManySink.java:27)
	at reactor.core.publisher.FluxRetryWhen$RetryWhenMainSubscriber.onError(FluxRetryWhen.java:190)
	at reactor.core.publisher.MonoCreate$DefaultMonoSink.error(MonoCreate.java:201)
	at reactor.netty.http.client.HttpClientConnect$MonoHttpConnect$ClientTransportSubscriber.onError(HttpClientConnect.java:311)
	at reactor.core.publisher.MonoCreate$DefaultMonoSink.error(MonoCreate.java:201)
	at reactor.netty.resources.DefaultPooledConnectionProvider$DisposableAcquire.onError(DefaultPooledConnectionProvider.java:160)
	at reactor.netty.internal.shaded.reactor.pool.AbstractPool$Borrower.fail(AbstractPool.java:475)
	at reactor.netty.internal.shaded.reactor.pool.SimpleDequePool.lambda$drainLoop$9(SimpleDequePool.java:431)
	at reactor.core.publisher.FluxDoOnEach$DoOnEachSubscriber.onError(FluxDoOnEach.java:186)
	at reactor.core.publisher.MonoCreate$DefaultMonoSink.error(MonoCreate.java:201)
	at reactor.netty.resources.DefaultPooledConnectionProvider$PooledConnectionAllocator$PooledConnectionInitializer.onError(DefaultPooledConnectionProvider.java:558)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.secondError(MonoFlatMap.java:192)
	at reactor.core.publisher.MonoFlatMap$FlatMapInner.onError(MonoFlatMap.java:259)
	at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:106)
	at org.springframework.security.config.annotation.web.configuration.SecurityReactorContextConfiguration$SecurityReactorContextSubscriber.onError(SecurityReactorContextConfiguration.java:192)
	at reactor.core.publisher.Operators.error(Operators.java:198)
	at reactor.core.publisher.MonoError.subscribe(MonoError.java:53)
	at reactor.core.publisher.Mono.subscribe(Mono.java:4490)
	at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:103)
	at reactor.netty.transport.TransportConnector$MonoChannelPromise.tryFailure(TransportConnector.java:569)
	at reactor.netty.transport.TransportConnector$MonoChannelPromise.setFailure(TransportConnector.java:515)
	at reactor.netty.transport.TransportConnector.lambda$doConnect$7(TransportConnector.java:261)
	at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590)
	at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583)
	at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559)
	at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492)
	at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636)
	at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:629)
	at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:118)
	at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:679)
	at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:698)
	at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567)
	at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:499)
	at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:407)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: org.springframework.web.reactive.function.client.WebClientRequestException: null: copy-catalog/172.24.0.253:8458; nested exception is io.netty.channel.AbstractChannel$AnnotatedNoRouteToHostException: null: copy-catalog/172.24.0.253:8458
	at org.springframework.web.reactive.function.client.ExchangeFunctions$DefaultExchangeFunction.lambda$wrapException$9(ExchangeFunctions.java:141)
Caused by: io.netty.channel.AbstractChannel$AnnotatedNoRouteToHostException: null: copy-catalog/172.24.0.253:8458
Caused by: java.net.NoRouteToHostException: null
	at io.netty.channel.unix.Errors.newConnectException0(Errors.java:158)
	at io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:131)
	at io.netty.channel.unix.Socket.finishConnect(Socket.java:359)
	at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:715)
	at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:692)
	at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567)
	at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:499)
 
YYYY-MM-DD HH:MM:SS WARN [] [https-jsse-nio-0.0.0.0-8447-exec-8] [00000000-0000-4000-a000-000000000000][][][TRACE_ID:a1e39e453b18b247;JOB_ID:93a1af84fd05235f][] [c.e.b.v.c.a.ActivityBuilder.complete(595)] -  Setting activity start time to now. 
YYYY-MM-DD HH:MM:SS INFO [] [https-jsse-nio-0.0.0.0-8447-exec-8] [00000000-0000-4000-a000-000000000000][][][TRACE_ID:a1e39e453b18b247;JOB_ID:93a1af84fd05235f][] [c.e.b.v.c.a.ActivityBuilder.complete(615)] -  Completed activity id=f9985066-458e-4856-b30d-1f1cebe05074, status=FAILED 
YYYY-MM-DD HH:MM:SS INFO [] [https-jsse-nio-0.0.0.0-8447-exec-8] [00000000-0000-4000-a000-000000000000][][][TRACE_ID:a1e39e453b18b247;JOB_ID:93a1af84fd05235f][] [c.e.b.v.v.a.VPEProtectOp.failActivity(1258)] -  EEMI ABV0006, Keys = [vproxy, vmName, vCenterName]

The System Services Status shows the Compliance Verification and Copy Catalog with Status "Down."

System Services Status

Starting the compliance-verification daemon fails with:

admin@coappdm:~> compliance-verification start
Starting...
Deleted Containers:
f4daad2507b98887f5dd871cd5323f809aadc7976d8b46efd0201259ac46f6f1

Total reclaimed space: 0B
[+] Running 1/0
 ✔ Container compliance-verification  Created0.1s
.Started [83618]
admin@coappdm:~> compliance-verification  | ERROR StatusLogger Log4j2 could not find a logging implementation. Please add log4j-core to the classpath. Using SimpleLogger to                   log to the console...
compliance-verification  | Logging system failed to initialize using configuration from '/compliance-verification/config/logback.xml'
compliance-verification  | java.lang.IllegalStateException: Logback configuration error detected:
compliance-verification  | ERROR in ch.qos.logback.core.joran.action.AppenderRefAction - Could not find an appender named [COPY_CATALOG]. Did you define it below instead of                   above in the configuration file?
compliance-verification  | ERROR in ch.qos.logback.core.joran.action.AppenderRefAction - See http://logback.qos.ch/codes.html#appender_order for more details.
compliance-verification  |      at org.springframework.boot.logging.logback.LogbackLoggingSystem.reportConfigurationErrorsIfNecessary(LogbackLoggingSystem.java:189)
compliance-verification  |      at org.springframework.boot.logging.logback.LogbackLoggingSystem.loadConfiguration(LogbackLoggingSystem.java:170)
compliance-verification  |      at org.springframework.boot.logging.AbstractLoggingSystem.initializeWithSpecificConfig(AbstractLoggingSystem.java:66)
compliance-verification  |      at org.springframework.boot.logging.AbstractLoggingSystem.initialize(AbstractLoggingSystem.java:57)
compliance-verification  |      at org.springframework.boot.logging.logback.LogbackLoggingSystem.initialize(LogbackLoggingSystem.java:132)
compliance-verification  |      at org.springframework.boot.context.logging.LoggingApplicationListener.initializeSystem(LoggingApplicationListener.java:332)
compliance-verification  |      at org.springframework.boot.context.logging.LoggingApplicationListener.initialize(LoggingApplicationListener.java:298)
compliance-verification  |      at org.springframework.boot.context.logging.LoggingApplicationListener.onApplicationEnvironmentPreparedEvent(LoggingApplicationListener.java                  :246)
compliance-verification  |      at org.springframework.boot.context.logging.LoggingApplicationListener.onApplicationEvent(LoggingApplicationListener.java:223)
compliance-verification  |      at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:178)
compliance-verification  |      at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:171)
compliance-verification  |      at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:145)
compliance-verification  |      at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:133)
compliance-verification  |      at org.springframework.boot.context.event.EventPublishingRunListener.environmentPrepared(EventPublishingRunListener.java:85)
compliance-verification  |      at org.springframework.boot.SpringApplicationRunListeners.lambda$environmentPrepared$2(SpringApplicationRunListeners.java:66)
compliance-verification  |      at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
compliance-verification  |      at org.springframework.boot.SpringApplicationRunListeners.doWithListeners(SpringApplicationRunListeners.java:120)
compliance-verification  |      at org.springframework.boot.SpringApplicationRunListeners.doWithListeners(SpringApplicationRunListeners.java:114)
compliance-verification  |      at org.springframework.boot.SpringApplicationRunListeners.environmentPrepared(SpringApplicationRunListeners.java:65)
compliance-verification  |      at org.springframework.boot.SpringApplication.prepareEnvironment(SpringApplication.java:344)
compliance-verification  |      at org.springframework.boot.SpringApplication.run(SpringApplication.java:302)
compliance-verification  |      at org.springframework.boot.SpringApplication.run(SpringApplication.java:1300)
compliance-verification  |      at org.springframework.boot.SpringApplication.run(SpringApplication.java:1289)
compliance-verification  |      at com.dell.ppdm.complianceverification.Main.main(Main.java:77)
compliance-verification  |      at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
compliance-verification  |      at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
compliance-verification  |      at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
compliance-verification  |      at java.base/java.lang.reflect.Method.invoke(Method.java:568)
compliance-verification  |      at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49)
compliance-verification  |      at org.springframework.boot.loader.Launcher.launch(Launcher.java:107)
compliance-verification  |      at org.springframework.boot.loader.Launcher.launch(Launcher.java:58)
compliance-verification  |      at org.springframework.boot.loader.thin.ThinJarLauncher.launch(ThinJarLauncher.java:195)
compliance-verification  |      at org.springframework.boot.loader.thin.ThinJarLauncher.main(ThinJarLauncher.java:141)
compliance-verification  |      at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
compliance-verification  |      at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
compliance-verification  |      at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
compliance-verification  |      at java.base/java.lang.reflect.Method.invoke(Method.java:568)
compliance-verification  |      at org.springframework.boot.loader.wrapper.ThinJarWrapper.launch(ThinJarWrapper.java:139)
compliance-verification  |      at org.springframework.boot.loader.wrapper.ThinJarWrapper.main(ThinJarWrapper.java:107)
compliance-verification  |      at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
compliance-verification  |      at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
compliance-verification  |      at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
compliance-verification  |      at java.base/java.lang.reflect.Method.invoke(Method.java:568)
compliance-verification  |      at com.emc.brs.ThinJarLauncher.directlyRun(ThinJarLauncher.java:33)
compliance-verification  |      at com.emc.brs.ThinJarLauncher.main(ThinJarLauncher.java:19)
compliance-verification  | ERROR SpringApplication Application run failed
compliance-verification  |  java.lang.IllegalStateException: java.lang.IllegalStateException: Logback configuration error detected:
compliance-verification  | ERROR in ch.qos.logback.core.joran.action.AppenderRefAction - Could not find an appender named [COPY_CATALOG]. Did you define it below instead of                   above in the configuration file?
compliance-verification  | ERROR in ch.qos.logback.core.joran.action.AppenderRefAction - See http://logback.qos.ch/codes.html#appender_order for more details.
compliance-verification  |      at org.springframework.boot.context.logging.LoggingApplicationListener.initializeSystem(LoggingApplicationListener.java:344)
compliance-verification  |      at org.springframework.boot.context.logging.LoggingApplicationListener.initialize(LoggingApplicationListener.java:298)
compliance-verification  |      at org.springframework.boot.context.logging.LoggingApplicationListener.onApplicationEnvironmentPreparedEvent(LoggingApplicationListener.java                  :246)
compliance-verification  |      at org.springframework.boot.context.logging.LoggingApplicationListener.onApplicationEvent(LoggingApplicationListener.java:223)
compliance-verification  |      at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:178)
compliance-verification  |      at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:171)
compliance-verification  |      at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:145)
compliance-verification  |      at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:133)
compliance-verification  |      at org.springframework.boot.context.event.EventPublishingRunListener.environmentPrepared(EventPublishingRunListener.java:85)
compliance-verification  |      at org.springframework.boot.SpringApplicationRunListeners.lambda$environmentPrepared$2(SpringApplicationRunListeners.java:66)
compliance-verification  |      at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
compliance-verification  |      at org.springframework.boot.SpringApplicationRunListeners.doWithListeners(SpringApplicationRunListeners.java:120)
compliance-verification  |      at org.springframework.boot.SpringApplicationRunListeners.doWithListeners(SpringApplicationRunListeners.java:114)
compliance-verification  |      at org.springframework.boot.SpringApplicationRunListeners.environmentPrepared(SpringApplicationRunListeners.java:65)
compliance-verification  |      at org.springframework.boot.SpringApplication.prepareEnvironment(SpringApplication.java:344)
compliance-verification  |      at org.springframework.boot.SpringApplication.run(SpringApplication.java:302)
compliance-verification  |      at org.springframework.boot.SpringApplication.run(SpringApplication.java:1300)
compliance-verification  |      at org.springframework.boot.SpringApplication.run(SpringApplication.java:1289)
compliance-verification  |      at com.dell.ppdm.complianceverification.Main.main(Main.java:77)
compliance-verification  |      at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
compliance-verification  |      at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
compliance-verification  |      at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
compliance-verification  |      at java.base/java.lang.reflect.Method.invoke(Method.java:568)
compliance-verification  |      at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49)
compliance-verification  |      at org.springframework.boot.loader.Launcher.launch(Launcher.java:107)
compliance-verification  |      at org.springframework.boot.loader.Launcher.launch(Launcher.java:58)
compliance-verification  |      at org.springframework.boot.loader.thin.ThinJarLauncher.launch(ThinJarLauncher.java:195)
compliance-verification  |      at org.springframework.boot.loader.thin.ThinJarLauncher.main(ThinJarLauncher.java:141)
compliance-verification  |      at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
compliance-verification  |      at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
compliance-verification  |      at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
compliance-verification  |      at java.base/java.lang.reflect.Method.invoke(Method.java:568)
compliance-verification  |      at org.springframework.boot.loader.wrapper.ThinJarWrapper.launch(ThinJarWrapper.java:139)
compliance-verification  |      at org.springframework.boot.loader.wrapper.ThinJarWrapper.main(ThinJarWrapper.java:107)
compliance-verification  |      at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
compliance-verification  |      at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
compliance-verification  |      at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
compliance-verification  |      at java.base/java.lang.reflect.Method.invoke(Method.java:568)
compliance-verification  |      at com.emc.brs.ThinJarLauncher.directlyRun(ThinJarLauncher.java:33)
compliance-verification  |      at com.emc.brs.ThinJarLauncher.main(ThinJarLauncher.java:19)
compliance-verification  | Caused by: java.lang.IllegalStateException: Logback configuration error detected:
compliance-verification  | ERROR in ch.qos.logback.core.joran.action.AppenderRefAction - Could not find an appender named [COPY_CATALOG]. Did you define it below instead of                   above in the configuration file?
compliance-verification  | ERROR in ch.qos.logback.core.joran.action.AppenderRefAction - See http://logback.qos.ch/codes.html#appender_order for more details.
compliance-verification  |      at org.springframework.boot.logging.logback.LogbackLoggingSystem.reportConfigurationErrorsIfNecessary(LogbackLoggingSystem.java:189)
compliance-verification  |      at org.springframework.boot.logging.logback.LogbackLoggingSystem.loadConfiguration(LogbackLoggingSystem.java:170)
compliance-verification  |      at org.springframework.boot.logging.AbstractLoggingSystem.initializeWithSpecificConfig(AbstractLoggingSystem.java:66)
compliance-verification  |      at org.springframework.boot.logging.AbstractLoggingSystem.initialize(AbstractLoggingSystem.java:57)
compliance-verification  |      at org.springframework.boot.logging.logback.LogbackLoggingSystem.initialize(LogbackLoggingSystem.java:132)
compliance-verification  |      at org.springframework.boot.context.logging.LoggingApplicationListener.initializeSystem(LoggingApplicationListener.java:332)
compliance-verification  |      ... 39 more

Cause

There is a syntax issue in the /usr/local/brs/lib/compliance-verification/config/logback.xml configuration file. The compliance-verification service is unable to get the appender information for the COPY_CATALOG. The cause of the logback.xml file changes is not known.

In this situation, there was a typo in the appender class name as it was COPY_CACOPY_CATALOG instead of COPY_CATALOG:

admin@my-ppdm:/usr/local/brs/lib/compliance-verification/config> cat logback.xml | grep -i copy_catalog
  <appender class="ch.qos.logback.core.rolling.RollingFileAppender" name="COPY_CACOPY_CATALOG">
  <appender class="ch.qos.logback.core.rolling.RollingFileAppender" name="COPY_CATALOG_METRICS">
    <appender-ref ref="COPY_CATALOG"/>
    <appender-ref ref="COPY_CATALOG_METRICS"/>

Resolution

Compare the compliance-verification logback.xml file with a working version for any typos.

  • Make a copy of the logback.xml file: cp /usr/local/brs/lib/compliance-verification/config/logback.xml /usr/local/brs/lib/compliance-verification/config/logback.xml.backup
  • Edit the logback.xml file: vi /usr/local/brs/lib/compliance-verification/config/logback.xml
  • Start the compliance-verification daemon: compliance-verification start
  • Validate the Compliance Verification and Copy Catalog services show the status "UP" in the System Service Status.
Article Properties
Article Number: 000348277
Article Type: Solution
Last Modified: 29 Sept 2025
Version:  1
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.