PowerProtect: Replication fails with Data Domain Error code ARP0030

摘要: Replication is consistently failing with error code ARP0030 timeout issue between PowerProtect Data Manager and DR Data Domain. For instance: Replication job was unsuccessful for MICROSOFT_SQL_DATABASE. ...

本文适用于 本文不适用于 本文并非针对某种特定的产品。 本文并非包含所有产品版本。
查看其他资源

症状

In the following log:  
...\Failed_106StripMillData_PSSQLDB02.soo.abc.com_a6d0########9115\PowerProtect Data Manager_a6d0########9115:

We see: 
replmgr.log: YYYY-MM-DDTHH:MM:59.888Z INFO [] [replmgr-job-worker-thread-48306] [00000000-0000-4000-a000-000000000000][][TASK_ID:ae36########44e4][TRACE_ID:911d########e63b;JOB_ID:a6d0########9115][plc.id:d3fb6870,asset.id:58f0839f,pcs.id:] c.e.b.d.ddboost.DDBoostAdapter.connect(331) -  Connecting using storage unit: PPDM-PROD_VLT_MSSQL_PROCESS_PROD-ppdm-R-44576 
replmgr.log: YYYY-MM-DDTHH:MM:59.888Z ERROR [] [replmgr-job-worker-thread-48306] [00000000-0000-4000-a000-000000000000][][TASK_ID:ae36########44e4][TRACE_ID:911d########e63b;JOB_ID:a6d0########9115][plc.id:d3fb6870,asset.id:58f0839f,pcs.id:] c.e.b.d.c.d.c.DdboostConnectionManager.connect(219) -  Failed to connect to PowerProtect DD, rethrow exception as DdConnectionException. 
com.emc.brs.datadomain.ddboost.DdboostException: Code: 5002, Message: connect operation failed with code: 5002. errstr=value exceeds upper bound, errorinfo=[ 6348] [139996585236224] Thu Oct  5 14:03:59 2023
    ddp_connect_with_config() failed, Hostname: my_dr_datadomain.abc.com, Err: 5002-max allowed connections exceeded

########## ---------- ##########

./Failed_106StripMillData_PSSQLDB02.abc.com_a6d0########9115/PowerProtect Data Manager_a6d0########9115.log:     ddp_connect_with_config() failed, Hostname: my_prod_datadomain.abc.com, Err: 5002-max allowed connections exceeded
./Failed_106StripMillData_PSSQLDB02.abc.com_a6d0########9115/PowerProtect Data Manager_a6d0########9115.log:     ddp_connect_with_config() failed, Hostname: my_prod_datadomain.abc.com, Err: 5002-max allowed connections exceeded
./Failed_106StripMillData_PSSQLDB02.abc.com_a6d0########9115/PowerProtect Data Manager_a6d0########9115.log:     ddp_connect_with_config() failed, Hostname: my_dr_datadomain.abc.com, Err: 5002-max allowed connections exceeded
./Failed_Cogen_OPA_ProcessGenSQL01.abc.com_ab9dc39ded42567b/PowerProtect Data Manager_ab9dc39ded42567b.log:      ddp_connect_with_config() failed, Hostname: my_prod_datadomain.abc.com, Err: 5002-max allowed connections exceeded
./Failed_Cogen_OPA_ProcessGenSQL01.abc.com_ab9dc39ded42567b/PowerProtect Data Manager_ab9dc39ded42567b.log:      ddp_connect_with_config() failed, Hostname: my_prod_datadomain.abc.com, Err: 5002-max allowed connections exceeded
./Failed_Cogen_OPA_ProcessGenSQL01.abc.com_ab9dc39ded42567b/PowerProtect Data Manager_ab9dc39ded42567b.log:      ddp_connect_with_config() failed, Hostname: my_dr_datadomain.abc.com, Err: 5002-max allowed connections exceeded
./Failed_CraneDashboardDev_SSMXSQLSSD02.abc.com_8eff47d26bf2292f/PowerProtect Data Manager_8eff47d26bf2292f.log: ddp_connect_with_config() failed, Hostname: my_prod_datadomain.abc.com, Err: 5002-max allowed connections exceeded
./Failed_CraneDashboardDev_SSMXSQLSSD02.abc.com_8eff47d26bf2292f/PowerProtect Data Manager_8eff47d26bf2292f.log: ddp_connect_with_config() failed, Hostname: my_prod_datadomain.abc.com, Err: 5002-max allowed connections exceeded
./Failed_DSPC_Delays_DSPCDB01.abc.com_890b4ea199a2cbfe/PowerProtect Data Manager_890b4ea199a2cbfe.log:   ddp_connect_with_config() failed, Hostname: my_prod_datadomain.abc.com, Err: 5002-max allowed connections exceeded
./Failed_DSPC_Delays_DSPCDB01.abc.com_890b4ea199a2cbfe/PowerProtect Data Manager_890b4ea199a2cbfe.log:   ddp_connect_with_config() failed, Hostname: my_prod_datadomain.abc.com, Err: 5002-max allowed connections exceeded
./Failed_DWConfiguration_DSPCDB01.abc.com_998b66035ec5524c/PowerProtect Data Manager_998b66035ec5524c.log:       ddp_connect_with_config() failed, Hostname: my_dr_datadomain.abc.com, Err: 5002-max allowed connections exceeded
./Failed_DWConfiguration_DSPCDB01.abc.com_998b66035ec5524c/PowerProtect Data Manager_998b66035ec5524c.log:       ddp_connect_with_config() failed, Hostname: my_dr_datadomain.abc.com, Err: 5002-max allowed connections exceeded
./Failed_model_provpsdbprd.abc.com_8adc301f67163e72/PowerProtect Data Manager_8adc301f67163e72.log:      ddp_connect_with_config() failed, Hostname: my_prod_datadomain.abc.com, Err: 5002-max allowed connections exceeded
./Failed_model_provpsdbprd.abc.com_8adc301f67163e72/PowerProtect Data Manager_8adc301f67163e72.log:      ddp_connect_with_config() failed, Hostname: my_prod_datadomain.abc.com, Err: 5002-max allowed connections exceeded
./Failed_model_stldb-prod.abc.com_97a781c68e4a4476/PowerProtect Data Manager_97a781c68e4a4476.log:       ddp_connect_with_config() failed, Hostname: my_prod_datadomain.abc.com, Err: 5002-max allowed connections exceeded
./Failed_model_stldb-prod.abc.com_97a781c68e4a4476/PowerProtect Data Manager_97a781c68e4a4476.log:       ddp_connect_with_config() failed, Hostname: my_prod_datadomain.abc.com, Err: 5002-max allowed connections exceeded
./Failed_ReportServerTempDB_PSSQLDB05.abc.com_a2dd38f4a44dc164/PowerProtect Data Manager_a2dd38f4a44dc164.log:   ddp_connect_with_config() failed, Hostname: my_dr_datadomain.abc.com, Err: 5002-max allowed connections exceeded
./Failed_ReportServerTempDB_PSSQLDB05.abc.com_a2dd38f4a44dc164/PowerProtect Data Manager_a2dd38f4a44dc164.log:   ddp_connect_with_config() failed, Hostname: my_dr_datadomain.abc.com, Err: 5002-max allowed connections exceeded
./Failed_TES_stldb-prod.abc.com_8ce8c387e035c399/PowerProtect Data Manager_8ce8c387e035c399.log: ddp_connect_with_config() failed, Hostname: my_prod_datadomain.abc.com, Err: 5002-max allowed connections exceeded
./Failed_TES_stldb-prod.abc.com_8ce8c387e035c399/PowerProtect Data Manager_8ce8c387e035c399.log: ddp_connect_with_config() failed, Hostname: my_prod_datadomain.abc.com, Err: 5002-max allowed connections exceeded

From \var\log\brs\monitoring\monitoring.log 
YYYY-MM-DDTHH:MM:10.613Z INFO [] [pool-8-thread-1] [][][][TRACE_ID:86c6########bb9b][] [c.e.b.t.i.LogTracingServiceImpl.startTrace(57)] -  === Start a new Log Trace for Compute System Health Metrics === 
YYYY-MM-DDTHH:MM:16.027Z INFO [] [pool-5-thread-1] [][][][][] [c.e.b.m.e.s.ExternalNotificationServiceMonitoring.getExternalNotificationsToCloseWindow(256)] -  0 External Notifications for summary email. 
YYYY-MM-DDTHH:MM:21.915Z INFO [] [AsyncResolver-bootstrap-executor-0] [][][][][] [c.n.d.s.r.a.ConfigClusterResolver.getClusterEndpoints(43)] -  Resolving eureka endpoints via configuration 
YYYY-MM-DDTHH:MM:40.698Z INFO [] [pool-9-thread-1] [][][][TRACE_ID:a5cb15899916efb5][] [c.e.b.t.i.LogTracingServiceImpl.startTrace(57)] -  === Start a new Log Trace for Compute System Manager Health Checks === 
YYYY-MM-DDTHH:MM:41.331Z INFO [] [https-jsse-nio-8449-exec-10] [][][][][] [c.e.b.a.c.AccessTokenRedeemingRequestFilter.withAuthenticationFromRequest(71)] -  REDEEM: success 
YYYY-MM-DDTHH:MM:41.331Z INFO [] [https-jsse-nio-8449-exec-10] [][][][TRACE_ID:a5cb15899916efb5][] [c.e.b.m.f.RestLogFilter.doFilter(45)] -  Enter RESTful Msg - Method: GET , Path: https://localhost:8443/api/v2/report-nodes   
YYYY-MM-DDTHH:MM:41.475Z INFO [] [https-jsse-nio-8449-exec-10] [][][][][] [c.e.b.m.f.RestLogFilter.doFilter(56)] -  Exit RESTful Msg -  Status:  200 , ContentType: application/json  
YYYY-MM-DDTHH:MM:01.335Z INFO [] [https-jsse-nio-8449-exec-2] [][][][][] [c.e.b.a.c.AccessTokenRedeemingRequestFilter.withAuthenticationFromRequest(71)] -  REDEEM: success 
YYYY-MM-DDTHH:MM:01.336Z INFO [] [https-jsse-nio-8449-exec-2] [][][][TRACE_ID:a1f2ee14131a5ccd][] [c.e.b.m.f.RestLogFilter.doFilter(45)] -  Enter RESTful Msg - Method: GET , Path: https://myppdm.abc.com:8443/api/v2/activities  ?filter=category+in+%28%22PROTECT%22%2C%22RESTORE%22%2C%22SYSTEM%22%2C%22EXPORT_REUSE%22%2C%22REPLICATE%22%2C%22CONFIG%22%2C%22VALIDATE%22%2C%22CLOUD_TIER%22%2C%22CLOUD_DR%22%2C%22DISCOVER%22%2C%22NOTIFY%22%2C%22DELETE%22%2C%22MANAGE%22%2C%22DISASTER_RECOVERY%22%29+and+classType+in+%28%22JOB_GROUP%22%2C%22JOB%22%29+and+%28%28updateTime+gt+%222023-10-05T17%3A53%3A25.002Z%22+and+updateTime+le+%222023-10-05T17%3A58%3A30.002Z%22+and+state+ne+%22COMPLETED%22%29+or+%28endTime+gt+%222023-10-05T17%3A53%3A25.002Z%22+and+endTime+le+%222023-10-05T17%3A58%3A30.002Z%22%29%29&orderby=startTime&pageSize=5000&queryState=BEGIN 
YYYY-MM-DDTHH:MM:01.357Z INFO [] [https-jsse-nio-8449-exec-2] [][][][][] [c.e.b.m.f.RestLogFilter.doFilter(56)] -  Exit RESTful Msg -  Status:  200 , ContentType: application/json  
YYYY-MM-DDTHH:MM:15.713Z INFO [] [pool-5-thread-1] [][][][][] [c.e.b.m.e.s.ExternalNotificationServiceMonitoring.getExternalNotificationsToCloseWindow(256)] -  0 External Notifications for summary email. 
################ ---------- ################ ---------- ################ ---------- ################
YYYY-MM-DDTHH:MM:00.094Z INFO [] [not.a.Spring.bean-8] [][][TASK_ID:9d91be8fab969815][TRACE_ID:911d########e63b;JOB_ID:998b66035ec5524c][] [c.e.b.m.a.n.ActivityAlertNotificationHandler.handleMessage(54)] -  Activity state change notification message received. body: {"activityId":"7ead8219-ca50-4dd8-9d91-be8fab969815","messageID":"ARP0030"}  
YYYY-MM-DDTHH:MM:00.099Z INFO [] [pool-4-thread-1] [][][][][] [c.e.b.m.a.s.ActivityAlertService.createOrUpdateAlert(196)] -  activityId 7ead8219-ca50-4dd8-9d91-be8fab969815, messageID ARP0030. 
YYYY-MM-DDTHH:MM:00.375Z ERROR [] [pool-4-thread-1] [][ALERT:ARP0030][][][] [c.e.d.e.c.s.m.DefaultMessageResourceModifier.printAlertLog(143)] -  Date: MMM DD, YYYY 14:04:00 PM; Summary: PPDM-PROD_VLT_MSSQL_PROCESS_PROD: Replication job was unsuccessful for MICROSOFT_SQL_DATABASE DWConfiguration on my_prod_datadomain.abc.com to my_dr_datadomain.abc.com because of data movement error.; Details: The backup copies cannot be replicated because data movement for the copies on the the storage system was unsuccessful.; Recommended Action: Follow the procedure in the PowerProtect Data Manager Administration and User Guide to create a log bundle, and then contact Dell Customer Support.; Detail Summaries: null; Status: UNACKNOWLEDGED 
YYYY-MM-DDTHH:MM:00.375Z INFO [] [pool-4-thread-1] [][][][][] [c.e.b.m.a.s.ActivityAlertService.createOrUpdateAlert(223)] -  Creating newAlertId: b86e8c2a-52e3-4e71-8bbd-3b82f0c7da39 for ancestryId: 51ea3ef5-8e03-429c-ad18-e86eb9e7263f, activityId: 7ead8219-ca50-4dd8-9d91-be8fab969815, messageID: ARP0030 
YYYY-MM-DDTHH:MM:00.483Z INFO [] [https-jsse-nio-8449-exec-5] [][][][][] [c.e.b.a.c.AccessTokenRedeemingRequestFilter.withAuthenticationFromRequest(71)] -  REDEEM: success 
YYYY-MM-DDTHH:MM:00.483Z INFO [] [https-jsse-nio-8449-exec-5] [][][][TRACE_ID:b4dbb37c03557ba7][] [c.e.b.m.f.RestLogFilter.doFilter(45)] -  Enter RESTful Msg - Method: GET , Path: https://myppdm.abc.com:8443/api/v2/alerts/b86e8c2a-52e3-4e71-8bbd-3b82f0c7da39   
YYYY-MM-DDTHH:MM:00.488Z INFO [] [https-jsse-nio-8449-exec-5] [][][][][] [c.e.b.m.f.RestLogFilter.doFilter(56)] -  Exit RESTful Msg -  Status:  200 , ContentType: application/json  
YYYY-MM-DDTHH:MM:04.321Z INFO [] [not.a.Spring.bean-1] [][][TASK_ID:ab2ac64b7c2ec32c][TRACE_ID:911d########e63b;JOB_ID:ab9dc39ded42567b][] [c.e.b.m.a.n.ActivityAlertNotificationHandler.handleMessage(54)] -  Activity state change notification message received. body: {"activityId":"721273bd-23e5-4ddd-ab2a-c64b7c2ec32c","messageID":"ARP0030"}  
YYYY-MM-DDTHH:MM:04.322Z INFO [] [pool-4-thread-1] [][][][][] [c.e.b.m.a.s.ActivityAlertService.createOrUpdateAlert(196)] -  activityId 721273bd-23e5-4ddd-ab2a-c64b7c2ec32c, messageID ARP0030. 
YYYY-MM-DDTHH:MM:04.327Z INFO [] [not.a.Spring.bean-5] [][][TASK_ID:b2e7a5ced8379d07][TRACE_ID:911d########e63b;JOB_ID:a2dd38f4a44dc164][] [c.e.b.m.a.n.ActivityAlertNotificationHandler.handleMessage(54)] -  Activity state change notification message received. body: {"activityId":"8923eb9c-997e-45f0-b2e7-a5ced8379d07","messageID":"ARP0030"}  
YYYY-MM-DDTHH:MM:04.337Z INFO [] [not.a.Spring.bean-4] [][][TASK_ID:a64cb2db470779a2][TRACE_ID:911d########e63b;JOB_ID:8adc301f67163e72][] [c.e.b.m.a.n.ActivityAlertNotificationHandler.handleMessage(54)] -  Activity state change notification message received. body: {"activityId":"81df708e-6406-4900-a64c-b2db470779a2","messageID":"ARP0030"}  
YYYY-MM-DDTHH:MM:04.371Z INFO [] [not.a.Spring.bean-7] [][][TASK_ID:ae36########44e4][TRACE_ID:911d########e63b;JOB_ID:a6d0########9115][] [c.e.b.m.a.n.ActivityAlertNotificationHandler.handleMessage(54)] -  Activity state change notification message received. body: {"activityId":"3026f531-1fdc-4e1f-ae36-edf3c6a044e4","messageID":"ARP0030"}  
YYYY-MM-DDTHH:MM:05.225Z INFO [] [not.a.Spring.bean-2] [][][TASK_ID:b9058ca36192e544][TRACE_ID:862d0db52e63ef03;JOB_ID:8e4c099fd821b4a2][] [c.e.b.m.a.n.ActivityAlertNotificationHandler.handleMessage(54)] -  Activity state change notification message received. body: {"activityId":"a1079476-fc94-44c5-b905-8ca36192e544","messageID":"ARP0030"}  
YYYY-MM-DDTHH:MM:05.280Z INFO [] [not.a.Spring.bean-6] [][][TASK_ID:9985aa79a8cbd17d][TRACE_ID:862d0db52e63ef03;JOB_ID:9f74f0e898e21669][] [c.e.b.m.a.n.ActivityAlertNotificationHandler.handleMessage(54)] -  Activity state change notification message received. body: {"activityId":"e8058a39-a816-4d09-9985-aa79a8cbd17d","messageID":"ARP0030"}  
YYYY-MM-DDTHH:MM:05.367Z INFO [] [not.a.Spring.bean-3] [][][TASK_ID:9aecaba1c9b27a55][TRACE_ID:911d########e63b;JOB_ID:890b4ea199a2cbfe][] [c.e.b.m.a.n.ActivityAlertNotificationHandler.handleMessage(54)] -  Activity state change notification message received. body: {"activityId":"3bc94ee5-21e5-4119-9aec-aba1c9b27a55","messageID":"ARP0030"}  
YYYY-MM-DDTHH:MM:05.373Z INFO [] [not.a.Spring.bean-8] [][][TASK_ID:abd3940d23f110e7][TRACE_ID:911d########e63b;JOB_ID:97a781c68e4a4476][] [c.e.b.m.a.n.ActivityAlertNotificationHandler.handleMessage(54)] -  Activity state change notification message received. body: {"activityId":"fdd4f333-db04-430b-abd3-940d23f110e7","messageID":"ARP0030"}  
YYYY-MM-DDTHH:MM:05.384Z INFO [] [not.a.Spring.bean-10] [][][TASK_ID:836373542d1eaeff][TRACE_ID:911d########e63b;JOB_ID:8ce8c387e035c399][] [c.e.b.m.a.n.ActivityAlertNotificationHandler.handleMessage(54)] -  Activity state change notification message received. body: {"activityId":"7bf67f00-a691-4d37-8363-73542d1eaeff","messageID":"ARP0030"}  
YYYY-MM-DDTHH:MM:05.434Z INFO [] [not.a.Spring.bean-1] [][][TASK_ID:b525c4bdf4612dc7][TRACE_ID:911d########e63b;JOB_ID:8eff47d26bf2292f][] [c.e.b.m.a.n.ActivityAlertNotificationHandler.handleMessage(54)] -  Activity state change notification message received. body: {"activityId":"883332f8-0a64-4231-b525-c4bdf4612dc7","messageID":"ARP0030"}  
YYYY-MM-DDTHH:MM:06.092Z ERROR [] [pool-4-thread-1] [][ALERT:ARP0030][][][] [c.e.d.e.c.s.m.DefaultMessageResourceModifier.printAlertLog(143)] -  Date: MMM DD, YYYY 14:04:00 PM; Summary: PPDM-PROD_VLT_MSSQL_PROCESS_PROD: Replication job was unsuccessful for MICROSOFT_SQL_DATABASE DWConfiguration on my_prod_datadomain.abc.com to my_dr_datadomain.abc.com because of data movement error.; Details: The backup copies cannot be replicated because data movement for the copies on the the storage system was unsuccessful.; Recommended Action: Follow the procedure in the PowerProtect Data Manager Administration and User Guide to create a log bundle, and then contact Dell Customer Support.; Detail Summaries: null; Status: UNACKNOWLEDGED 
YYYY-MM-DDTHH:MM:06.092Z INFO [] [pool-4-thread-1] [][][][][] [c.e.b.m.a.s.ActivityAlertService.createOrUpdateAlert(216)] -  Updating existingAlertId: b86e8c2a-52e3-4e71-8bbd-3b82f0c7da39 for ancestryId: 51ea3ef5-8e03-429c-ad18-e86eb9e7263f, activityId: 721273bd-23e5-4ddd-ab2a-c64b7c2ec32c, messageID: ARP0030 instead of creating new alert 
YYYY-MM-DDTHH:MM:06.321Z INFO [] [pool-4-thread-1] [][][][][] [c.e.b.m.a.s.ActivityAlertService.createOrUpdateAlert(196)] -  activityId 8923eb9c-997e-45f0-b2e7-a5ced8379d07, messageID ARP0030. 
YYYY-MM-DDTHH:MM:07.853Z ERROR [] [pool-4-thread-1] [][ALERT:ARP0030][][][] [c.e.d.e.c.s.m.DefaultMessageResourceModifier.printAlertLog(143)] -  Date: MMM DD, YYYY 14:04:00 PM; Summary: PPDM-PROD_VLT_MSSQL_PROCESS_PROD: Replication job was unsuccessful for MICROSOFT_SQL_DATABASE DWConfiguration on my_prod_datadomain.abc.com to my_dr_datadomain.abc.com because of data movement error.; Details: The backup copies cannot be replicated because data movement for the copies on the the storage system was unsuccessful.; Recommended Action: Follow the procedure in the PowerProtect Data Manager Administration and User Guide to create a log bundle, and then contact Dell Customer Support.; Detail Summaries: null; Status: UNACKNOWLEDGED 
YYYY-MM-DDTHH:MM:07.853Z INFO [] [pool-4-thread-1] [][][][][] [c.e.b.m.a.s.ActivityAlertService.createOrUpdateAlert(216)] -  Updating existingAlertId: b86e8c2a-52e3-4e71-8bbd-3b82f0c7da39 for ancestryId: 51ea3ef5-8e03-429c-ad18-e86eb9e7263f, activityId: 8923eb9c-997e-45f0-b2e7-a5ced8379d07, messageID: ARP0030 instead of creating new alert 
YYYY-MM-DDTHH:MM:07.896Z INFO [] [pool-4-thread-1] [][][][][] [c.e.b.m.a.s.ActivityAlertService.createOrUpdateAlert(196)] -  activityId 81df708e-6406-4900-a64c-b2db470779a2, messageID ARP0030. 
YYYY-MM-DDTHH:MM:09.571Z ERROR [] [pool-4-thread-1] [][ALERT:ARP0030][][][] [c.e.d.e.c.s.m.DefaultMessageResourceModifier.printAlertLog(143)] -  Date: MMM DD, YYYY 14:04:00 PM; Summary: PPDM-PROD_VLT_MSSQL_PROCESS_PROD: Replication job was unsuccessful for MICROSOFT_SQL_DATABASE DWConfiguration on my_prod_datadomain.abc.com to my_dr_datadomain.abc.com because of data movement error.; Details: The backup copies cannot be replicated because data movement for the copies on the the storage system was unsuccessful.; Recommended Action: Follow the procedure in the PowerProtect Data Manager Administration and User Guide to create a log bundle, and then contact Dell Customer Support.; Detail Summaries: null; Status: UNACKNOWLEDGED 
YYYY-MM-DDTHH:MM:09.571Z INFO [] [pool-4-thread-1] [][][][][] [c.e.b.m.a.s.ActivityAlertService.createOrUpdateAlert(216)] -  Updating existingAlertId: b86e8c2a-52e3-4e71-8bbd-3b82f0c7da39 for ancestryId: 51ea3ef5-8e03-429c-ad18-e86eb9e7263f, activityId: 81df708e-6406-4900-a64c-b2db470779a2, messageID: ARP0030 instead of creating new alert 
YYYY-MM-DDTHH:MM:09.667Z INFO [] [pool-4-thread-1] [][][][][] [c.e.b.m.a.s.ActivityAlertService.createOrUpdateAlert(196)] -  activityId 3026f531-1fdc-4e1f-ae36-edf3c6a044e4, messageID ARP0030. 
YYYY-MM-DDTHH:MM:10.367Z ERROR [] [pool-4-thread-1] [][ALERT:ARP0030][][][] [c.e.d.e.c.s.m.DefaultMessageResourceModifier.printAlertLog(143)] -  Date: MMM DD, YYYY 14:04:00 PM; Summary: PPDM-PROD_VLT_MSSQL_PROCESS_PROD: Replication job was unsuccessful for MICROSOFT_SQL_DATABASE DWConfiguration on my_prod_datadomain.abc.com to my_dr_datadomain.abc.com because of data movement error.; Details: The backup copies cannot be replicated because data movement for the copies on the the storage system was unsuccessful.; Recommended Action: Follow the procedure in the PowerProtect Data Manager Administration and User Guide to create a log bundle, and then contact Dell Customer Support.; Detail Summaries: null; Status: UNACKNOWLEDGED 
YYYY-MM-DDTHH:MM:10.367Z INFO [] [pool-4-thread-1] [][][][][] [c.e.b.m.a.s.ActivityAlertService.createOrUpdateAlert(216)] -  Updating existingAlertId: b86e8c2a-52e3-4e71-8bbd-3b82f0c7da39 for ancestryId: 51ea3ef5-8e03-429c-ad18-e86eb9e7263f, activityId: 3026f531-1fdc-4e1f-ae36-edf3c6a044e4, messageID: ARP0030 instead of creating new alert 
YYYY-MM-DDTHH:MM:10.471Z INFO [] [pool-4-thread-1] [][][][][] [c.e.b.m.a.s.ActivityAlertService.createOrUpdateAlert(196)] -  activityId a1079476-fc94-44c5-b905-8ca36192e544, messageID ARP0030. 
YYYY-MM-DDTHH:MM:10.687Z ERROR [] [pool-8-thread-1] [][][][TRACE_ID:86c6########bb9b][] [c.d.b.m.h.i.SystemHealthServiceImpl.computeSystemHealthMetrics(129)] -  Failed to compute system health metrics 
java.util.concurrent.CompletionException: com.dell.brs.monitoring.health.exception.HealthServiceException: Health Check: SupportConfigHealthCheck timed out. 
    at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:314)
    at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:319)
    at java.base/java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:990)
    at java.base/java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:970)
    at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
    at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088)
    at java.base/java.util.concurrent.CompletableFuture$Timeout.run(CompletableFuture.java:2792)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: com.dell.brs.monitoring.health.exception.HealthServiceException: Health Check: SupportConfigHealthCheck timed out. 
    at com.dell.brs.monitoring.health.SystemHealthCalculatorImpl.lambda$computeHealthChecks$13(SystemHealthCalculatorImpl.java:309)
    at java.base/java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:986)
    ... 10 common frames omitted
Caused by: java.util.concurrent.TimeoutException: null
    ... 7 common frames omitted
YYYY-MM-DDTHH:MM:10.688Z INFO [] [pool-8-thread-1] [][][][TRACE_ID:bb798ef84b9e2a29][] [c.e.b.t.i.LogTracingServiceImpl.startTrace(57)] -  === Start a new Log Trace for Compute System Health Metrics === 
YYYY-MM-DDTHH:MM:11.074Z ERROR [] [pool-4-thread-1] [][ALERT:ARP0030][][][] [c.e.d.e.c.s.m.DefaultMessageResourceModifier.printAlertLog(143)] -  Date: MMM DD, YYYY 14:04:11 PM; Summary: PPDM-PROD_VLT_MSSQL_PROCESS_DEV: Replication job was unsuccessful for MICROSOFT_SQL_DATABASE CokemakingDev on my_prod_datadomain.abc.com to my_dr_datadomain.abc.com because of data movement error.; Details: The backup copies cannot be replicated because data movement for the copies on the the storage system was unsuccessful.; Recommended Action: Follow the procedure in the PowerProtect Data Manager Administration and User Guide to create a log bundle, and then contact Dell Customer Support.; Detail Summaries: null; Status: UNACKNOWLEDGED 
YYYY-MM-DDTHH:MM:11.074Z INFO [] [pool-4-thread-1] [][][][][] [c.e.b.m.a.s.ActivityAlertService.createOrUpdateAlert(223)] -  Creating newAlertId: 8edc9bb2-65b6-40a6-9ebb-aa61536e8378 for ancestryId: 16b707c2-7c84-4f21-9046-a1f4cac8923d, activityId: a1079476-fc94-44c5-b905-8ca36192e544, messageID: ARP0030 
YYYY-MM-DDTHH:MM:11.121Z INFO [] [pool-4-thread-1] [][][][][] [c.e.b.m.a.s.ActivityAlertService.createOrUpdateAlert(196)] -  activityId e8058a39-a816-4d09-9985-aa79a8cbd17d, messageID ARP0030. 
YYYY-MM-DDTHH:MM:11.162Z INFO [] [https-jsse-nio-8449-exec-9] [][][][][] [c.e.b.a.c.AccessTokenRedeemingRequestFilter.withAuthenticationFromRequest(71)] -  REDEEM: success 
YYYY-MM-DDTHH:MM:11.162Z INFO [] [https-jsse-nio-8449-exec-9] [][][][TRACE_ID:8b1fec7af5b93e3d][] [c.e.b.m.f.RestLogFilter.doFilter(45)] -  Enter RESTful Msg - Method: GET , Path: https://myppdm.abc.com:8443/api/v2/alerts/8edc9bb2-65b6-40a6-9ebb-aa61536e8378   
YYYY-MM-DDTHH:MM:11.249Z INFO [] [https-jsse-nio-8449-exec-9] [][][][][] [c.e.b.m.f.RestLogFilter.doFilter(56)] -  Exit RESTful Msg -  Status:  200 , ContentType: application/json  
YYYY-MM-DDTHH:MM:11.764Z ERROR [] [pool-4-thread-1] [][ALERT:ARP0030][][][] [c.e.d.e.c.s.m.DefaultMessageResourceModifier.printAlertLog(143)] -  Date: MMM DD, YYYY 14:04:11 PM; Summary: PPDM-PROD_VLT_MSSQL_PROCESS_DEV: Replication job was unsuccessful for MICROSOFT_SQL_DATABASE CokemakingDev on my_prod_datadomain.abc.com to my_dr_datadomain.abc.com because of data movement error.; Details: The backup copies cannot be replicated because data movement for the copies on the the storage system was unsuccessful.; Recommended Action: Follow the procedure in the PowerProtect Data Manager Administration and User Guide to create a log bundle, and then contact Dell Customer Support.; Detail Summaries: null; Status: UNACKNOWLEDGED 
YYYY-MM-DDTHH:MM:11.764Z INFO [] [pool-4-thread-1] [][][][][] [c.e.b.m.a.s.ActivityAlertService.createOrUpdateAlert(216)] -  Updating existingAlertId: 8edc9bb2-65b6-40a6-9ebb-aa61536e8378 for ancestryId: 16b707c2-7c84-4f21-9046-a1f4cac8923d, activityId: e8058a39-a816-4d09-9985-aa79a8cbd17d, messageID: ARP0030 instead of creating new alert 
YYYY-MM-DDTHH:MM:11.799Z INFO [] [pool-4-thread-1] [][][][][] [c.e.b.m.a.s.ActivityAlertService.createOrUpdateAlert(196)] -  activityId 3bc94ee5-21e5-4119-9aec-aba1c9b27a55, messageID ARP0030. 
YYYY-MM-DDTHH:MM:12.474Z ERROR [] [pool-4-thread-1] [][ALERT:ARP0030][][][] [c.e.d.e.c.s.m.DefaultMessageResourceModifier.printAlertLog(143)] -  Date: MMM DD, YYYY 14:04:00 PM; Summary: PPDM-PROD_VLT_MSSQL_PROCESS_PROD: Replication job was unsuccessful for MICROSOFT_SQL_DATABASE DWConfiguration on my_prod_datadomain.abc.com to my_dr_datadomain.abc.com because of data movement error.; Details: The backup copies cannot be replicated because data movement for the copies on the the storage system was unsuccessful.; Recommended Action: Follow the procedure in the PowerProtect Data Manager Administration and User Guide to create a log bundle, and then contact Dell Customer Support.; Detail Summaries: null; Status: UNACKNOWLEDGED 
YYYY-MM-DDTHH:MM:12.475Z INFO [] [pool-4-thread-1] [][][][][] [c.e.b.m.a.s.ActivityAlertService.createOrUpdateAlert(216)] -  Updating existingAlertId: b86e8c2a-52e3-4e71-8bbd-3b82f0c7da39 for ancestryId: 51ea3ef5-8e03-429c-ad18-e86eb9e7263f, activityId: 3bc94ee5-21e5-4119-9aec-aba1c9b27a55, messageID: ARP0030 instead of creating new alert 
YYYY-MM-DDTHH:MM:12.561Z INFO [] [pool-4-thread-1] [][][][][] [c.e.b.m.a.s.ActivityAlertService.createOrUpdateAlert(196)] -  activityId fdd4f333-db04-430b-abd3-940d23f110e7, messageID ARP0030. 
YYYY-MM-DDTHH:MM:12.927Z ERROR [] [pool-4-thread-1] [][ALERT:ARP0030][][][] [c.e.d.e.c.s.m.DefaultMessageResourceModifier.printAlertLog(143)] -  Date: MMM DD, YYYY 14:04:00 PM; Summary: PPDM-PROD_VLT_MSSQL_PROCESS_PROD: Replication job was unsuccessful for MICROSOFT_SQL_DATABASE DWConfiguration on my_prod_datadomain.abc.com to my_dr_datadomain.abc.com because of data movement error.; Details: The backup copies cannot be replicated because data movement for the copies on the the storage system was unsuccessful.; Recommended Action: Follow the procedure in the PowerProtect Data Manager Administration and User Guide to create a log bundle, and then contact Dell Customer Support.; Detail Summaries: null; Status: UNACKNOWLEDGED 
YYYY-MM-DDTHH:MM:12.927Z INFO [] [pool-4-thread-1] [][][][][] [c.e.b.m.a.s.ActivityAlertService.createOrUpdateAlert(216)] -  Updating existingAlertId: b86e8c2a-52e3-4e71-8bbd-3b82f0c7da39 for ancestryId: 51ea3ef5-8e03-429c-ad18-e86eb9e7263f, activityId: fdd4f333-db04-430b-abd3-940d23f110e7, messageID: ARP0030 instead of creating new alert 
YYYY-MM-DDTHH:MM:12.966Z INFO [] [pool-4-thread-1] [][][][][] [c.e.b.m.a.s.ActivityAlertService.createOrUpdateAlert(196)] -  activityId 7bf67f00-a691-4d37-8363-73542d1eaeff, messageID ARP0030. 
YYYY-MM-DDTHH:MM:13.195Z ERROR [] [pool-4-thread-1] [][ALERT:ARP0030][][][] [c.e.d.e.c.s.m.DefaultMessageResourceModifier.printAlertLog(143)] -  Date: MMM DD, YYYY 14:04:00 PM; Summary: PPDM-PROD_VLT_MSSQL_PROCESS_PROD: Replication job was unsuccessful for MICROSOFT_SQL_DATABASE DWConfiguration on my_prod_datadomain.abc.com to my_dr_datadomain.abc.com because of data movement error.; Details: The backup copies cannot be replicated because data movement for the copies on the the storage system was unsuccessful.; Recommended Action: Follow the procedure in the PowerProtect Data Manager Administration and User Guide to create a log bundle, and then contact Dell Customer Support.; Detail Summaries: null; Status: UNACKNOWLEDGED 
YYYY-MM-DDTHH:MM:13.195Z INFO [] [pool-4-thread-1] [][][][][] [c.e.b.m.a.s.ActivityAlertService.createOrUpdateAlert(216)] -  Updating existingAlertId: b86e8c2a-52e3-4e71-8bbd-3b82f0c7da39 for ancestryId: 51ea3ef5-8e03-429c-ad18-e86eb9e7263f, activityId: 7bf67f00-a691-4d37-8363-73542d1eaeff, messageID: ARP0030 instead of creating new alert 
YYYY-MM-DDTHH:MM:13.233Z INFO [] [pool-4-thread-1] [][][][][] [c.e.b.m.a.s.ActivityAlertService.createOrUpdateAlert(196)] -  activityId 883332f8-0a64-4231-b525-c4bdf4612dc7, messageID ARP0030. 
YYYY-MM-DDTHH:MM:13.356Z ERROR [] [pool-4-thread-1] [][ALERT:ARP0030][][][] [c.e.d.e.c.s.m.DefaultMessageResourceModifier.printAlertLog(143)] -  Date: MMM DD, YYYY 14:04:00 PM; Summary: PPDM-PROD_VLT_MSSQL_PROCESS_PROD: Replication job was unsuccessful for MICROSOFT_SQL_DATABASE DWConfiguration on my_prod_datadomain.abc.com to my_dr_datadomain.abc.com because of data movement error.; Details: The backup copies cannot be replicated because data movement for the copies on the the storage system was unsuccessful.; Recommended Action: Follow the procedure in the PowerProtect Data Manager Administration and User Guide to create a log bundle, and then contact Dell Customer Support.; Detail Summaries: null; Status: UNACKNOWLEDGED 
>>>>>>>>>>>> YYYY-MM-DDTHH:MM:13.357Z INFO [] [pool-4-thread-1] [][][][][] [c.e.b.m.a.s.ActivityAlertService.createOrUpdateAlert(216)] -  Updating existingAlertId: b86e8c2a-52e3-4e71-8bbd-3b82f0c7da39 for ancestryId: 51ea3ef5-8e03-429c-ad18-e86eb9e7263f, activityId: 883332f8-0a64-4231-b525-c4bdf4612dc7, messageID: ARP0030 instead of creating new alert 
YYYY-MM-DDTHH:MM:15.765Z INFO [] [pool-5-thread-1] [][][][][] [c.e.b.m.e.s.ExternalNotificationServiceMonitoring.getExternalNotificationsToCloseWindow(256)] -  0 External Notifications for summary email. 
YYYY-MM-DDTHH:MM:40.697Z INFO [] [pool-9-thread-1] [][][][TRACE_ID:a5d20d83be23e52a][] [c.e.b.t.i.LogTracingServiceImpl.startTrace(57)] -  === Start a new Log Trace for Compute System Manager Health Checks === 
YYYY-MM-DDTHH:MM:40.818Z INFO [] [https-jsse-nio-8449-exec-3] [][][][][] [c.e.b.a.c.AccessTokenRedeemingRequestFilter.withAuthenticationFromRequest(71)] -  REDEEM: success 
YYYY-MM-DDTHH:MM:40.819Z INFO [] [https-jsse-nio-8449-exec-3] [][][][TRACE_ID:a5d20d83be23e52a][] [c.e.b.m.f.RestLogFilter.doFilter(45)] -  Enter RESTful Msg - Method: GET , Path: https://localhost:8443/api/v2/report-nodes   
YYYY-MM-DDTHH:MM:40.839Z INFO [] [https-jsse-nio-8449-exec-3] [][][][][] [c.e.b.m.f.RestLogFilter.doFilter(56)] -  Exit RESTful Msg -  Status:  200 , ContentType: application/json  
YYYY-MM-DDT18:05:15.702Z INFO [] [pool-5-thread-1] [][][][][] [c.e.b.m.e.s.ExternalNotificationServiceMonitoring.getExternalNotificationsToCloseWindow(256)] -  0 External Notifications for summary email. 
YYYY-MM-DDT18:06:10.712Z ERROR [] [pool-8-thread-1] [][][][TRACE_ID:bb798ef84b9e2a29][] [c.d.b.m.h.i.SystemHealthServiceImpl.computeSystemHealthMetrics(129)] -  Failed to compute system health metrics 
java.util.concurrent.CompletionException: com.dell.brs.monitoring.health.exception.HealthServiceException: Health Check: SupportConfigHealthCheck timed out.

原因

Connection between PowerProtect Data Manager and Data Domain device is timing out. Causing replication failure with error code ARP0030.

解决方案

On PowerProtect Data Manager CLI perform the following:

1. Create /usr/local/brs/lib/replmgr/config/application.properties file: 
touch /usr/local/brs/lib/replmgr/config/application.properties
2. Run: 
chmod ug+rx /usr/local/brs/lib/replmgr/config/application.properties
3. Run: 
chown admin:app /usr/local/brs/lib/replmgr/config/application.properties
4. Run: 
vi /usr/local/brs/lib/replmgr/config/application.properties
5. Add these lines: 
ddboost.dynamic.connection.check.enable=false 
ddboost.dedicated.connection.max.coun=64 
ddboost.shared.connection.max.count=64
6. Save the file - Run: 
:x!
7. Perform a restart of the replmgr - Run: 
replmgr restart
8. Test the Replication.

产品

PowerProtect Software
文章属性
文章编号: 000218427
文章类型: Solution
上次修改时间: 23 10月 2024
版本:  4
从其他戴尔用户那里查找问题的答案
支持服务
检查您的设备是否在支持服务涵盖的范围内。