PowerFlex: Replication Initial Copy Failure

Summary: Initial copy progress is stuck. Sometimes, it might succeed to reach 100% completion for small volumes, while large volumes failed to complete the initial copy.

This article applies to This article does not apply to This article is not tied to any specific product. Not all product versions are identified in this article.

Check out other resources

Symptoms

If there are network issues or low bandwidth between the source and the target systems, the initial copy stage might never finish.

A possible scenario is that the initial copy will be completed for small-sized volumes, however on Replication Consistency Groups (RCGs) that contain large volumes - the source and target volumes are not able to reach 100% consistency, and source journal capacity utilization is increasing as new writes are received by the source volume.

Note: Even if the large source volume can finish the initial copy, or if an offline initial copy was done, if there is network issues, and writes/rewrites on the source volume we might not meet RPO and lose consistency between source and target.

Symptoms

The initial copy progress is stuck, for example, the "Initial Copy progress" will show permanently 70%:

 # scli --query_all_replication_pairs
Local Id: bf50999700000000 Remote Id: bad02d5600000000 Copy Type: ONLINE_COPY Initial Copy State: ACTIVE Name: Rep
Local Volume Id: 2236ff5400000001 Remote Volume Id: 831e94c40000000d Local Volume Name: vol_lir1 Remote Volume Name: vol_lir1
Initial Copy progress: 70%
query_all_replication_pairs returned 1 Replication Pairs.

2- Checking the SDR traces the next errors are identified: Network issues

 27/02 13:39:11.051442 (nil):netCon_ExchangeNewCon_CK:03589: ERROR: Con 0x7f42a80fc5c8 sock 0x7f42a80fd148, Can't send NEW_CON message
27/02 13:39:11.051454 (nil):netCon_ConnectToPeer_CK:03519: Con 0x7f42a80fc5c8 sock 0x7f42a80fd148 rollback connect
27/02 13:39:11.092475 (nil):netCon_IsKaNeeded:04338: ERROR: CON 0x7f42a80fb660 failed to send for 180 iterations. Marking as down
.
.
.
27/02 13:39:12.628337 (nil):mapVolIO_ReportIOErrorIfNeeded:00469: Vol ID 0x21c5f4c600000032. Last vol network error status NOT_CONN(4) Reason (NOT_CONN) RC (NOT_CONN) Retry count (0) chan (0) Total IO millis (1010)
27/02 13:39:12.637268 (nil):mapVolIO_Do_CK:00884: Mit: 0x7f40d59b17d8. Vol ID 0x21c5f4c600000032. Send error. rc NOT_CONN(4)
27/02 13:39:12.637288 (nil):mapVolIO_Do_CK:00884: Mit: 0x7f40d5a46030. Vol ID 0x21c5f4c600000032. Send error. rc NOT_CONN(4)

Initial copy state, where small volumes are OK but larger once are unable to finish.

 27/02 13:39:10.774502 0x7f42814f9db0:initialCopier_SendSingleVaeDoneMsg:01871: (SDR_INITIAL_COPIER) Initial Copier - Got 2 SYSTEM_INITIAL_COPY CGs and 1 done VAEs
27/02 13:39:10.774520 0x7f42814f9db0:initialCopierCg_FillCgVaeDoneMsg:01963: (SDR_INITIAL_COPIER) Filled VAE Done MSG for Pair-Id de9ddcc900000001, GenNum: 2137, VaeId: 14
27/02 13:39:10.774633 0x7f42814f9db0:initialCopier_SendSingleVaeDoneMsg:01921: (SDR_INITIAL_COPIER) Initial Copier - Total number of 1 VAEs will be reported as Done

3- Checking SDR connectivity, you might see some or all SDR to SDR connections as disconnected.

 scli --query_disconnection --print_all
peer_system ID: ff74bce300000000 Name: DR_PFlex_GW
SDR-SDR connectivity status: Multiple disconnections
SDR ID: 2d01669c00000000 Name: Sdr-172.24.194.137 is disconnected from:
	SDR ID: 67b9def800000000 Name: N/A IPs: 172.24.105.4 172.24.105.2 172.24.100.2	SDR ID: 67ba060800000001 Name: N/A IPs: 172.24.105.6 172.24.105.4 172.24.100.4	SDR ID: 67ba060900000002 Name: N/A IPs: 172.24.105.5	SDR ID: 67ba2d1800000003 Name: N/A IPs: 172.24.135.7 172.24.100.7
SDR ID: 2d01669d00000001 Name: Sdr-172.24.194.138 is disconnected from:
	SDR ID: 67ba060800000001 Name: N/A IPs: 172.24.105.6 172.24.105.4 172.24.100.4	SDR ID: 67b9def800000000 Name: N/A IPs: 172.24.105.4 172.24.105.2 172.24.100.2	SDR ID: 67ba060900000002 Name: N/A IPs: 172.24.105.5	SDR ID: 67ba2d1800000003 Name: N/A IPs: 172.24.135.7 172.24.100.7
SDR ID: 2d01669e00000002 Name: Sdr-172.24.194.135 is disconnected from:
	SDR ID: 67b9def800000000 Name: N/A IPs: 172.24.105.4 172.24.105.2 172.24.100.2	SDR ID: 67ba060800000001 Name: N/A IPs: 172.24.105.6 172.24.105.4 172.24.100.4	SDR ID: 67ba060900000002 Name: N/A IPs: 172.24.105.5	SDR ID: 67ba2d1800000003 Name: N/A IPs: 172.24.135.7 172.24.100.7
SDR ID: 2d01669f00000003 Name: Sdr-172.24.194.136 is disconnected from:
	SDR ID: 67b9def800000000 Name: N/A IPs: 172.24.105.4 172.24.105.2 172.24.100.2	SDR ID: 67ba060800000001 Name: N/A IPs: 172.24.105.6 172.24.105.4 172.24.100.4	SDR ID: 67ba060900000002 Name: N/A IPs: 172.24.105.5	SDR ID: 67ba2d1800000003 Name: N/A IPs: 172.24.135.7 172.24.100.7

4- Even if all SDR to SDR connectivity is fine, a possible condition with the same symptoms of initial copy failure might occur if the network capacity is poor and there is not enough bandwidth to finish successfully the initial copy.

To test the bandwidth between the two sites, you can use different tools - for example, "iperf":

Any testing of bandwidth might impact the network between the sites - if there is WAN usage for Site2Site replication, let the customer know that this test should be conducted during low business hours and considering that the network that the test is conducted for will not be usable during the duration of the test.

Validate you run the test long enough to have proper readings - like transferring large volumes - also note that this test stresses customers' network.

Site1

 [root@112AX-7 ~]# iperf3 -s
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 172.24.1.174, port 54882
[  5] local 172.24.1.173 port 5201 connected to 172.24.1.174 port 54886
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-1.00   sec  2.76 GBytes  23.7 Gbits/sec                  
[  5]   1.00-2.00   sec  2.88 GBytes  24.7 Gbits/sec                  
[  5]   2.00-3.00   sec  2.88 GBytes  24.7 Gbits/sec                  
[  5]   3.00-4.00   sec  2.87 GBytes  24.7 Gbits/sec                  
[  5]   4.00-5.00   sec  2.88 GBytes  24.7 Gbits/sec                  
[  5]   5.00-5.04   sec   114 MBytes  24.7 Gbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-5.04   sec  0.00 Bytes  0.00 bits/sec                  sender
[  5]   0.00-5.04   sec  14.4 GBytes  24.5 Gbits/sec                  receiver
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------

Site2

 [root@112AX-8 ~]# iperf3 -c 172.24.1.173 -i 1 -t 5
Connecting to host 172.24.1.173, port 5201
[  4] local 172.24.1.174 port 54886 connected to 172.24.1.173 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec  2.87 GBytes  24.7 Gbits/sec   12    734 KBytes       
[  4]   1.00-2.00   sec  2.88 GBytes  24.7 Gbits/sec    0    778 KBytes       
[  4]   2.00-3.00   sec  2.88 GBytes  24.7 Gbits/sec    0    778 KBytes       
[  4]   3.00-4.00   sec  2.87 GBytes  24.7 Gbits/sec    0    786 KBytes       
[  4]   4.00-5.00   sec  2.88 GBytes  24.7 Gbits/sec    0    804 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-5.00   sec  14.4 GBytes  24.7 Gbits/sec   12             sender
[  4]   0.00-5.00   sec  14.4 GBytes  24.7 Gbits/sec                  receiveriperf Done.

Impact

Replication is not established - site1 will not be consistent with site2, if there is replication from a Production to DR site, the DR site cannot overtake if there is a failover scenario.

Cause

Validate that the SDR process is started and stable on both sites - if it does, then the general root cause is most probably related to network issues between site1 and site2, hence PowerFlex is not responsible for the initial copy issue. Having said that, the initial copy will not work unless the network issues are treated.

The problem could be one of the following:

Wrong configuration of the SDR IPs
No route between SDRs on site1 and SDRs on site2
SDR IP roles are misconfigured
Check that the firewall validates that Port 11088 is not blocked
Wrong/unaligned MTU configuration between the sites
Any other network issue

Resolution

Workaround

To validate connectivity between SDRs, run scli --query_disconnection --print_all - validate there are no disconnected SDRs.
If all are connected, but the initial copy still fails, please perform a bandwidth test between site1 and site2 as described above in the iperf example.

If there are no network issues found and SDR to SDR connectivity is fine, perform a bandwidth test, and if the test returns successful, and yet the initial copy fails to complete. Please open a case with support for further assistance.

Note: If the network is fine but the source volume is large and the initial copy fails to finish, check the user manual for an "offline initial copy" option.

Impacted Versions

All PowerFlex versions running replication

Fixed In Version

Not a PowerFlex related issue

Affected Products

PowerFlex rack, ScaleIO

Article Number: 000204785

Article Type: Solution

Last Modified: 21 Oct 2025

Version: 6

Check if your device is covered by Support Services.

PowerFlex: Replication Initial Copy Failure

Summary: Initial copy progress is stuck. Sometimes, it might succeed to reach 100% completion for small volumes, while large volumes failed to complete the initial copy.

Symptoms

Cause

Resolution

Affected Products

Article Properties

Find answers to your questions from other Dell users

Support Services

Article Properties

Find answers to your questions from other Dell users

Support Services

PowerFlex: Replication Initial Copy Failure

Summary: Initial copy progress is stuck. Sometimes, it might succeed to reach 100% completion for small volumes, while large volumes failed to complete the initial copy.

Detailed Article

Symptoms

Cause

Resolution

Affected Products

Symptoms

Cause

Resolution

Affected Products

Article Properties

Find answers to your questions from other Dell users

Support Services

Article Properties

Find answers to your questions from other Dell users

Support Services