The destination DD is already hosting data from other production environments, so I’m not able to do the Collection Replication.
I previously verified that the replication traffic was going over the private network a few times. But that command was an interesting view of each port utilization.
The private network was setup as a round-robin aggregation of three 1gb links. Per the various port monitoring options, all ports were evenly supporting the total traffic passed to the veth port. Other network configurations were tried, to the point of the current configuration of a single direct link between the two DDs with no network switch. The 1gb Ethernet ports are auto-sensing, and don’t require a cross-over cable. I probably should have tried this sooner. In any case, there has been no difference to the replication rate or the amount of traffic being passed. I think I have ruled out any network connectivity and configuration issues at this point.
You may be on to something with the deduplication suggestion though. I am doing some replication performance testing with various data types, and nothing that has been on either DD before. I’ll let you know what I see.
I had considered the OS version differences, but per the Data Domain Administrators Guide, Replication should work find provided the two DDs are within two revision codes of each other, which I believe these are. A review of these DDOS versions found no known issues with the replication feature. If I had more time to work on this, I suppose I would do an upgrade anyway, but having to work other tasks, I am not likely going to get to that any time soon.
I did some CIFS Replication performance testing with a variety of data types, to include typical misc. text based documents files, large videos, virtual machine images, ISO images, pre-compressed files, etc…
The majority of these file types replicated relatively quickly. So I believe the network and DD Replication configurations are fine. I suspect that the slow performance may just be with this particular data.
I recently learned that Data Domains are not able to process encrypted and/or pre-compressed data as efficiently as other data. The data I am replicating is the backup files from our SourceOne server that backs up our MS Exchange mail servers. So I suspect this may be the case, but I am awaiting confirmation from our system admin who manages those environments.
In any case, I don’t believe there is anything more that I can do but to just let it ride as-is and wait it out.
Thank you all for your feedback and suggestions. They definitely helped.
This sounds very similar to what I am seeing. I have been seeding data from one DD990 to another DD990 with a 10Gbit NIC on both sides for the replication traffic over CIFS. I have 3 different replication contexts going. One is Oracle datapump/export type traffic. Another is RMAN based backups (archive log and full backups), and the 3rd is SQL Server backups (tlogs and full backups). The SQL Server backups are loaded on a CIFS share. Typically I am seeing about 15MB/s running on the SQL replication context where I see 90-120MB/s on the RMAN replication. I really do think it's an artifact of the type of data it is.
On a side note, my replication performance increased when I upgraded to 5.4.4.2. I was on 5.4.0.8, so it's not a surprise, but it did help by upgrading both units.
Hi Blake. I think your wright about the OS version, I probably should have had the DD660 upgraded before getting too deep in this replication.
Fortunately, as the replication progressed, the displayed estimated completion time started to fall. It actually completed in about eight days as opposed to the two weeks it originally estimated.
With the DD660 now offline, I will be getting the OS of my remaining DDs upgraded and in sync.
bryan_washburn
42 Posts
0
December 17th, 2014 08:00
Hi Jonathan. Thanks for your input on this.
The destination DD is already hosting data from other production environments, so I’m not able to do the Collection Replication.
I previously verified that the replication traffic was going over the private network a few times. But that command was an interesting view of each port utilization.
The private network was setup as a round-robin aggregation of three 1gb links. Per the various port monitoring options, all ports were evenly supporting the total traffic passed to the veth port. Other network configurations were tried, to the point of the current configuration of a single direct link between the two DDs with no network switch. The 1gb Ethernet ports are auto-sensing, and don’t require a cross-over cable. I probably should have tried this sooner. In any case, there has been no difference to the replication rate or the amount of traffic being passed. I think I have ruled out any network connectivity and configuration issues at this point.
You may be on to something with the deduplication suggestion though. I am doing some replication performance testing with various data types, and nothing that has been on either DD before. I’ll let you know what I see.
Thanks.
bryan_washburn
42 Posts
0
December 18th, 2014 08:00
Hi rjgreg. Thanks for the suggestion.
I had considered the OS version differences, but per the Data Domain Administrators Guide, Replication should work find provided the two DDs are within two revision codes of each other, which I believe these are. A review of these DDOS versions found no known issues with the replication feature. If I had more time to work on this, I suppose I would do an upgrade anyway, but having to work other tasks, I am not likely going to get to that any time soon.
Thanks anyway.
bryan_washburn
42 Posts
0
December 18th, 2014 08:00
I did some CIFS Replication performance testing with a variety of data types, to include typical misc. text based documents files, large videos, virtual machine images, ISO images, pre-compressed files, etc…
The majority of these file types replicated relatively quickly. So I believe the network and DD Replication configurations are fine. I suspect that the slow performance may just be with this particular data.
I recently learned that Data Domains are not able to process encrypted and/or pre-compressed data as efficiently as other data. The data I am replicating is the backup files from our SourceOne server that backs up our MS Exchange mail servers. So I suspect this may be the case, but I am awaiting confirmation from our system admin who manages those environments.
In any case, I don’t believe there is anything more that I can do but to just let it ride as-is and wait it out.
Thank you all for your feedback and suggestions. They definitely helped.
Bryan
Blake_Roberts
14 Posts
0
December 24th, 2014 06:00
This sounds very similar to what I am seeing. I have been seeding data from one DD990 to another DD990 with a 10Gbit NIC on both sides for the replication traffic over CIFS. I have 3 different replication contexts going. One is Oracle datapump/export type traffic. Another is RMAN based backups (archive log and full backups), and the 3rd is SQL Server backups (tlogs and full backups). The SQL Server backups are loaded on a CIFS share. Typically I am seeing about 15MB/s running on the SQL replication context where I see 90-120MB/s on the RMAN replication. I really do think it's an artifact of the type of data it is.
On a side note, my replication performance increased when I upgraded to 5.4.4.2. I was on 5.4.0.8, so it's not a surprise, but it did help by upgrading both units.
bryan_washburn
42 Posts
0
December 29th, 2014 07:00
Hi Blake. I think your wright about the OS version, I probably should have had the DD660 upgraded before getting too deep in this replication.
Fortunately, as the replication progressed, the displayed estimated completion time started to fall. It actually completed in about eight days as opposed to the two weeks it originally estimated.
With the DD660 now offline, I will be getting the OS of my remaining DDs upgraded and in sync.
Thanks for the feedback.