UNSOLVED

bryan_washburn

updated

12 years ago

0

5948

December 15th, 2014 10:00

Data Domain Initial Directory Replication Speed?

Relatively new to Data Domain administration, so apologies if this lacks information.

To decommission our DD660, I need to move an 8TB (compressed) CIFS directory to a DD670 that is on the same network. Referring to the Data Domain Administration Guide, I prepared a CIFS directory replication session between two DDs on the same local network. I have also redirected the replication traffic over a private network of three aggregated 1Gb links.

Source: DD660 5.2.2.4-370754

Source directory is off of /backup/

Source Date is 8 TBs compressed, and I believe around 25TBs uncompressed.

Destination: DD670 5.4.1.1-411752

Despite this high bandwidth private network connection, the data transfer rate for the initial data copy process appears to be significantly slower then what I would have expected. I have been referring to the 'replication watch' command and the Stats graph view in Enterprise Manager to monitor progress and activity.

The Replication Summery view shows all references for time completion as unknown. About 16 hours into the replication, the 'replication watch' command indicates only 10% of the files have been copied. Assuming this maintained rate, the initial copy will take nearly seven days to complete.

Pet the Status / Stats graphs on source DD:

  • Disk read averages between 12 and 28 MiB/s

  • Private network ports averaging between 1 and 5 MiB/s per port

  • Replication peaks and valleys between 2 and 12 MiB/s

  • The CPU activity for both DDs is relatively light

From the DD Admin Guide and internet searches, I was not able to get a clear understanding of how fast an initial transfer for most replication types should take between two DDs on the same network. However, a few articles indicate transfer rates much faster then what I am seeing.

Some online resources have referred to ensuring the compression type is the same on both source and destination. I have not been able to find this information on the DDs. Other sources discuss increasing streams as well as converting the directory to an Mtree, then performing an Mtree replication. I have not had time to look into these further.

I have reviewed the recent article Replication Sizing Guide (https://community.emc.com/docs/DOC-40889). I have no concerns with the discussed items, but this does support my belief that I should be seeing better performance on this replication.

I would appreciate some feedback on what kind of replication performance I should actually be seeing.

If possible, how might I improve the performance.

Is there a more effective way to copy the CIFS directory data to the destination DD?

Thank you very much,

Bryan

  • 2194

    1

    Posted December 15th, 2014 10:00

    Thank you, rprnairj.

    Both DDs appear to be set as LZ compression.

  • rprnairj

    1 Rookie

    •

    45 Posts

    2194

    0

    Posted December 15th, 2014 10:00

    Just wanted to share,

    "Some online resources have referred to ensuring the compression type is the same on both source and destination. I have not been able to find this information on the DDs."


    This information can be seen on the enterprise manager, Data Management>filesystem>configuration.

  • rprnairj

    1 Rookie

    •

    45 Posts

    2194

    0

    Posted December 15th, 2014 11:00

    Also, as you mentioned, that you have redirected the replication traffic, you see traffic moving on that specific interface?

  • rprnairj

    1 Rookie

    •

    45 Posts

    2194

    0

    Posted December 15th, 2014 11:00

    what is your stream utilization on both source and destination data domain,

    you can check this using #sys sh perf

  • rprnairj

    1 Rookie

    •

    45 Posts

    2194

    0

    Posted December 15th, 2014 12:00

    it would be under rd/wr/r+/w+

  • 2194

    0

    Posted December 15th, 2014 12:00

    Yes, traffic is passing through each of the three aggregated ports at bot ends.

    However, a coworker suggested switching to LACP from round robin.

    I will give that a shot shortly.

  • 2194

    0

    Posted December 15th, 2014 12:00

    Sorry, I am not sure which columns of information you are looking for.

    I have collected the output of the last few hours from each DD to a log.

    Trying to post it.

    1 Attachment

  • 2194

    0

    Posted December 15th, 2014 13:00

    No difference when changing the aggregate protocol.

    However, I just learned there may be a problem with the switch that I am using.

    I will let you know.

  • 2194

    0

    Posted December 15th, 2014 13:00

    For the last entries of each:

    Source shows 0/ 1/ 0/ 0

    Dest, shows 0/ 0/ 0/ 0

    A sys perf output file is posted. You may want to look at that.

  • dynamox

    11 Legend

    •

    20419 Posts

    •

    87439 Points

    1147

    0

    Posted December 16th, 2014 11:00

    i am not sure what kind of data you replicate but when i replicate Oracle RMAN backups, i get around 150MB/s from DD880 to DD890 (10G interfaces) on layer 2 network.