UNSOLVED

NaagTheOne

updated

10 years ago

0

2216

April 26th, 2017 18:00

SyncIQ Offline Seeding

Hi all,

We're trying to create an offsite copy from an Isilon in a Remote Location to another Isilon at a Data Center. The remote location has very little bandwidth and it will probably take months to complete the baseline copy.

Netapp has SMTape option to seed the remote site offline. I'm relatively new to Isilon and was wondering if SyncIQ has any such feature.

Thanks in advance!

  • crklosterman

    450 Posts

    1371

    2

    Posted April 27th, 2017 06:00

    VNX has a similar option where you can do tape-out when creating a replicator session.  In the industry, it's sometimes referred to as "tape silvering". 

    That said, I don't believe that SyncIQ has such an option (unless it's been added recently).  Why?  Because low-bandwidth, high-latency links are far less common today between datacenters than they used to be.  That said there are other ways to get the data there, and you can tell SyncIQ (in the CLI only) that the target directory exists already.  Be aware though that it'll have to do a treewalk for the first full.

    ~Chris Klosterman

    Principal SE, Datadobi

    chris.klosterman@datadobi.com

  • 1371

    0

    Posted April 27th, 2017 14:00

    Thanks for the response Chris. That is exactly what we are looking for. Will you be able to point me to some sort of documentation or guidelines on how to accomplish that?

  • sluetze

    2 Intern

    •

    300 Posts

    1371

    1

    Posted April 28th, 2017 01:00

    maybe an alternative way is to move the isilon (if not needed elseway) from remote to the Location of the current isilon, sync it, move the Isilon back again.

    also if the bandwidth is that bad, you should calculate the replication-traffic and if you can ever Keep the copys in sync over that link.

    If not: don't sync the isilons. search for alternatives.

  • crklosterman

    450 Posts

    1371

    1

    Posted April 28th, 2017 06:00

    Look in the CLI documentation for your version of OneFS;  When you create the SyncIQ policy you're looking for an option like this:

            [--target-compare-initial-sync ]

    ~Chris Klosterman

    Principal SE, Datadobi

    chris.klosterman@datadobi.com

  • dynamox

    11 Legend

    •

    20419 Posts

    •

    87439 Points

    1371

    1

    Posted April 28th, 2017 14:00

    i am running one of this "target compare initial sync" job right now on a directory with 12 millions files. It's been running for 2 weeks and only 50% done. I am sure a lot depends on the backend (NL-SAS, X410) in my case but  something to consider if you have very tight RPO.

  • crklosterman

    450 Posts

    1371

    0

    Posted April 28th, 2017 15:00

    Indeed it's not for the faint-of-heart.  With a high-file-count and small average file size it can certainly be faster to just copy it all over the WAN instead.  The first pass through has to do a full treewalk and comparison on every file, so it's going to be pretty slow.

    And as sluetze  said above often times in a case like this DellEMC will deploy the DR cluster next to the production cluster to seed it, then pack it and ship it to the other side, and re-establish replication.

    A truck can move data sometimes far faster than any other method.

     

    ~Chris

  • 1371

    0

    Posted May 1st, 2017 15:00

    Thanks all for your responses. We decided to replicate over Internet VPN using NAT. Based on the speeds, it looks like we will be able to complete the baseline in 2 weeks

  • SKT2

    2 Intern

    •

    1340 Posts

    1371

    0

    Posted May 3rd, 2017 01:00

    what is the internet link speed? what is the distance between the two locations?

  • 1371

    0

    Posted May 4th, 2017 08:00

    Internet Link is 150Mbps. Source is Colorado and Destination is Los Angeles. We have about 13TB of data.

  • crklosterman

    450 Posts

    334

    2

    Posted May 5th, 2017 07:00

    13TB and 150Mbps?  not bad:

    The math works out like this:

    13 TB of Data
    13312 GB of Data
    13631488 MB of Data
    150 Link Speed in Mbps
    18.75 Link Speed in MBps
    727012.6933 Seconds for first copy at line rate
    12116.87822 Minutes for first copy at line rate
    201.9479704 Hours for first copy at line rate
    8.414498765 Days for first copy at line rate

    So that's at line rate.  Now maybe factoring in the latency let's say it's only 1/2 that good.  That's just 17 days for the first full, then your first incremental or two will have a bunch of data to copy (depending on your change rate), but the changes are block level based upon the difference between 2 snapshots, and there is no tree walk required.  So all in all not too bad.  After the first incremental or two then you settle into a pretty steady rhythm.  One thing to keep in mind is that you should think carefully about how frequently you run your SyncIQ jobs.  For instance if you're running a sync ever 2 hours and it takes let's say 90 minutes to run, is there room to make it more frequent? Most people would say no at first glance but stop and think about it for a minute longer.  If in those 90 minutes let's say it copied 10GB just for a round number.  Assuming a constant rate of change if you ran ever 1 hour instead of 2, then you'd copy only 5GB every hour, and that job in theory would only take 1/2 of the time or 45 minutes.  Want to run more frequently?  How about copying 2.5GB every 30 minutes, with a job that completes in 22.5, and so-on.  There is of course some overhead to each job, but not a ton.  So keep that in mind is all I'm saying.

    Hope this helps,

    ~Chris Klosterman

    Principal SE, Datadobi

    chris.klosterman@datadobi.com