Isilon Snapshots from other clusters.

Something that might be worth exploring / trying out.

Assuming you are on OneFS 7.1.x.x and up you could create a new policy with a 'manual' schedule and sync the data based on a source snapshot.  Then go on the target cluster and take a snapshot after the sync is successfully completed with the appropriate name and expiry date.  Repeat the sync but with the next snapshot and take a snapshot on the target cluster... rinse and repeat.

This is tedious though and if you skip/forget a snapshot you can't go back to a previous one.  You also have to ensure the sync completed successfully before you take the snapshot on the target cluster.

So how this works:

You create your new SyncIQ policy like you normally do but don’t specify a schedule since you will want to run it manually to rebuild all the snapshot history.

Example: isi sync policies create --action=sync --name=ifs_data_nfs --source-root-path=/ifs/data/nfs --target-host=static.vm7105c.yfisi.local --target-path=/ifs/R/yfvm-7111/data/nfs --target-snapshot-archive=true

Assumption is target cluster has no data in the specified ‘--target-path’.

The flag ‘--target-snapshot-archive=true’ is to say I want to automatically create a snapshot on the target post successful sync.  In your case might want to set this to false and manually take the snapshot on the target after each successful sync with the correct name and expiry date.

You start your first run but you specify which source snapshot (by name or by id) you want to use.  You would use your oldest snapshot. Example: isi sync jobs start --policy-name=ifs_data_nfs --source-snapshot=ifs_data_initial

There’s a restriction though to prevent the user from overwriting data based on an old snapshot.  If I try again to run this policy with the same source snapshot (or an older one), I’ll get this error:

yfvm-7111-2# isi sync jobs start --policy-name=ifs_data_nfs --source-snapshot=ifs_data_initial

Sync policy error: Manually specified snapshot (snapid 136) is the same or older than snapshot (snapid 136) used for previous sync

EDIT: The source-root-path of your SyncIQ policy must match the path of the snapshot.  As an example, you can't use this approach if your snapshot was taken on /ifs/data/foo and your policy is on /ifs/data/foo/nfs.

if you cannot tolerate any outage, then the isilon is the wrong product.

*duck and cover*

Yan's method looks cool - even though i didn't tried it. Our solution was always, to sync our syncs/copies with the snapshot schedule (so the sync finished by the time the snapshot is taken on the target site) and wait until the snapshots "aged out" so we have the same snapshotdata on source and target.

also in my opinion it will be very hard to implement a solid functionality to "migrate" snapshots from source to target. This is always a tinkered solution, because you just can't have identical blocks / blockmaps etc. on source and target. (in an isilon architecture)

Yan,

i don't understand how does it work. You create a policy that points to snapshot /ifs/.snapshots/filesystem/blah/9.1.2015/mydata. This is my source and i replication it to target cluster. Next i want to change to snapshot that was created on 9.2.2015, what do i do next  ? If i modify existing policy by changing the source, won't it tell me that changing the source requires full resync ?

Thanks

Brian, this is far from a push-button solution.

First mount the old snaps to the target cluster so you can use rsync on the target locally, then:

rsync -av --compare-dest=OLDSNAP  NEWSNAP DIFFSNAP

will populate DIFFSNAP with files from OLDSNAP that are not in NEWSNAP (most recent, e.g. from live data)

Work backwards towards the oldest snapshot.

Using --link-dest instead of -compare-dest with require a writable filesystem as NEWSNAP, which means starting from  live data instead of a snapshot; and will create a full tree in DIFFSNAP with hardlinks into NEWSNAP. Easier to browse, but this will modify the live data's inodes , and is prone to accidental modifications of the live data.

Neither method will be very efficient with tons of past renames or moves of files or (beware) directories.

End-users will /not/ be able to access any OLDSNAP through the .snapshot pseudo-directory, of course.

This just retains the changed and deleted files from the old snaps

in a somewhat efficient way on the new cluster -- that's the main point here.

Works as a convenience for admins, saving the need to reach out for the tapes...

-- Peter

Thanks Peter.  Is this documented somewhere?  The problem I think we're in right now is the target has all the source data but NOT the snapshots.  And since the target is now technically the "source" as it's being used for production, we have to keep the old cluster powered up for six months so we can go back and retrieve snapshots.

I think your idea is awesome, but maybe that really should be integrated into Isilon if you plan to migrate data from source to a NEW target cluster.  That way you do not have to worry about keeping the old cluster around just to retrieve old snaps.

The other problem was as the data was migrated from source the target, the folder structure changed as well.  Based on your information it would have been better to move the snaps first, run the "rsync -av --compare-dest=OLDSNAP  NEWSNAP DIFFSNAP" (all this before it's handed to production), verify the snaps are good and the data is good, then change the folder structure.  Not even sure if would work correctly but it's where my thought process is right now.

This would have been good information BEFORE we started SyncIQ....

Hi Dynamox,

You do not use the /ifs/.snapshot in this case.  You use your normal directory path when creating your SyncIQ policy.  When you manually run the policy (from CLI) you tell it which snapshot to use as the source of the data.  You can specify the snapshot by name or by id.

Example:

You have your data in /ifs/data/foo and you have some snapshots that were taken on /ifs/data/foo with these names:

ifs_data_foo_initial

ifs_data_foo_monthly_2015_07_01_02_00

ifs_data_foo_monthly_2015_08_01_02_00

ifs_data_foo_monthly_2015_09_01_02_00

So to replicate the data based on snapshot ifs_data_foo_initial I would first create my SyncIQ policy:

isi sync policies create --source-root-path=/ifs/data/foo --target-host=blah.host.name --target-path=/ifs/R/yfvm-7114s/ifs/data/foo --action=sync my_policy_name

By default it gets a 'manual' schedule which is what I want.

Then I would run this policy with my first snapshot as the data source:

isi sync job start my_policy_name --source-snapshot=ifs_data_foo_initial

To sync the delta up to the 2nd snapshot I would run again but with:

isi sync job start my_policy_name --source-snapshot=ifs_data_foo_monthly_2015_07_01_02_00

and so on.

But yes this is very tedious and not automated so it's error prone and it's not a 'block' sync so you are pretty much trying to reconstruct the snapshot history 'by hand' on the target cluster but using SyncIQ to send the data.

Brian, there is no specific documentation for using rsync with Isilon snapshots. rsync is a swiss army knife type of tool, and --compare-dest is just one of many options, with a terse yet concise description in the man page. Building a concrete solution is... left as exercise to the reader or to the Superna guys . And you have already started thinking it through. Totally agree that retro-engineering a plan for cases like yours is somewhat harder than designing it upfront.

Cheers

-- Peter

With the approach of specifying the source snapshot, perhaps it'd be possible also to use diff sync (aka target compare initial sync) to sync to the same directory on target while, at the same time, taking a snapshot on the target after each run so that you're dealing with less space used.

The downside to diff sync is the CPU usage on both clusters and the amount of time it'd take to run the syncs given that a full treewalk has to be performed.  My high-level steps come out to something like this:

1. Create sync policy and set --target-compare-initial-sync=yes from the CLI
2. Run policy once using the --source-snapshot option
3. Take snapshot on target cluster - I did this manually and didn't test having SyncIQ do it for me
4. Reset policy on source
5. Repeat steps 2-4, each time specifying a different source snapshot

A quick test here shows that this procedure works for getting data moved over and snapshots recreated on target although, like Yan's suggestion, it's very manual and possibly prone to errors.

Ah thanks, Pete. I don't use it which is why I need to read the MAN pages, lol. I'm actually working on a Superna solution right now!

@Bernie Case:  It does seem very manual but definitely open to.  Appreciate it.

Bernie,

isn't Yan method more efficient ? Your method will need to traverse entire directory structure everytime it runs where Yan's method uses the snapshot to determine what's changed and needs to be transferred ?

Thanks

dynamox,

Yes, Yan's is much more efficient. SyncIQ does keep track of the replication state so when you specify a new source snapshot, it just compares the new source snapshot with the previous snapshot used for the sync. The result is a much faster sync.

Thanks,

Bernie