Highlighted
2 Bronze

Partially Cloudpooled data migration to S3

Hi All,


We are a big user of on prem ECS, (cloudpool target) and I have a particular scenario I'm hoping can be solved.


We have an Isilon share (SMB) which is approx 600TB - 50TB hot and the rest on cloudpools.
There is a requirement to move this data elsewhere (AWS S3) for a project.


I did a sample Robocopy test on the CP'ed data and as feared second copies were not diffs this was not a surprise as robocopy isn't stub aware but still I figured the timestamp and size were the same. I also seen on here perhaps EMCOPY or DataDobi may help?

 

If not the other solution we thought about:
Perform a simple continuous SyncIQ deepcopy locally to another area on the source Isilon to rehydrate the data so its full fat. We can this use this to feed the S3 bucket. However we do not have the space locally. We are now looking to add some spare HD400 nodes to the cluster so we can perform this work.

Question is really, how can we copy data that is partially cloudpooled so diff copies only copy changed files?

Thanks!!

0 Kudos
5 Replies
Highlighted
2 Bronze

Re: Partially Cloudpooled data migration to S3

I tried robocoping (diff) for a sample set of data for a second time, and the smartlinks weren't skipped they get copied in full even though they haven't changed 😞

0 Kudos
Highlighted
Moderator
Moderator

Re: Partially Cloudpooled data migration to S3

Hello DarrenBra,

Here is the link to Isilon OneFS  CloudPools Administration Guide. https://dell.to/2yFynY9

Have you looked into OneFS to do the migration?

Please let us know if you have any other questions.

DELL-Sam L
Dell | Social Outreach Services - Enterprise
Download the Dell Quick Resource Locator app today to access PowerEdge support content on your mobile device! (iOS, Android, Windows)

0 Kudos
Highlighted
2 Bronze

Re: Partially Cloudpooled data migration to S3

thanks for the reply, i've read the doco much appreciated

0 Kudos
Highlighted
4 Beryllium

Re: Partially Cloudpooled data migration to S3

As long as the robocopy or whatever sync tools has to rely on reading the full file content to detect any differences, one would be out of luck here. This has nothing to do with OneFS CloudPools, which are completely invisible to the SMB or NFS client.

Shortcuts often used are the "archive bit" with genuine Windows systems, and the regular timestamp of last file modification.

Idealy the sync tool would maintain the source modification timestamp on the sync target, i.e. override any timestamp that the target storage creates for the sync copy itself. Check wether this can be realised in your environment. Also the sync tool must to configured to assume source and target files as identical and skip copying when the timestamps match. 

hth

-- Peter

0 Kudos
Highlighted
2 Bronze

Re: Partially Cloudpooled data migration to S3

thanks peter this reallys help to understand why I am seeing this behaviour. We are looking at forthcoming native cloud rep tool for isilon and superna software that is smartlink aware, cheers!

0 Kudos