Unsolved
4 Posts
0
1657
Partially Cloudpooled data migration to S3
Hi All,
We are a big user of on prem ECS, (cloudpool target) and I have a particular scenario I'm hoping can be solved.
We have an Isilon share (SMB) which is approx 600TB - 50TB hot and the rest on cloudpools.
There is a requirement to move this data elsewhere (AWS S3) for a project.
I did a sample Robocopy test on the CP'ed data and as feared second copies were not diffs this was not a surprise as robocopy isn't stub aware but still I figured the timestamp and size were the same. I also seen on here perhaps EMCOPY or DataDobi may help?
If not the other solution we thought about:
Perform a simple continuous SyncIQ deepcopy locally to another area on the source Isilon to rehydrate the data so its full fat. We can this use this to feed the S3 bucket. However we do not have the space locally. We are now looking to add some spare HD400 nodes to the cluster so we can perform this work.
Question is really, how can we copy data that is partially cloudpooled so diff copies only copy changed files?
Thanks!!
DarrenBra
4 Posts
0
April 6th, 2020 10:00
I tried robocoping (diff) for a sample set of data for a second time, and the smartlinks weren't skipped they get copied in full even though they haven't changed
DELL-Sam L
Moderator
Moderator
•
6.9K Posts
0
April 6th, 2020 16:00
Hello DarrenBra,
Here is the link to Isilon OneFS CloudPools Administration Guide. https://dell.to/2yFynY9
Have you looked into OneFS to do the migration?
Please let us know if you have any other questions.
Peter_Sero
1.2K Posts
0
April 8th, 2020 08:00
As long as the robocopy or whatever sync tools has to rely on reading the full file content to detect any differences, one would be out of luck here. This has nothing to do with OneFS CloudPools, which are completely invisible to the SMB or NFS client.
Shortcuts often used are the "archive bit" with genuine Windows systems, and the regular timestamp of last file modification.
Idealy the sync tool would maintain the source modification timestamp on the sync target, i.e. override any timestamp that the target storage creates for the sync copy itself. Check wether this can be realised in your environment. Also the sync tool must to configured to assume source and target files as identical and skip copying when the timestamps match.
hth
-- Peter
DarrenBra
4 Posts
0
April 8th, 2020 09:00
thanks for the reply, i've read the doco much appreciated
DarrenBra
4 Posts
0
April 8th, 2020 09:00
thanks peter this reallys help to understand why I am seeing this behaviour. We are looking at forthcoming native cloud rep tool for isilon and superna software that is smartlink aware, cheers!