NFS data copy tool - rsync and what other open source tools ??

if source and destination are non-emc production why are posting on EMC forum ?

Hi Dynamox,

I would have to migrate a lot of data and one of the sources is Isilon

i've always used rsync, if directory structure allows run multipel rsync instances at the same time.

I use rsync but frequently wrap it in scripts after looking at the source directory tree to see what parallelization I can apply.

I frequently use the GNU parallel tool to fire off multiple streams, sometimes on multiple hosts.  I can push a lot of data fairly quickly.

Thanks Chughh and Ed

Considering ideal resource conditions, are tar, cpio and GNU faster than rsync in copying data

There are even more options:

Use Hadoop distcp if both sides speak HDFS:

Backing Up Hadoop To Isilon

And google for parsyncfp, a wrapper for running rsync in parallel.

-- Peter

Thanks again for your responses.

I still haven't worked on GNU parallel and i am just looking for other tools which can copy data faster than rsync, although rsync is reliable and i have been using it from a long time

I tried tar and cpio. I can't find much of parameters for cpio for incremental copies but tar has some features for incremental copies and it resulted to be faster than rsync. The only problem i can find at this point is that tar is not target aware. If we delete files on target directory and run an incremental copy using the snapshot file created in previous run, it wont copy the deleted file to target though it is present in source as it is just looking at the snapshot.