chjatwork ,

You're certainly right, rsync can move data, but it can't give you a chain-of-custody level report with md5 hashes for instance for every file source to target, (including permissions).  Is this a 1-time move/migration or an on-going synchronization?  In either case my companies' products can fit the need.  Reach out of you want to chat about it offline sometime.

~Chris Klosterman

Principal SE, Datadobi

www.datadobi.com

chris.klosterman@datadobi.com

Chris K.,

As much as I would love to purchase your tool.  I will most likely be restricted to finding a solution that is home brewed.  I just sort of wanted to find out what methods others admins are using to solve this issue? 

Thank you,

Cross your fingers, find a four-leaf clover, hope and pray, etc.  That's about it, only 1/2 joking. 

BeyondCompare I've seen used from time to time, but it's still commercial software you have to pay for.

Rsync has a --checksum option specified in the man page, but be prepared for let's say less-than-stellar performance.  Like emcopy's --cm md5 option it'll hash every single file on every single pass.  That's why the question is important of is this a 1-time thing you need to do, or ongoing.

~Chris

On the isilon ssh command line you could create a hash tree with something like `find /path/ -type f -exec md5 {} \; > sums.txt` but you would need a method to compare that to the source.

If you want to migrate from one cluster to another i would use SyncIQ. Then you:

- have the integrity-checks integrated

- don't need external servers for configuration

Chris K.,

This would be for anytime we need to move data from one cluster to another to possibly create space on a cluster that maybe low on space.  So not very often, but it happens from time to time as users run into quota issues.

Eric,

so once I get the sums report from the source and the destination I would simply just run a diff against both files?

Sluetze,

good point, but if we archive to any other brand of storage like outside of ECS or Isilon I want to make sure I know what the industries best practices are.

In a simple case, yes, but in reality you may need to massage and sort the output to make the two platforms produce identical output.  It's not a hard problem but it could end up being non-trivial if the platform is not linux/bsd.

There exists another way to transfer data in bulk but I would not recommend it to the non-savvy.  If you have a means of using NDMP it's just a pax/tar stream with some additional records.  YMMV offer not valid in all 50 states you must test, but the stream does contain all the file data and it can easily stream at 700MB/s if your receiver can handle it.