This post is more than 5 years old
5 Practitioner
•
274.2K Posts
1
1605
Using Isilon as a Hadoop distCP target
I have a customer that is using the Hadoop utility distCP (distributed copy) to move data from one Hadoop cluster to another. That got me thinking that if Isilon can be a distCP target, then we might have uncovered another Isilon and Hadoop use case.
So the question is; Can Isilon be an effective Hadoop distCP target?
Anonymous
5 Practitioner
5 Practitioner
•
274.2K Posts
0
June 28th, 2013 14:00
Yes, Isilon can be an effective Hadoop distCP target!
Thanks to Andy Pernsteiner for doing the tests to confirm this. Andy reports that depending on the file size, you should see decent write throughput (around 350MB/sec/node). Andy mentioned that his bottleneck could have been the direct attached storage his Hadoop cluster was using or something else, but at least we know that it works and that it should perform relatively well.
See Andy's notes on his distCP test here:
http://one.emc.com/clearspace/blogs/andypern/2013/06/07/random-hadoop-tidbits
Natchanon.p
5 Posts
0
December 12th, 2021 21:00
Anyone has the new link because i tried to access the same link but it seem that link already gone.
DELL-Josh Cr
Moderator
Moderator
•
8.7K Posts
0
December 13th, 2021 10:00
Hi,
Here is some other documentation on doing it. https://dell.to/3DQv5gG
https://dell.to/3yn3WAK