This post is more than 5 years old
16 Posts
0
3582
ISILON hdfs - fs.trash.interval? How does the trash cleaninig up process work on ISILON HDFS
ISILON hdfs - fs.trash.interval? How does the trash cleaninig up process work on ISILON HDFS
Does ISILON hdfs has some configuration files of ite own or does it work out of core-site.xml or core-default.xml?
I do not see on ISILON hdfs
/user/XYZ/.Trash/Current not getting deleted.
Any insight into this process.
Thanks for your help and input.
Ashok
russ_stevenson_
25 Posts
0
February 10th, 2015 10:00
Is the issue here that files in the .Trash are not getting deleted?
But yes, the fs.trash.interval is in the core-site.xml,
fs.trash.interval
360
if you set the value to 0 files should not be moved to .Trash and no cleanup is needed.
[yarn@hdp4 conf]$ hadoop fs -rm /user/yarn/test.txt
15/02/10 10:19:53 INFO fs.TrashPolicyDefault: Namenode trash configuration: Deletion interval = 0 minutes, Emptier interval = 0 minutes.
Deleted /user/yarn/test.txt
or use the -skipTrash option
Also remember since you can access the Isilon file system outside of HDFS and without hdfs tools, you can delete files using normal methods, these will not be moved to trash either.
russ
abeesam
16 Posts
0
February 10th, 2015 10:00
Hi Russ,
Is this core-site.xml on ISILON nodes somewhere? If yes where exactly can I find the configurations for hdfs on ISILON.
Compute cluster configuration, I can set those but I am not sure on ISILON which runs datanode/namenode services.
Thanks
for your time and help.
Ashok
russ_stevenson_
25 Posts
1
February 10th, 2015 12:00
hey Ashok,
The core-site.xml is not on the Isilon cluster, it will be found on all hadoop nodes in the compute cluster. Location of the core-site.xml on each node will be specific to the specific hadoop distribution you are using and management of file may be under the hadoop cluster manager if in use (pivotal, cloudera manager or ambari and /or zookeeper). But it will need updating on each host in the hadoop cluster.
Yes, the cluster is acting as NN, SN & DN but it's not running the HDFS services in the same way as a native hadoop cluster would, the core-site.xml on each client will be honored for configuration and operation of the host and we use core-site.xml to tell each host where the NN is for each resource and service it needs, aka the Isilon, go there for NN, SN & DN services.
Hope this helps
russ
abeesam
16 Posts
0
February 10th, 2015 12:00
Currently :
Compute cluster core-site.xml configured as
fs.trash.interval
1440
Number of minutes after which the checkpoint
gets deleted. If zero, the trash feature is disabled.
This option may be configured both on the server and the
client. If trash is disabled server side then the client
side configuration is checked. If trash is enabled on the
server side then the value configured on the server is
used and the client configuration value is ignored.
Even then
Trash is not getting cleaned on ISILON hdfs side even after 24 hours.
Should we configure similar things on ISILON server nodes. Please advice.
russ_stevenson_
25 Posts
1
February 10th, 2015 13:00
Make the change to core-site.xml and deploy to all clients. You'll need to restart hdfs, yarn & mr2 services and potentially other services you use so this change is picked up.
Once you have completed this, when deleting a file, you will no longer see the file move to .Trash
[hdfs@cdhcm hadoop-mapreduce]$ hadoop fs -ls /user/hdfs/
Found 3 items
drwxrwxrwx - hdfs hadoop 25 2015-02-10 15:33 /user/hdfs/.Trash
drwx------ - hdfs hadoop 0 2015-01-20 15:43 /user/hdfs/.staging
-rw-r--r-- 1 hdfs hadoop 0 2015-02-10 15:54 /user/hdfs/3.txt
[hdfs@cdhcm hadoop-mapreduce]$ hadoop fs -rm /user/hdfs/3.txt
15/02/10 15:56:29 INFO fs.TrashPolicyDefault: Namenode trash configuration: Deletion interval = 0 minutes, Emptier interval = 0 minutes.
Deleted /user/hdfs/3.txt
[hdfs@cdhcm hadoop-mapreduce]$ hadoop fs -ls /user/hdfs/
Found 2 items
drwxrwxrwx - hdfs hadoop 25 2015-02-10 15:33 /user/hdfs/.Trash
drwx------ - hdfs hadoop 0 2015-01-20 15:43 /user/hdfs/.staging
[hdfs@cdhcm hadoop-mapreduce]$ hadoop fs -ls /user/hdfs/.Trash/Current/user/hdfs
Found 1 items
-rw-r--r-- 1 hdfs hadoop 0 2015-02-10 15:59 /user/hdfs/.Trash/Current/user/hdfs/1.txt