Unsolved
This post is more than 5 years old
16 Posts
0
1636
Even though space was available why ISILON hdfs replying to the clients saying that QUOTA EXCEEDED? Later we added 1T to the limit. Everything started working. Please descriptiion and details below.
isi quota quotas view /ifs/is4/HDFS/elz_uat directory
Path: /ifs/is4/HDFS/elz_uat
Type: directory
Snapshots: No
Thresholds Include Overhead: No
Usage
Files : 239705
With Overhead : 5.45T
W/O Overhead : 4.50T
Over: -
Enforced: Yes
Container: Yes
Linked: -
Thresholds
Hard Threshold : 10.00T
Hard Exceeded : No
Hard Last Exceeded : 1969-12-31T18:00:00
Advisory Threshold : -
Advisory Exceeded : No
Advisory Last Exceeded : -
Soft Threshold : -
Soft Exceeded : No
Soft Last Exceeded : -
Soft Grace : -
We had 10T hard limit on our hdfs root. When I ran
hdfs dfsadmin -report
it said 100% used up.
I have cleaned up close to 2T space on hdfs.
Still ISILON hdfs keeps replying back to the client saying Disk quota exceeded. But in fact spqace was available.
Error in the logs:
is4-1(id1) isi_hdfs_d: open() (for create RPC) on `/tmp/hadoop-yarn/staging/pcenteru/.staging/job_1422987739140_2873/job_1422987739140_2873_1.jhist' failed: Disk quota exceeded
Error to the client:
------------------------------
2015-02-06 06:18:01 INFO: Hadoop_Native_Log :ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:pcenteru (auth:SIMPLE) cause:org.apache.hadoop.hdfs.protocol.DSQuotaExceededException: The DiskSpace quota of /tmp/hadoop-yarn/staging/pcenteru/.staging/job_1422987739140_2996/files/Read_TOA_ACCOUNT_DELTA_SUPPL_MAPPING_632694103933462_632695441942222.bin is exceeded: quota=10995116277760 diskspace consumed=10995184998077
Later admin added 1T to the limit, things started working.
isi quota quotas view /ifs/is4/HDFS/elz_uat directory
Path: /ifs/is4/HDFS/elz_uat
Type: directory
Snapshots: No
Thresholds Include Overhead: No
Usage
Files : 239705
With Overhead : 5.45T
W/O Overhead : 4.50T
Over: -
Enforced: Yes
Container: Yes
Linked: -
Thresholds
Hard Threshold : 11.00T
Hard Exceeded : No
Hard Last Exceeded : 1969-12-31T18:00:00
Advisory Threshold : -
Advisory Exceeded : No
Advisory Last Exceeded : -
Soft Threshold : -
Soft Exceeded : No
Soft Last Exceeded : -
Soft Grace : -
Even though space was available why ISILON hdfs replying to the clients saying that QUOTA EXCEEDED? Later we added 1T to the limit. Everything started working.
Could you please explain this error and behavior?
Thanks
Ashok
chughh
122 Posts
1
February 10th, 2015 04:00
Hello,
You need to check quota applied on path /ifs/is4/HDFS/elz_uat if quota was set with include overhead then it could be reason for the alert.
Thanks
cadiletta
106 Posts
1
February 10th, 2015 09:00
The quote output provided appears healthy and doesn't look like this was the source of the exceeded quota message. Per chughh's comment the output indicates overhead is not included, and the size shows either 4.5 or 5.4T of space either way, so about 50% usage. So it is unclear if the increase to 11T of this quota actually resolved things, or just coincided with some cleanup that cleared the alert.
From the error message:
The DiskSpace quota of /tmp/hadoop-yarn/staging/pcenteru
This seems to indicate there is another quota that is being violated here, and to Dodgy1's comment, the hadoop job is tossing around some temp files as part of the job that is exceeding a quota that probably shouldn't be there.
Are there other directory quotas set?
Anonymous
5 Practitioner
5 Practitioner
•
274.2K Posts
1
February 10th, 2015 09:00
Hi,
Do you have any "sub" quotas set for the Hadoop /tmp directory or lower sub directories?
regards