Unsolved
This post is more than 5 years old
1 Rookie
•
79 Posts
0
1488
November 19th, 2018 13:00
virtual cluster space issues
I have a virtual cluster that is running out of space and no matter what I clean, I am not regaining the space
# isi stat
Cluster Name: isilon-virtual
Cluster Health: [ ATTN]
Cluster Storage: HDD SSD Storage
Size: 32.1G (56.2G Raw) 0 (0 Raw)
VHS Size: 24.1G
Used: 36.7G (> 99%) 0 (n/a)
Avail: -4.6G (< 1%) 0 (n/a)
Here is what I see under /ifs
# du -sh /ifs/data
3.7G /ifs/data
# du -sh /ifs/.ifsvar
12G /ifs/.ifsvar
I deleted all snapshots but I still can't reclaim anything on it. This cluster used to be a target for another virtual cluster that we deleted because of vmware issues.Rebooted the cluster just in case there are process hung, no luck.
I ran multiscan but it keeps failing on it.
Any ideas where all the space is going?
Thanks
-Dan
0 events found
sjones51
252 Posts
0
November 20th, 2018 11:00
Hi Dan,
Couple questions:
What version of OneFS?
Did you run snapshotdelete after deleting the snaps?
Dtek1
1 Rookie
•
79 Posts
0
November 20th, 2018 13:00
It is on v8.1.0.4.
Yes, I have run SnapshotDelete
# isi job status | grep -i snapshot
10439 SnapshotDelete Succeeded 2018-11-19T16:20:47
#
Thank you
sjones51
252 Posts
0
November 21st, 2018 10:00
Thanks Dan.
There are a couple things that catch my attention based on what you posted. I see that /ifs/.ifsvar is taking up a good chunk of space compared to what is available. I question whether the issue is something in that directory filling up because of a problem, or if that directory is a normal size and there just isn't enough space. The last time I looked, I believe that virtual clusters required a minimum of 42G per node and minimum of three nodes. I don't know if those requirements changed with OneFS 8.x.
Is there one file or directory taking up the bulk of /ifs/.ifsvar?
Dtek1
1 Rookie
•
79 Posts
0
November 26th, 2018 09:00
Thank you for your response.
I check .ifsvar
# du -sh /ifs/.ifsvar
12G /ifs/.ifsvar
#
and these are the two directories that are taking most of the space
# du -sh modules
9.0G modules
# du -sh upgrade
2.1G upgrade
#
Not sure if we can clean up these directories.
You brought up a good point about the required 42GB per virtual node but this cluster was on 8.0.0.4 and upgraded to 8.1.0.4 and I believe the space requirement is the same for both versions. We started experiencing this issue after we upgrade to 8.1.
So is our option now to destroy this and re-create a new virtual cluster using the 8.1 image or if we allocate more space on the VMware side, will the cluster see the new allocated space?
Thanks
-Dan