This post is more than 5 years old
57 Posts
0
8744
How to clean an over-capacity OneFS 7.1.1 VM?
I have set up a OneFS 7.1.1 VM 3 months ago. The entire 17GB of OneFS is filling up even I have only 8.5MB data in it. How can I clean up the VM to free up some space?
This is a event I see from OneFS Web UI:
2014-12-07 21:11:46 There is at least one smartpool at or over capacity.
Checking on the disk usage, and also directory capacity, OneFS is 103% full, but I only have 8.5Mb data in it (/ifs/data).
isilon-vm-1# df -h
Filesystem Size Used Avail Capacity Mounted on
/dev/mirror/root1 989M 450M 460M 49% /
devfs 1.0K 1.0K 0B 100% /dev
/dev/mirror/var1 989M 70M 840M 8% /var
/dev/mirror/var-crash 1.9G 1.8M 1.8G 0% /var/crash
/dev/mirror/keystore 31M 10K 28M 0% /keystore
/dev/md0 62M 684K 56M 1% /tmp/ufp
OneFS 34G 17G -453M 103% /ifs
isilon-vm-1# du -ch /ifs/data | grep -i total
8.5M total
Note that there's only 17G in the VM, because I have only set up 3 nodes instead of 6 nodes.
Not sure if this matters: I once upgraded the VM from OneFS 7.1 to 7.1.1. I have deleted the tar.gz already, but I'm not sure if there's anything else I need to delete to free up space.
peglarr
99 Posts
0
December 8th, 2014 13:00
You are reserving space equivalent to one virtual drive. Uncheck that box 'deny new data writes' and (I am guessing for your tests) the 'ignore reserved space' as well.
jenny_lam
57 Posts
0
December 8th, 2014 10:00
The files in /ifs/data tend to be small, and a few big files on occasion. I suppose not many space are used for 8.1MB of data even if they are mirrored? Is there a way to find out where all the 17GB are occupied?
Phil.Lam
1 Rookie
1 Rookie
•
567 Posts
0
December 8th, 2014 10:00
How big are the files? Files less than 128k will be mirrored.
Phil.Lam
1 Rookie
1 Rookie
•
567 Posts
0
December 8th, 2014 11:00
try "isi stat -d -q"
Phil.Lam
1 Rookie
1 Rookie
•
567 Posts
0
December 8th, 2014 11:00
It's your virtual hot spare (VHS) taking all the space:
VHS Size: 17G
jenny_lam
57 Posts
0
December 8th, 2014 11:00
Thanks Phill, so could you help me understand what this means? Anything to concern about over-capacity then? How to free up VHS if that is a concern?
crklosterman
450 Posts
0
December 8th, 2014 11:00
Do an du -sh /ifs/* on your VM, and see what top level directory is consuming all the space. If it's in /ifs/.ifsvar then it's likely log files consuming all the space. Or on the node do a df -kh, and you'll see which filesystem is filling up.
~Chris
jenny_lam
57 Posts
0
December 8th, 2014 11:00
isilon-vm-1# isi stat -d -q
Cluster Name: isilon-vm
Cluster Health: [ ATTN]
Cluster Storage: HDD SSD Storage
Size: 17G (34G Raw) 0 (0 Raw)
VHS Size: 17G
Used: 17G (> 99%) 0 (n/a)
Avail: 0 (< 1%) 0 (n/a)
Throughput (bps) HDD Storage SSD Storage
Name Health| In Out Total| Used / Size |Used / Size
-------------------+-----+-----+-----+-----+-----------------+-----------------
iq_vmware | OK | 145K| 8.7M| 8.8M| 17G/ 17G(> 99%)|(No Storage SSDs)
-------------------+-----+-----+-----+-----+-----------------+-----------------
OK = Ok, U = Too few nodes, M = Missing drives,
D = Some nodes or drives are down, S = Some nodes or drives are smartfailed,
R = Some nodes or drives need repair
Phil.Lam
1 Rookie
1 Rookie
•
567 Posts
0
December 8th, 2014 11:00
You don’t need a VHS on a VM, for live physical boxes, yes. It preallocate space for hot spare use for physical disks if they die.
peglarr
99 Posts
0
December 8th, 2014 12:00
You can either use the GUI to completely disable the VHS, or from the CLI check out page 724-725 of the OneFS CLI Admin Guide.
isi storagepools settings modify ...
Phil.Lam
1 Rookie
1 Rookie
•
567 Posts
0
December 8th, 2014 12:00
From GUI, SmartPools->Settings->At least x % of total storage, where x=0.
jenny_lam
57 Posts
0
December 8th, 2014 12:00
I don't know how to disable VHS. The SmartPools settings is the default after the VM is installed. Any idea?
isilon-vm-1# isi storagepool settings view
Automatically Manage Protection: files_at_default
Automatically Manage Io Optimization: files_at_default
Protect Directories One Level Higher: Yes
Global Namespace Acceleration: disabled
Virtual Hot Spare Deny Writes: Yes
Virtual Hot Spare Hide Spare: Yes
Virtual Hot Spare Limit Drives: 1
Virtual Hot Spare Limit Percent: 0
Global Spillover: anywhere
SSD L3 Cache Default Enabled: Yes
jenny_lam
57 Posts
0
December 8th, 2014 13:00
It's has always been set to 0%... Is there any other configuration to disable it completely? And is there any way to reclaim spaces occupied by VHS?
isilon-vm-1# isi storagepool settings view
Automatically Manage Protection: files_at_default
Automatically Manage Io Optimization: files_at_default
Protect Directories One Level Higher: Yes
Global Namespace Acceleration: disabled
Virtual Hot Spare Deny Writes: Yes
Virtual Hot Spare Hide Spare: Yes
Virtual Hot Spare Limit Drives: 1
Virtual Hot Spare Limit Percent: 0
Global Spillover: anywhere
SSD L3 Cache Default Enabled: Yes
Anonymous User
170 Posts
1
December 8th, 2014 14:00
Don't forget about /ifs/.ifsvar & /ifs/.isilon. These directories can accumulate cruft and OneFS doesn't clean them up very well. There are critical configuration files in there, so don't go blasting stuff away unless you know what you're doing.
jenny_lam
57 Posts
0
December 11th, 2014 16:00
Hello, I have to come back again, for disabling didn't resolve it. The disk is now completely full. I need help restoring the system to a working state.
FYI, I have already disabled VHS, and even rebooted the VM afterward. I never make any snapshot, etc. All configuration settings have been the "DEFAULT" except for the VHS setting I have changed here.
Two things I have noticed:
isilon-vm-1# du -sh /
5.2G /
isilon-vm-1# du -sh /ifs/.ifsvar/modules/stats
4.5G /ifs/.ifsvar/modules/stats
isilon-vm-1# ls -l /ifs/.ifsvar/modules/stats
total 4763099
-rw-r----- 1 root wheel 1623195648 Dec 11 16:24 persistence.db
-rw-r----- 1 root wheel 935312 Dec 11 16:24 persistence.db-journal