How to clean an over-capacity OneFS 7.1.1 VM?

Just for clarity: root / includes the OS and /ifs, so you might want to du /ifs.

But it won't help to find those 17 GB...

One important aspect for "dark matter" issues:

When did the most recent MultiScan or Collect

jobs run successfully? These jobs free orphaned blocks,

which accumulate while individual nodes

are down during /ifs file system activity.

hth

-- Peter

Time for a new OneFS 7.2 VM?

Phil Lam

The jobs usually do run on virtual nodes!

Can you add three more nodes and try again?

(Did you double check for snapshots?

SyncIQ and NDMP create snapshots, too.)

-- Peter

The jobs don't "usually" run on virtual nodes, they _do_ run.

This cluster is best destroyed and a new virtual cluster created.  You'll save far more time and energy starting over than trying to get this cluster operating normally.  Notice your /var/crash partition is full, too, meaning you've crashed virtual nodes (often)  - and just like their real counterparts, running a cluster with /var/crash at 0% available is poor practice.

Not 100% sure how you ended up in this state - but again the VM is designed for quick buildup/teardown, so I believe your best bet is to create a new cluster, since you have so little data on it anyway.

BTW I believe it makes zero sense to upgrade a virtual node.  Just destroy the cluster and spin up the new version.  I do this in my homelab every release.

Hi Rob

"Capacity 0%" is "used", not "left" amount -- I certainly dislike the df header line

As for upgrading virtual nodes, I respectfully disagree. Training, and

testing upgrades with complex custom configs make great cases

according to my experience.

Yes, the virtual nodes are not supported, so the outcome is not guaranteed.

But in lieu of a physical test cluster, virtual nodes are better than nothing.

For rigorous testing of specific OneFS versions,

fresh installs are best suited, of course!

I also believe that successfully maneuvering out of

an overloaded /ifs situation will be highly instructive,

though of course getting there should be avoided

by all means on a production cluster.

Cheers

-- Peter

Thanks Peter, I did indeed invert the columns, so /var/crash is not an issue.  Please forgive me.

Anyway, I totally agree that using the VM for dev/test is appropriate - that's why we designed and built/issued it in the first place.

But upgrading a virtual node is not at all like upgrading a physical cluster - since there is no IB fabric, and IB plays such an important role in inter-cluster communication.  It's useful to test upgrades using VMs, but only to a point.

But the original question was why is my /ifs so full.  Ed Wilts nailed it - clusters (real or virtual) accumulate data in /ifs for internal purposes as well as persisting user data.  In this case, the statistics database is quite large, which tells me this virtual cluster has been running for quite some time with workload.

Since this is a virtual cluster, there is no harm in either a) scaling out (adding nodes to increase /ifs capacity) or b) tearing it down and starting again with a clean cluster.

So while it's handy to 'test' upgrades with (say) 7.1 to 7.1.1 via upgrade VM, the disk sizes are so small in the VM that this scenario where the stats D/B consumes nearly all space can easily happen, where as it is extremely rare to see this in a real cluster with real drives.

Besides, in a VM scenario, if you really want to remove the persistence information, be careful what you ask for - go ahead, remove it, then kill the isi_stats_hist_d daemon. It's just a VM, after all :-)

The cluster is only for development and testing, and I didn't make any snapshot, replication, etc. I was hoping some quick resolution to free up some space, but looks like it is not ending that way.

I will try to add 3 more nodes (there's only 3 nodes on this cluster now).  But likely I will ask the same question again ~3 months later if the cluster is filling up in the same speed (hope not)...

Thanks for all the helps you have offered!

I had the same issue on my VM and couldn't figure how to clear off the space. What did it for me was running the Collect job. I'm running a 7.2 VM.

isi job jobs start collect

Here's what the CLI guide says about collect.

Reclaims free space that previously could not be freed because the node or drive was unavailable. Run as part of MultiScan, or automatically by the system if MultiScan is disabled.

Greg