Information on disk pool usage

Question

One of the challenges that I have been tasked with is providing a summary of the storage used by each NFS/SMB share at a node pool level. i.e. how much X400 and NL400 space does /ifs/data/Library use? This is possible by walking the entire directory tree for each directory under /ifs/data but that's also time consuming. Looking at what resources InsightIQ uses, it appears that file /ifs/.ifsvar/modules/fsa/pub/latest/results.db has a lot of useful information in a nice and small SQLite3 database file except for one thing... In all of the 'stats_disk_pool' tables there is one piece of information missing: the LIN. Is there any way to add the LIN to any of those tables or otherwise associate the information in those tables with the 'disk_usage' table? an example of a query that I can run now is: select name from disk_usage where parent in (select lin from disk_usage where name = 'data' and parent = 2); which is a long way of doing 'ls /ifs/data'. Now where it gets interesting is being able to join the output of select name,value from system_properties where name like 'disk_pool:name%'; select disk_pool,sum(log_size) from stats_disk_pool_atime_log_size group by disk_pool; ... but that takes some manual effort (to get the pool number out of 'name') and only gives names to the disk pools. The 'squash' tables present the LIN but not the disk_pool. Is there any way to join the squash tables to the disk_pool tables that I'm missing?

Peter_Sero · Answer

It seems the overall capacity information is not recorded simultaneously by path/LIN and by disk_pool.

What comes closest:

list_top_n_files_by_log_size

list_top_n_files_by_phys_size

but obviously only for a small set, the largest files.

You can get pool statistics per single subtree with

isi filepool apply -r -s DIR

which runs quickest shortly after a SmartPools job.

(actually is gives you "file pool" statistics, which would need to

be mapped into "disk pools" in a post processing step.)

I am currently experimenting with statistics by progressive

random sampling of /ifs (pick random LINs, run

as long as needed, about 1000 LIN/min in a single thread). Interested?

Cheers

-- Peter

rakosnicek · Answer

It feels like the 'isi filepool apply ..' is also doing a disk walk My problem is that one directory under /ifs/data owns about 162,000,000 files.With that many files, even 1000 LIN/min is just too slow.

Peter_Sero · Answer

This idea is to sample only a tiny fraction of files, say 10 or 100 thousand, and extrapolate...

rakosnicek · Answer

If I were a statistician or otherwise not being an engineer, then maybe I'd go with the random sample approach.

But as an engineer I can't take the risk of missing outliers that might mean missing a 100GB file among a collection of files that are mostly 20-30GB in size.

It's just a shame that without the LIN key there is no way to join any of the stats_* tables in results.db with disk_usage.

Another use for the information for the data in disk_usage would be to create a graph or presentation similar to what "Sequoia View" does for Windows filesystems.

Peter_Sero · Answer

Correct, and it gets even worse when you have a "few" GB-files file among millions of KB-files...

Random sampling is most useful for file counts -- it can give a reasonable picture of where those (100s of) million file are located within an hour or even minutes.

Cheers

-- Peter

rakosnicek · Answer

Is there a way to file an RFE to get the 'stats_*' tables updated in results.db to include the LIN for each file?

Isilon

Was this post helpful?