Unsolved
This post is more than 5 years old
4 Posts
0
2243
Information on disk pool usage
One of the challenges that I have been tasked with is providing a summary of the storage used by each NFS/SMB share at a node pool level.
i.e. how much X400 and NL400 space does /ifs/data/Library use?
This is possible by walking the entire directory tree for each directory under /ifs/data but that's also time consuming.
Looking at what resources InsightIQ uses, it appears that file /ifs/.ifsvar/modules/fsa/pub/latest/results.db has a lot of useful information in a nice and small SQLite3 database file except for one thing...
In all of the "stats_disk_pool" tables there is one piece of information missing: the LIN.
Is there any way to add the LIN to any of those tables or otherwise associate the information in those tables with the "disk_usage" table?
an example of a query that I can run now is:
select name from disk_usage where parent in (select lin from disk_usage where name = 'data' and parent = 2);
which is a long way of doing "ls /ifs/data".
Now where it gets interesting is being able to join the output of
select name,value from system_properties where name like 'disk_pool:name%';
select disk_pool,sum(log_size) from stats_disk_pool_atime_log_size group by disk_pool;
... but that takes some manual effort (to get the pool number out of "name") and only gives names to the disk pools.
The "squash" tables present the LIN but not the disk_pool.
Is there any way to join the squash tables to the disk_pool tables that I'm missing?
Peter_Sero
1.2K Posts
0
January 6th, 2016 22:00
It seems the overall capacity information is not recorded simultaneously by path/LIN and by disk_pool.
What comes closest:
list_top_n_files_by_log_size
list_top_n_files_by_phys_size
but obviously only for a small set, the largest files.
You can get pool statistics per single subtree with
isi filepool apply -r -s DIR
which runs quickest shortly after a SmartPools job.
(actually is gives you "file pool" statistics, which would need to
be mapped into "disk pools" in a post processing step.)
I am currently experimenting with statistics by progressive
random sampling of /ifs (pick random LINs, run
as long as needed, about 1000 LIN/min in a single thread). Interested?
Cheers
-- Peter
rakosnicek
4 Posts
0
January 6th, 2016 23:00
It feels like the "isi filepool apply .." is also doing a disk walk
My problem is that one directory under /ifs/data owns about 162,000,000 files.
With that many files, even 1000 LIN/min is just too slow.
Peter_Sero
1.2K Posts
0
January 7th, 2016 04:00
This idea is to sample only a tiny fraction of files, say 10 or 100 thousand, and extrapolate...
rakosnicek
4 Posts
0
January 10th, 2016 16:00
If I were a statistician or otherwise not being an engineer, then maybe I'd go with the random sample approach.
But as an engineer I can't take the risk of missing outliers that might mean missing a 100GB file among a collection of files that are mostly 20-30GB in size.
It's just a shame that without the LIN key there is no way to join any of the stats_* tables in results.db with disk_usage.
Another use for the information for the data in disk_usage would be to create a graph or presentation similar to what "Sequoia View" does for Windows filesystems.
Peter_Sero
1.2K Posts
0
January 12th, 2016 07:00
Correct, and it gets even worse when you have a "few" GB-files file among millions of KB-files...
Random sampling is most useful for file counts -- it can give a reasonable picture of where those (100s of) million file are located within an hour or even minutes.
Cheers
-- Peter
rakosnicek
4 Posts
0
January 12th, 2016 22:00
Is there a way to file an RFE to get the "stats_*" tables updated in results.db to include the LIN for each file?