Retrieving "estimated usable capacity remaining"

dynamox

Ask your local Isilon SE to submit an FR for this feature in IIQ.

InsightIQ Export of Dashboard Values

Phil

Phil,

i can certainly do that but i need something right now. It's a calculated value that i am sure i can do the math if i know what values are used and from which tables in the database i can get them from. 

Thank you

Peter,

I do not  use PAPI just yet, where on the file system can i find /platform/3/fsa/results/0/directories ?

Thank you

One more thing...  live quotas right on the Isilon would also do it!

SmartQuotas always collect both logical and physical usage,

so if your relevant directories happen to be nicely covered by quotas

there is not need for the round-trip to InsightIQ at all.

-- Peter

That's not how the Force works...

The PAPI uses HTTPS to connect to the Isilon,

and a URL containing /platform/3/fsa/results/0/directories would be (put everything on one line):

https://user:passwd@myisilon.example.com/platform/3/fsa/results/0/directories

Or use the Isilon SDK, where these queries are wrapped inside nice regular Python function calls.

The Info Hub for both the plain API and the SDK is Isilon SDK Info Hub

There is a bit of a learning curve with the API,

so if you want to go with tradional shell scripts and tools like grep and awk,

just use "df" on the Isilon, and do the following on the InsightIQ machine:

Find the most recent report via

iiq_data_export fsa list  --reports CLUSTERNAME

Then:

iiq_data_export fsa export -c CLUSTERNAME -d directories -o REPORT_ID -n OUTPUT_FILE.CSV

The output will look like this:

path[directory:/ifs/],dir_cnt(count),file_cnt(count),ads_cnt,other_cnt(count),log_size_sum(bytes),phys_size_sum(bytes),log_size_sum_overflow,report_date:1512491989

/ifs/data,225,98,0,0,446296338,448048128,0

/ifs/home,5,1,0,0,756,18944,0

/ifs/upgrade_error_logs,1,0,0,0,0,2048,0

In contrast to the API, where stats are available for /ifs itself,

there is now one line for each directory immediately BELOW /ifs .

log_size_sum and phys_size_sum are the totals for the subtrees under the listed directories.

So if you have multiple 'relevant' directories below /ifs (like /ifs/dept1, /ifs/dept2 or so),

then you need to sum up all log_size_sum fields and all log_size_sum fields first

before computing the ratio.

hth

-- Peter

Peter,

Thank you so much for your help, greatly appreciate it.  I use REST with XtremIO and VMAX U4V already, was trying to do something quick and dirty with bash.  Just tried the end points your provided and see that's it's very easy to get those fields.  I use perl module to work with JSON so it's similar to what i already do with other EMC platforms.  This metric will be very valuable in my Grafana portal.

Thank you again

Hi guys,

All really good replies here.

I've just tried using Platform API to look for the FSA estimation of usable capacity.  This currently does not exist as an API call. It is calculated within InsightIQ, using a specific FSA result set, to estimate usable capacity.

As the other contributors have pointed out, quotas can help, as well as iiq_data_export (and then manually calculating logical vs physical).

For further reading/research into this subject:

Online API doc: http://doc.isilon.com/onefs/8.0.0/api/en-us/03-ifs-br-system-config-api.htm

Browser search for "FSA resources" to see a list of API calls relating to the FSA results.

A related blog: https://community.emc.com/community/products/isilon/blog/2017/06/28/accessing-fsa-results-through-platform-api

Sure, performancewise I wouldn't be overly scared of having one more SmartQuota domain,

as long as it is not set at the /ifs toplevel.

Wether to include snapshots or not, it's your call -- after all the purpose is to estimate

the protection rate of FUTURE data to be written into the available space.

-- Peter

I do that all the time, Dynamox.  Sometimes I cannot use hard quotas (if for example the networking/active directory team handles it) so I always use accounting quotas to keep track of how much space each group is using.

No just do an accounting quota.  It doesn't put any high overhead as far as I have seen. Just a regular tree walk.

I run accounting quotas against:

/ifs/data

/ifs/data/SHARE

/ifs/data/SHARE/SHARE

In other words, wherever I need to track usage, an accounting quota is placed.  Hard quotas are only used on my multi-tenant clusters.  Dedicated clusters have just an accounting quota against /ifs/data.

Brian,

I am somewhat hesitant of setting a quota that high in the file system structure,  concerned of potential performance impact to this 2PB cluster.

Brian,

How big is your cluster ?  I want to give my customer an estimate of how much data they "could potentially" store in the remaining free raw file system capacity.  it's an estimation but gives them a balk park number.  Do you do anything with the number you gather from these accounting quotas other then telling you/customers how much logical space they are using ?

He's an example of one of my big clusters (58 nodes).  It has 7.4PB total capacity and I use 95% as the ABSOLUTE hard limit to protect their data.

Type      AppliesTo  Path                        Snap  Hard  Soft  Adv   Used

---------------------------------------------------------------------------------

directory DEFAULT    /ifs/data                   Yes   7.00P -     6.45P 6.71052P

---------------------------------------------------------------------------------

It's pretty easy to estimate the used and free capacity along with overhead (based on protection level). What I tell them is based on the free capacity, use the protection level and provide a pretty “close to accurate” available capacity limit.  That’s what I use when I talk to internal customers (which needs to also include snapshots if you’re using them because of course it’s going to use up a significant amount of space).

BTW: This cluster is almost always sitting between 89-92% but that’s fine, it’s dedicated to one internal customer and they know their capacity levels.  Are you trying to give the customer a 100% accurate “free space” quote?