dynamox

updated

9 years ago

D

dynamox

11 Legend

20419 Posts

87439 Points

0

7819

December 5th, 2017 19:00

Retrieving "estimated usable capacity remaining"

Hello folks,

How can i retrieve this metric either from FSA report or directly from PostgreSQL on IIQ system ? I would like to pull this value every so often via an automated process.

12-5-2017 9-46-46 PM.png

Thank you

  • Peter_Sero

    6 Operator

    1169 Posts

    3257

    0

    Posted December 6th, 2017 05:00

    Dynamox, if you can use the Platform API, here is how to query and compute the "usable" remaining capacity:

    1. Get the "raw" available capacity excluding VHS spare as "f_bavail' from /platform/1/cluster/statfs (basically a "df")

    2. To estimate the protection overhead, get

         total_usage.phys_size_sum and total_usage.log_size_sum from /platform/3/fsa/results/0/directories

          Notes:

          - Report id "0" means: retrieve the most recent report from the Isilon, IIQ server is not involved here.

          - No directory LIN given as query argument means: report stats at the toplevel dir, i.e. /ifs

    3. Compute:

        ratio := total_usage.phys_size_sum / total_usage.log_size_sum   (--> ratio is a number larger than one, like 1.18...)

        blocksize := 8192

        usable available capacity in bytes := f_bavail * blocksize / ratio    (--> so "usable" is less than "raw" in bytes)

    Makes sense?

    -- Peter

  • Phil.Lam

    4 Apprentice

    641 Posts

    12 Points

    3255

    0

    Posted December 5th, 2017 21:00

    dynamox


    Ask your local Isilon SE to submit an FR for this feature in IIQ.

    InsightIQ Export of Dashboard Values

    Phil

  • dynamox

    11 Legend

    20419 Posts

    87439 Points

    3255

    0

    Posted December 6th, 2017 05:00

    Phil,

    i can certainly do that but i need something right now. It's a calculated value that i am sure i can do the math if i know what values are used and from which tables in the database i can get them from. 

    Thank you

  • dynamox

    11 Legend

    20419 Posts

    87439 Points

    3257

    0

    Posted December 6th, 2017 08:00

    Peter,

    I do not  use PAPI just yet, where on the file system can i find /platform/3/fsa/results/0/directories ?


    Thank you

  • Peter_Sero

    6 Operator

    1169 Posts

    3257

    0

    Posted December 6th, 2017 10:00

    One more thing...  live quotas right on the Isilon would also do it!

    SmartQuotas always collect both logical and physical usage,

    so if your relevant directories happen to be nicely covered by quotas

    there is not need for the round-trip to InsightIQ at all.

    -- Peter

  • Peter_Sero

    6 Operator

    1169 Posts

    3257

    1

    Posted December 6th, 2017 10:00

    That's not how the Force works...

    The PAPI uses HTTPS to connect to the Isilon,

    and a URL containing /platform/3/fsa/results/0/directories would be (put everything on one line):

    https://user:passwd@myisilon.example.com/platform/3/fsa/results/0/directories

    Or use the Isilon SDK, where these queries are wrapped inside nice regular Python function calls.

    The Info Hub for both the plain API and the SDK is Isilon SDK Info Hub

    There is a bit of a learning curve with the API,

    so if you want to go with tradional shell scripts and tools like grep and awk,

    just use "df" on the Isilon, and do the following on the InsightIQ machine:

    Find the most recent report via

    iiq_data_export fsa list  --reports CLUSTERNAME

    Then:

    iiq_data_export fsa export -c CLUSTERNAME -d directories -o REPORT_ID -n OUTPUT_FILE.CSV

    The output will look like this:

    path[directory:/ifs/],dir_cnt(count),file_cnt(count),ads_cnt,other_cnt(count),log_size_sum(bytes),phys_size_sum(bytes),log_size_sum_overflow,report_date:1512491989

    /ifs/data,225,98,0,0,446296338,448048128,0

    /ifs/home,5,1,0,0,756,18944,0

    /ifs/upgrade_error_logs,1,0,0,0,0,2048,0

    In contrast to the API, where stats are available for /ifs itself,

    there is now one line for each directory immediately BELOW /ifs .

    log_size_sum and phys_size_sum are the totals for the subtrees under the listed directories.

    So if you have multiple 'relevant' directories below /ifs (like /ifs/dept1, /ifs/dept2 or so),

    then you need to sum up all log_size_sum fields and all log_size_sum fields first

    before computing the ratio.


    hth


    -- Peter

  • dynamox

    11 Legend

    20419 Posts

    87439 Points

    3257

    1

    Posted December 6th, 2017 11:00

    Peter,

    Thank you so much for your help, greatly appreciate it.  I use REST with XtremIO and VMAX U4V already, was trying to do something quick and dirty with bash.  Just tried the end points your provided and see that's it's very easy to get those fields.  I use perl module to work with JSON so it's similar to what i already do with other EMC platforms.  This metric will be very valuable in my Grafana portal.

    Thank you again

  • 3257

    0

    Posted December 6th, 2017 14:00

    Hi guys,

    All really good replies here.

    I've just tried using Platform API to look for the FSA estimation of usable capacity.  This currently does not exist as an API call. It is calculated within InsightIQ, using a specific FSA result set, to estimate usable capacity.

    As the other contributors have pointed out, quotas can help, as well as iiq_data_export (and then manually calculating logical vs physical).

    For further reading/research into this subject:

    Online API doc: http://doc.isilon.com/onefs/8.0.0/api/en-us/03-ifs-br-system-config-api.htm

    Browser search for "FSA resources" to see a list of API calls relating to the FSA results.

    A related blog: https://community.emc.com/community/products/isilon/blog/2017/06/28/accessing-fsa-results-through-platform-api

  • Peter_Sero

    6 Operator

    1169 Posts

    2143

    1

    Posted December 7th, 2017 04:00

    Sure, performancewise I wouldn't be overly scared of having one more SmartQuota domain,

    as long as it is not set at the /ifs toplevel.

    Wether to include snapshots or not, it's your call -- after all the purpose is to estimate

    the protection rate of FUTURE data to be written into the available space.

    -- Peter

  • dynamox

    11 Legend

    20419 Posts

    87439 Points

    3257

    1

    Posted December 7th, 2017 04:00

    Rob and Peter,

    In terms of using smart quotas for this exercise,  are you suggesting setting an accounting quota on /ifs/access_zonename and then getting the "usage with overhead"  and "usage without overhead" values for my calculations ?   I kind of like that option because then i don't have to wait on FSAnalyze to complete which in my case takes a couple of days.

    Thank you