chjatwork

updated

9 years ago

C

chjatwork

2 Intern

356 Posts

0

2522

August 15th, 2017 06:00

Isilon - (Script) - Run "du -hxcd1" against output

Community,

I need to know what script will allow me to run a "du - hxcd1" against the output of "Target Path" column for the command "isi sync target list".

If possible the output should be "Name","Source","Target Path","RESULT OF THE DU TOTAL"

Where you at?  Peter Serocka

Thank you in advance!!

  • Peter_Sero

    6 Operator

    1169 Posts

    1485

    0

    Posted August 16th, 2017 08:00

    Thanks for the feedback! I think you got the quote marks right by copy-pasting.

    Had to add another line to match a newer output format:

    isi sync target list -v |

    awk -F": " '

    { print $0; }

    /^Target Dir:/ { printf "du: "; system("du -hxd1 \""$2"\""); }

    /^ *Target Path:/ { printf "                        du: "; system("du -hxd1 \""$2"\""); }

    '

    The script simple adds a line tagged with "du:" after the Target Path line.

    The path is repeated within the output from the du command.

    sample output:

                          Name: TEST

                        Source: isi810

                   Target Path: /ifs/data/target 1

                            du: 2.5K /ifs/data/target 1

                Last Job State: finished

                    FOFB State: writes_disabled

           Source Cluster GUID: 000c2982a34602941c59a81c747b2bfa5644

    Last Source Coordinator IP: 127.0.0.1

                 Legacy Policy: No

                   Last Update: 2017-08-16T11:22:16

    (Note: the example target path contains a space in "target 1" to show that spaces in paths are handled correctly.

    That's what those multiple quote marks are for...)

    If even different output formats are found, please post a sample.

    Cheers

    -- Peter

  • Peter_Sero

    6 Operator

    1169 Posts

    1485

    0

    Posted August 15th, 2017 11:00

    Printing pretty tables would required some more lines of code,

    but for the list-style of the isi sync output I can offer what is essentially a decorated one-liner:

    isi sync target list -v |

    awk -F": " '

    { print $0; }

    /^Target Dir:/ { printf "du: "; system("du -hxd1 \""$2"\""); }

    '


    hth

    -- Peter

  • chjatwork

    2 Intern

    356 Posts

    1485

    0

    Posted August 16th, 2017 03:00

    Peter Serocka

    Thank you for taking time out to help me with this.  Hey, maybe I am doing something wrong cause I am just getting the printout for the:

    # isi sync target list -v

    Its not providing any additional information in reference to disk usage on each filesystem.  I am not sure if its because the script is broken up and I did not assemble it properly in a one liner?  I ran it like this:

    # isi sync target list -v | awk -F": " ' { print $0; } /^Target Dir:/ { printf "du: "; system("du -hxd1 \""$2"\""); }'

    and I also just cut and paste it in how you had it above:

    # isi sync target list -v |

    pipe>

    pipe> awk -F": " '

    pipe quote>

    pipe quote> { print $0; }

    pipe quote>

    pipe quote> /^Target Dir:/ { printf "du: "; system("du -hxd1 \""$2"\""); }

    pipe quote>

    pipe quote> '

    Both instances I just got the output for "isi sync target list -v"

    I will try and brake the script up into smaller pieces to see what I maybe doing wrong, but if you can see let me know.

  • chjatwork

    2 Intern

    356 Posts

    1485

    0

    Posted August 17th, 2017 03:00

    Now if I could find a way to make these scripts run faster   It takes forever when you got TBs of data.

  • sluetze

    2 Intern

    300 Posts

    1485

    0

    Posted August 17th, 2017 04:00

    The way to make it faster is against your requirement of using du –hxcd1.

    Because that’s the process doing the “real” work. The rest of peters code is only formatting.

    To be faster you could leverage FSA Jobs. Or generate the “du” metadata on a schedule and only use the data when needed.

    Rgds

    -- sluetze

  • chjatwork

    2 Intern

    356 Posts

    1485

    0

    Posted August 17th, 2017 04:00

    @sluetze how do I leverage FSA? I just added the cluster to InsightIQ so

    it's going to take a serveral weeks before we can get enough data to start

    seeing any kind of trend.

  • Peter_Sero

    6 Operator

    1169 Posts

    1485

    0

    Posted August 17th, 2017 04:00

    With a SmartQuotas license you can set directory quotas

    just for accounting and reporting, i.e. no hard limits if not needed.

    Would volunteer to update the script accordingly

    FSA reports can be accessed through the platform API,

    scripts tend to get lengthy with boilerplate.

    -- Peter

  • sluetze

    2 Intern

    300 Posts

    1485

    0

    Posted August 17th, 2017 05:00

    as peter said, you can access it through P-API. There are some ways to directly access the database but I would not recommend that, since upgrades brick the accessmethod regularly :P

    You also won’t see any trends with du -hxcd1.

    I suggest you to try to define what you want to achieve – maybe someone already faced this and can tell you a stable way to reach your aim.

    Rgds

    --sluetze

  • chjatwork

    2 Intern

    356 Posts

    1485

    0

    Posted August 17th, 2017 09:00

    I agree your not going to get trending from running du once, but if a daily du is achievable on a cluster then you can see those report over a period of time and establish a trend.

    Peter Serocka

    Id totally welcome the script that would create a directory quotas just for accounting and reporting.