Unsolved
This post is more than 5 years old
5 Practitioner
•
274.2K Posts
0
1831
fsa export unable to go deep enough into the paths or multiple paths
Hi
Purpose: Extracting data from the FSA report via command line, in order to obtain size of file paths with top level directories and much deeper than displayed on iiq web gui.
Command:
iiq_data_export fsa export --cluster clustername --data-module directories --filter-rule directory:data/ifs/resolve -o 6688 -n g.csv --path /mnt/172.27.210.1/ifs/data/ifs/eng/insightiq
The data that I could withdraw was good, but I was unable to go deep enough into the paths or multiple paths. I tried so many different possibilities, but none of them was able to produce what I needed.
Question:
is there a way to change the log size (extend report size) in order to obtain more detailed information from the paths, so sort if a --max-depth so the report shows more details as the paths are deeper than is reported?
More detail:
- iiq_data_export fsa export --cluster clustername --data-module directories -o 6688 --path /mnt/172.27.210.1/ifs/data/ifs/eng/insightiq/
-works also: Gives out 1000 paths but not per directory?
- iiq_data_export fsa export --cluster clustername --data-module directories -m 1000 -o 6688 --path /mnt/172.27.210.1/ifs/data/ifs/eng/insightiq/
-works: gives out all top level folders in FSA
SO WITH THE ABOVE COMMAND I WAS UNABLE TO GO ANY FURTHER INTO THE PATHS THAN TOP LEVEL DIRECTORIES
iiq_data_export fsa export --cluster clustername --data-module directories --filter-rule directory:data/ifs/resolve -o 6688 -n g.csv --path /mnt/172.27.210.1/ifs/data/ifs/eng/insightiq
THIS WOULD ALSO GIVE ME JUST TOP LEVEL AND NOT ANY FURTHER
Summary:
unable to query directory & directories (data modules & breakouts) within the same query...
unable to go deep enough into the paths or multiple paths
Any assistance greatly appreciate, has any of this functionality changed in v4.0, currently running 3.2.2
Peter_Sero
1.2K Posts
2
March 10th, 2016 03:00
The report depth is limited to what has been scanned
by the FSAnalyze job of the cluster, but the depth level be changed.
(Prior to running a new job!)
In the InsightIQ WebUI, go to Settings -> [your cluster] -> FSA Configuration:
Adjust the depth CAREFULLY, increment by one and run FSAnalyze.
Beware, the database might grow EXPONENTIALLY with the max directory depth!
hth
-- Peter
Anonymous
5 Practitioner
5 Practitioner
•
274.2K Posts
0
March 10th, 2016 05:00
I extended to 25 and the reports were coming back at 40-50GB!
Even with this I could query each path individually, with extended depth results.
However from a top level query it wouldn't go further than the top folders within that path.
RobChang-Isilon
136 Posts
0
March 24th, 2016 18:00
Hi Ian,
I'm having trouble visualizing what it is you are having issues with. With the iiq_data_export command for the directories data-module, you are supposed to only get top-level directories unless you specify a filter-rule. With the filter-rule applied, you will only see the directory specified in the filter-rule as the top level directory and no farther. To query for multiple directories, you'd have to issue iiq_data_export on each of them (using the filter rule). The path option specifies the target directory where the CSV file should be stored (and not to be confused with the path you are looking to analyze).
I don't think this has changed in version 4.0.
There's one thread on ECN that covers a little bit of what you are trying to do: Exporting InsightIQ data
I'm pasting the relevant chunk here:
The following has been tested on InsightIQ versions 3.2.0 and 3.2.1.
On the node that is running your InsightIQ instance:
$ iiq_data_export fsa list --reports
Available Reports for: Time Zone: PDT
ID FSA Job Start FSA Job End Size
--------------------------------------------------------------------------------
765 Sep 30 2015, 10:00 PM Oct 01 2015, 11:28 PM 3.500G
--------------------------------------------------------------------------------
$ iiq_data_export fsa export --data-module directories --cluster --report 765 --filter-rule directory:TEST/MY/products/xxxx/vxx
Successfully exported data to: directories_ _765_1443800888.csv
$ cat directories_ _765_1443800888.csv
path[directory:/ifs/TEST/MY/products/xxxx/vxx/],dir_cnt (count),file_cnt (count),ads_cnt,other_cnt (count),log_size_sum (bytes),phys_size_sum (bytes),log_size_sum_overflow,report_date: 1443675617
TEST/MY/products/xxxx/vxx/test,1,0,0,0,0,2048,0
Note: The directory specified in the filter rule "TEST/MY/products/xxxx/vxx" -- this directory exists on the cluster as "/ifs/TEST/MY/products/xxxx/vxx".
Hope this helps.