Unsolved
12 Posts
0
1198
September 24th, 2021 04:00
Isilon: what data does a node pool contain
Hello.
Our Isilon cluster contains two Gen6 Isilons (isi1, isi2). Each of them is containing one node pool with H500 nodes, and one A2000 nodes. Only the H500 nodes of isi1 and isi2 are accessible for users (active client connections). We are having file pool policies running which are supposed to move files older than x months over to the A2000 node pool. Over time the A2000's should fill up, the H500 nodes should empty. The job reports 'SmartPool (phase 1)' on both Isilons are showing that this policy is working.
According to the file system explorer in OneFS (v8.2.1) there should only be approx. 25 % of data (ca. 250 TB, user accessible data) in the H500 node pool. But OneFS (File system > Storage pools > Smart pools > HDD used) and InsightIQ (capacity reporting > capacity forecast > filter by node pool) are reporting that this node pool is used with 52 % (550 TB). Unfortunately neither OneFS nor InsightIQ are showing the details of the data in that node pool. We would like to find out where the other 27 % (approx. 270 TB) of data are coming from.
How to find out what data are located in a node pool? Was looking for a command but could just find where files are located, which is not helpful with millions of files. On the other hand, possibly SyncIQ is putting data from isi2 on the H500 node pool of isi1?
Thank you.
Peter Sero
1 Rookie
•
9 Posts
1
September 24th, 2021 07:00
You can use accounting quotas for a general 'inventory' of the data you have in different parts of the filesystem. Although quotas do not report per node pool, they are helpful to establish certain baselines.
E.g., how much data there is in user directories, how much in system directories (under /ifs/.ifsvar and /ifs/data/Isilon_Support), how much in SyncIQ targets (as set up in your config). Also protection and snapshot overhead can be separately shown. Quotas can be set up according to your specific needs and therefore be a 'sharper tool' than job reports and InsightIQ sometimes.
In your case, it would be worth looking at the system directories, at snapshots (which might be retained on the H500 nodes), and at the SyncIQ target directory. Ideally there would be a simple path-based file pool policy for the SyncIQ target that assigns data directly to the archive nodes without hitting the H500 first.
The magic 'isi get -D /path/to/file' command is insanely helpful to identify the file pool / disk pool / node pool for a specific file. Using it manually, one can do a few checks to see if file pools generally 'seem' to be applied correctly as intended The 'isi get -D' is usually to slow to be applied to (tens or hundreds of) millions of files, but with a bit of scripting one can sub-sample the file system to inspect say every 1000th file.
Spill-over only applies when one node pool is full, and should not (yet be relevant here.
hth
-- Peter
KWI
12 Posts
0
September 28th, 2021 05:00
Accounting quotas are in use already. Because of these and the feature 'Cluster Storage Usage' in Eyeglass we were able to see that there is a big difference between expected and existing amount of data in this node pool.
The amount of data of the system directories on this cluster is negligibly small with 0,23 TB. Most part of the total difference of approx. 250 TB can be explained with SyncIQ data (SyncIQ > Local targets > Coordinator IP) from the other cluster, like you pointed out. This is an approach from the other side, by guessing a source in the hope of finding the correct target, like using 'isi get -D /path/to/file', but that guessing brought us a bit further.
Fiddling around with a script checking every 1000th file is still a lot of output when having tens of millions of files to check. A more useful and detailed and preferably graphical solution like the file explorer in OneFS or File System Analytics in InsightIQ would be helpful.
So far we are looking ahead, before spill-over comes in to play. Trying to manage the data in a best possible way to avoid a too soon expansion of that pool.
Thank you for the hints.