This post is more than 5 years old
9 Posts
0
3699
Best, most efficient, protection level for small files - 55KB
Hello,
What would be the best protection level and node count for 55KB files (we store billions of 55k files..) in regards to overhead/efficiency, or does it even really matter with this size? We are about to purchase a new cluster and would like to get some thoughts on the matter. We currently have an 18 node cluster with multiple pools but are getting killed with the overhead. I'm getting contradicting info from my reps...
Thanks,
Oz
kipcranford
125 Posts
1
July 7th, 2016 08:00
Ultimately the protection level you end up running will depend on the amount of data you actually have, as that will determine the size and makeup of the cluster (I wasn't clear if you're asking about the 18 node cluster or some new cluster). Your SE can take you through those permutations and show you which configurations (from a reliability standpoint) will be valid simply from a capacity standpoint. Be sure to also consider any performance requirements you might have, i.e. you may want to opt for dense X-series nodes over dense archive (NL or HD) nodes, even though the latter could physically hold your data just fine.
As for the protection of your small files... at 55 KiB, these files will be mirrored. At the OneFS default protection of +2d:1n, that means 3x mirrored and so every 55 KiB file will consume 169.5 KiB in the filesystem. At a protection of +3d:1n1d (a likely setting for a larger cluster), that 55 KiB file will consume 226 KiB and will be 4x mirrored. You can use these efficiencies to estimate your total needed capacity.
You might also want to discuss future OneFS features like "containerization" with your account team. For some very specific workflows, the ability to combine small files into large buckets can help tremendously with protection overhead.
sjones51
252 Posts
1
July 7th, 2016 08:00
Hi ozb,
kipcranford is correct. Files under 128KB will be mirrored. Here is some additional documentation that goes into explanation of the protection levels (page 10, File Layout going on to Flexible Protection):
White Paper: High Availability and Data Protection with EMC Isilon Scale-out NAS
https://support.emc.com/docu42429
Additionally, there is a best practice guide for helping to maintain free space:
Best Practice Guide for Maintaining Enough Free Space on Isilon Clusters and Pools:
https://support.emc.com/docu48119
ozb
9 Posts
0
July 7th, 2016 09:00
Thank you Kip and sjones5!
After some reading and your input, I believe the best way forward for me and my situation is to keep the pools small and stay with the default protection level.
Thanks again,
Oz
ozb
9 Posts
0
July 7th, 2016 10:00
Kip,
I have mentioned containerization with the account team since learning about it at EMCWorld but have yet to get anything meaningful back.
How would the different PLs(other than the ones you mentioned) affect the mirroring rate? Could it go to 8x?
Thanks,
Oz
kipcranford
125 Posts
1
July 7th, 2016 20:00
>
> I have mentioned containerization with the account team since learning about it at EMCWorld but have yet to get anything meaningful back.
>
Hmm, that's odd, sorry to hear that. I would suggest trying again, maybe just with your SE. He or she should be able to give you some details on what's coming and whether or not it would make sense in your environment.
>
> How would the different PLs(other than the ones you mentioned) affect the mirroring rate? Could it go to 8x?
>
The mirroring levels map like this (assuming enough nodes for a full mirror set):
+2n and variants (+2d:1n): 3x
+3n and variants (+3d:1n, +3d:1n1d): 4x
+4n and variants (+4d:1n, +4d:2n): 5x
Since +4 is the highest FEC protection we offer today, small file effective mirroring (like we're discussing) is therefore limited to 5x. However, OneFS does offer mirroring *instead of* FEC, and supports levels up to 8x.
sluetze
300 Posts
0
July 8th, 2016 01:00
if you ONLY have this 55k files you could also use mirroring instead of 2d:1n. I have something in my head that calculation of mirroring is faster than an FEC calculation which would result in mirroring.
But that could have changed.
Peter_Sero
1.2K Posts
0
July 8th, 2016 02:00
ozb
what has been your experience on the old cluster with respect to
running "restripe" background job like FlexProtect and MultiScan?
It has been some time since users/admin used to complain
about these job never finishing with 100s of millions of small files...
Which is pretty much in line with my own observation that things
really have improved since OneFs 6.5, and with the use of SSDs
for metadata acceleration.
Nevertheless, in your scenario with particularly small files
you might want to carefully assess the performance
of restripe jobs when moving to denser
node types -- jumping from say 1 TB drives to 6 TB drives
might still yield surprises even on the X series with metadata on SSD.
Cheers
-- Peter
kipcranford
125 Posts
0
July 8th, 2016 06:00
> I have something in my head that calculation of mirroring is faster than an FEC calculation which would result in mirroring.
In general, using a mirroring policy instead of a FEC policy will save a bit on CPU, and in some cases you can avoid read-modify-write overhead when mirroring is in place. However, I don't think either apply to this situation of small files.
ozb
9 Posts
0
July 8th, 2016 13:00
Thanks to everyone for the help and responses.
It is appreciated.
Oz
ozb
9 Posts
0
July 8th, 2016 13:00
Peter,
The Flexprotect jobs seem to running without issue. Multiscan jobs take days to finish.
Our biggest issue is with InsightIQ and FSAnalyze. They will not complete(Billions of files...). We haven't had usable/meaningful IQ info for the better part of a year.
The hope is that going to version 8 of OneFS and the latest version of IQ will give fix the issue.
Thanks,
Oz
Peter_Sero
1.2K Posts
0
July 13th, 2016 02:00
> Our biggest issue is with InsightIQ and FSAnalyze. They will not complete(Billions of files...). We haven't had usable/meaningful IQ info for the better part of a year.
I have a small script that progressively samples the /ifs completely at random
and can give "representative" insights within tens of minutes, or give
it a few hours for improved sampling.
It acts on the LIN ("inode") numbers rather than on paths, so
the sampling is unbiased with respect to the structure of the directory tree.
Great for seeing "where" al the files are,
i.e. getting a quick draft of the tree structure (with file counts) up to a certain depth.
Less suitable for capacity estimating in the case of
"few large files among many small files", as the sampling
error on rare types of files is naturally quite high.
Interested?
-- Peter
ozb
9 Posts
0
July 13th, 2016 16:00
That would be a tremendous help!
I am very interested to check it out.
Let me know how.
Thanks,
Oz
Peter_Sero
1.2K Posts
0
July 14th, 2016 05:00
I'll polish the script a bit and post it here.
stay tuned
KennySabarese
36 Posts
0
July 21st, 2016 09:00
7.2 improves the FSA job significantly. I have a customer with 5 billion files and they've never had an FSA job complete, after the upgrade to 7.2, they finally had one finish. And with OneFS 8, incremental FSA jobs use snapshots so they should complete much faster.
The script sounds cool though to take a random sampling of files to get an idea of what you have.