This post is more than 5 years old
2 Intern
•
300 Posts
0
10362
October 28th, 2015 08:00
Enabling L3 Cache and populating it
Hi Guys,
I'm planning to enable L3 Cache on several upgraded clusters. And have some questions regarding that process.
Current Situation:
- Freshly upgraded Clusters on 7.1.1.x
- MultiScan Jobs etc. are finished
- The Clusters are productive for our customer
- We use our SSDs for Metadata-Read acceleration.
- The current workload is mostly random access and files with <128K are also quite often in use --> we should profit from L3 Cache
- Smartpool license is NOT in use
- on the clusters is only one nodepool used
- the SSDs have space left
Planned process:
isi storagepool nodepool modify -l3 true
wait for jobs to complete
profit.
Since the L3 activation process requires the metadata copies to be evacuated from the SSD with SetProtectPlus and FlexProtect before reformatting the drives as L3 cache, we won't profit from metadataread nor L3 cache until reformatting finished. I want to minimize the performance impact for our customer and thus have the following questions
L3 Cache is populated by evicted blocks from L2 cache. My plan is to have the L3 Cache populated with metadata at the time the customer accesses the data (after the weekend), so he has no performance impact by not having metadataread enabled.
Can i force a population of L3 cache with metadata by just stupidly doing treewalks (via SMB/NFS)?
In my mind the metadata will reside either in L1/L2 or L3 Cache after a complete treewalk (and nothing else!) finished. As soon as data is queried, the meta and data will be evicted from L1/L2 and populated to L3 . if it gets old, it will also be evicted from L3. Since metadata operations (in my environment) are more frequently used than data reads i hope to have a lot of often used metadata and additional data in L3 Cache.
Does FSAnalyze or Snapshot(Deletes) populate metadata to Caches?
If i delete a Directorytree with files do i populate this data into the caches or are caches cleared from deleted files? (i guess the first...)
Is a timecalculation based on experiences from another (comparable) cluster valid?
i.e.
Cluster 1 has 3Nodes and 1Million LINs. it needs 15 Minutes to evacuate Metadata and reformat the SSDs.
Cluster 2 has 3Nodes and 10Million LINs (based on last FSAnalyze). I calculate 2:30 hours for metadata evacuation and reformating of the SSDs
Thanks and Regards
--sluetze
Adam_M1
1 Rookie
•
16 Posts
0
January 20th, 2016 17:00
Thanks Kenny,
unfortunately the pool is in place already, and we are using smartpools to evacuate the NL pools (without L3 being enabled).
Interesting note on OneFS.Next. I will have to watch the cache sizes in OneFS.Next as we will have a large number of small files.
Peter_Sero
6 Operator
•
1.2K Posts
0
January 20th, 2016 22:00
I was told by the Isilon team that any job (SmartPools, FSAnalyze, NDMP, SyncIQ, etc) is using the caches L1-3 as any "regular" local of NASA file access does.
So if your SSDs can hold all metadata in L3, there shouldn't be any need to explicitly warm up L3 after a SmartPools migration.
However if the SSD space used for L3 is too small, these jobs are likely to "poison" (= waste) your cache capacity.
Better control of SSD usage for L3 metadata cache will be highly welcome (but I haven't found this in the roadmaps I have seen so far.)
Interesting times...
-- Peter
KennySabarese
36 Posts
0
January 21st, 2016 04:00
The context was a migration to brand new nodes with no data so I'm not sure I agree with the term "waste" only because even if your SSD's aren't large enough to hold all of your metadata, after the migration, at least _some_ amount of metadata will be in L3 cache which is more than 0. It's no different than warming it with FSA or any other full tree walk.
Regarding the roadmap: all I know is that the goal is for node pools that have enough SSD for all your metadata, that L3 performs equal to the traditional "meta read" policy with the added bonus of data caching with what SSD space is left over instead of it being wasted.
Peter_Sero
6 Operator
•
1.2K Posts
0
January 21st, 2016 04:00
"wasting" in the sense of "poisoning" a cache: filling it with rarely used items while eliminating items from the users' working sets (if the cache can't hold all -- otherwise the warming up is beneficial).
fwiw
-- Peter
KennySabarese
36 Posts
1
January 21st, 2016 05:00
carlilek:
"Regarding the roadmap: all I know is that the goal is for node pools that have enough SSD for all your metadata, that L3 performs equal to the traditional "meta read" policy with the added bonus of data caching with what SSD space is left over instead of it being wasted."
I apologize if my statement above wasn't clear. I believe that your concern will be directly addressed in the future. From what I've heard, the goal for future versions of L3 is to NEVER allow data caching to push metadata out of L3. (maybe there will be some options to tune this for people who want to prioritize data, or just buy more SSD if you want more data caching)
carlilek
2 Intern
•
205 Posts
0
January 21st, 2016 05:00
Thing is, even if there is enough space for all metadata, some will be pushed out by data unless it is pinned to the SSDs, which is what I've been requesting for quite some time. L3 would work fine if no one ever went back and looked at old stuff, but that's not how (my) people work.
KennySabarese
36 Posts
0
January 21st, 2016 05:00
That's a really good question. I would imagine the capability exists. I would suggest having your account team engage an advisory engineer who can escalate to engineering if necessary for an answer of this depth.
KennySabarese
36 Posts
0
January 21st, 2016 05:00
Peter: All of my comments have been focused on the situation of migrating to brand new, empty nodes, so the caches on those new nodes would be completely empty. But yes, I agree that if you have a workload, cache can get dirtied by jobs.
Peter_Sero
6 Operator
•
1.2K Posts
0
January 21st, 2016 05:00
Thanks! Got the point about empty nodes.
Do you know a way to measure how many blocks of an L3 SSD are actually holding metadata and how many are holding data? Other than trying to deduce from hit rate statistics.
Peter_Sero
6 Operator
•
1.2K Posts
0
January 21st, 2016 06:00
Okay -- thanks again.
sluetze
2 Intern
•
300 Posts
0
February 18th, 2016 00:00
After having some talks with EMC Engineers some additions:
yes the system-jobs repopulate the L3 Cache (like FS Analyze)
FSAnalyze is using snapshots from 8.0 up and wont populate / poison the cache with old (meta)data anymore (and will be a lot faster)
also if you enable L3 Cache on an upgraded cluster: dont forget to change the default
isi storagepool settings modify --ssd-l3-cache-default-enabled=true
Alexander_Pursc
11 Posts
0
February 18th, 2016 03:00
Regarding FSAnalyze, it's interesting to know that the access method will change from LIN (like in 7.1) to the new access method "changelist". This method compares two snapshots to find LIN which changed instead of a Inode scan.
sluetze
2 Intern
•
300 Posts
0
February 18th, 2016 08:00
Hi Alex,
lets put our two Statements together:
<8.0: Access is through LINs and Treewalk, L3 Cache gets refreshed
>8.0: Access is through Changeliste API ("snapshot like behavior) and L3 Cache does not get refreshed / poisoned
Peter_Sero
6 Operator
•
1.2K Posts
0
February 18th, 2016 08:00
Can this be configured per folder? I don't want to have full snapshots of /ifs
?
-- Peter