My company recently deployed FAST-VP on a V-Max 10K using 3 disk pools, EFD, FC and SATA.
After running FAST-VP for 8 months it's apparent that the FAST controller over weights the higher tiers of disk (i.e. EFD and FC).
Our EFD pool is always above 95% utilized, which is good, our FC pool hovers around 70% utilized and SATA around 30%.
So my question is at what level of FC pool utilization should we be planning to add more disk? 75%, 80%, 85%?
It seems like the FAST controller is demoting from the FC tier, to the SATA tier in order to avoid running out of space.
We don't want to get into a situation where we're demoting data because we HAVE to, we only want to demote to SATA when the data is truly cold.
tzvb32, Let me first give you a brief about how FAST controller works, FAST controller makes its decisions on extents promotion/demotion based on promotion/demotion thresholds run on sub lun level. These thresholds are calculated using the collected performance metrics and the available capacity on each tier. The goal of these thresholds is to maximize the utilization of the highest performance tier while demoting inactive or less accessed extents to less performing tier (like SATA)
For your case, I won't make my decisions to add new disks based on FAST movements. You should consider the amount of allocated tracks for your TDEVs and the over-subscription ratio for your pools. These would be the factors which will make me decide to add new disks.
There is a feature that i highly recommend you to enable, VP Allocation by FAST policy. with this feature enabled you will never have your IOs halted due to lack of space of the pool your TDEVs are bound to.This feature allows new allocations to come from any of the thin pools included in the FAST VP policy that the thin device is associated with.
Once you enable this feature, It attempts to allocate new writes in the most appropriate tier first, based on available performance metrics. If no performance metrics are available, an attempt is made to allocate to the pool to which the device is bound. If the pool initially chosen to allocate the data is full, FAST VP then looks to other pools contained within the FAST VP policy and allocates from there. As long as there is space available in at least one pool within the policy, all new extent allocations will be successful.
You can also control FAST movements by increasing the PRC on Pool level.
I think FAST VP is working as expected in your case I have 2 question though, May i ask what is your FAST VP policies configuration? and which pool TDEVs are bound to? is it only FC pool or a mix of FC and SATA pools?
Hope it helps.
Mohammed Salem @yankoora
We have set the pool alert at 80% for FC & Sata, we dont care about EFD much as we don't bind any tdevs to EFD pool (its only for FAST VP purpose). Mgmt are usually slow when it comes to approving the PO to buy more disks. Usually it takes 1 month in our company to approve PO and around 15-20 days for EMC to do shipping and installation. So, just to keep a bit of cushion, we have set the alert for FC & SATA at 80%.
Also, we keep a monthly check on the pool utilization and look at the usage trend. This gives a tentative date by which we will be ending up with 100% utilization. We also keep a monthly track on each tdev utilization, if there is a huge diff between allocated & written tracks, we reclaim that space.
VP Policy Configuration
T1_FAST: 35% EFD, 100% FC, 25% SATA
T2_FAST: 15% EFD, 75% FC, 75% SATA
T3_FAST: 50% FC, 100% SATA
We always bind all TDEVs to the FC pool, LUNs are never bound to the EFD and SATA pools directly.
PRC is set to 1% on EFD, 10% on FC and 1% on SATA.
Allocation by FAST policy is enabled.
I'm curious why your higher performance policies exclude the option of 100% SATA? While I know it is unlikely anything assigned the policies with the higher "performance potential" would ever use 100% there is no harm in allowing anything that doesn't need to performance to drop down.
We have two different types of policies in play in our environment. The "traditional policies" are the ones that include all three tiers in each. They all start with 100% SATA as a base and they increase the FC and EFD tier allowances as they go up. I do have a few of our top tier applications that consume the full 15% EFD we allow them but barely touch the FC tier even though they could go 100% if they need it.
Just for completeness (since I mentioned the other tier type) we also have a set of policies that we are looking to implement which virtually isolate specific tiers. We have one that allows 100% SATA and 5% FC, one that allows 100% FC and 1% EFD, and one that will only allow 100% EFD.
That's a good question, we do this primarily for our Oracle environments that run monthly batch cycles.
Limiting the amount of SATA in the T1_FAST policy insulates us from performance issues associated with data that has been demoted to SATA and needs to be promoted quickly for a reporting or batch cycle. Having said that we're considering bumping up the amount of SATA exposure in the T1 policy to 30% or 35% based on what we're seeing in our FAST-VP compliance reports.
I can understand that, but then I wonder why you are able to allow any of the data to drop to the SATA tier. Wouldn't allowing even 25% of it to drop potentially cause the same issue (since you don't directly control which data moves).
I hope I'm not coming off as argumentative here... I truly want to understand since we have these types of discussions regularly around here and every bit of data/knowledge we can bring to the table helps in the end 🙂
We started out with a much lower percentage of SATA in our T1_FAST policy and have been testing and slowly moving the allowed SATA % up based on SG metrics and feedback from the application owners. So we haven't seen an issue with allowing 25% in SATA and we'll continue to increase the amount of SATA exposure until we start seeing an impact to our month end processing and then back it off. Unisphere for V-Max with SPA has been a great tool to help us make good decisions with respect to our T1_FAST policy.
Guys, generally speaking I will not recommend making more than one policy for same tiers. I will make it 100/100/100.FAST VP will take care of everything else related to data movements.
You can check Sean Cummins own blog about FAST VP Best practices. One of the greatest posts about FAST VP:
Hope it helps
Mohammed Salem @yankoora
Mohammed, can you clarify what you mean by not "making more than one policy for the same tiers"? If you only make a single 100/100/100 policy then how do you ensure that applications needing more performance can get it while other apps that shouldn't waste your more expensive tiers can't "waste your money".
I agree that if money was no issue and you could put unlimited amounts of EFD and FC in your array then a 100/100/100 policy would be great. Our budgets are a bit more constrained though.
Maybe I'm just misunderstanding your point...?