How will ESX handle Virtual Provisioned LUN

Good questions. I am interested in these questions also. Eager for answers.

there might be some answers in this paper

White Paper: Implementing EMC Symmetrix Virtual Provisioning with VMware vSphere — Applied Technology

1. Yes, ESX will not be aware of thinlun

2. No, creating a VMFS volume will not allocate the entire LUN. ESX will only allocate disk space when you actually USE the space. Exception for the Eagerzeroed thick VMDK that will allocate all disk space upfront.

3. Answer above.

4. None. VASA is used for administration benefit. (basically VM placement/migration decision). BUT you want to have VAAI. VAAI will enable ThinProvisioning stun that will put VM "on hold" when storage runs out of disks space.

Thanks all for your response.

Cody, I'm reading your WP and will check the VMAX pool  to verify allocation for different provisions ESXi use. Will update everyone of my findings.

I have read Cody's WP and tested it on the lab box. Amazing document, it has answered some questions that puzzled me for a while. As below are answers for all questions.

1. If VASA is not configured, will ESX be aware of this is a thin LUN not? My guess is no.

Yes, ESX will identify thin provisioned LUNs using thin provisioning enabled (TPE) bit from READ CAPACITY (16) response. This is of course the arrays are T10 compliant which EMC arrays are.

2. If ESX doesn't know it's thin LUN, will VMFS datastore creation allocate all space of the thin device? My guess is yes

ESX knows it's thin LUN. RDM, Thick Provision Lazy Zeroed, and Thin Provision will only allocate the meta data from pool. Only the creation of Thick Provision Eager Zeroed Datastore will allocate all assigned space from thin pool to ESX. The Metadata consumptionis 970MB to be reserved and 1MB for each 100GB in VMFS5 volumes. Even Metadata does not be written compltely at the volume creation.

3. Is there a way not to allocate the whole device capacity? Currently I create a fairly small, 6G VMFS datastore from thin device which can be expanded later. Will only the 6G be allocated? What if we use RDM, does RDM work?

Same as Q2.

4. If VASA is configured, what different does it make for thin device provision, Datastore creation?

VASA can pass Symmetrix thin pool  out-of-space warning to Vcenter so that ESXi can handle storage accordingly.

It's recommended to create virtual VMFS datastore on thick Symmetrix Lun and create Thick Provision Lazy Zeroed datastore on thin device. thick on thin  or thin on thick model. Zeroedthick lun creation will allocate space on ESX level, but won't initialize with zero on storage level, therefore no track need to be allocated on thin pool.

I also find after adding a thin VMDK drive to VM,   Windows XP format utility (with or without quick option)  won't allocate any space on storage thin pool at all. Initiailizing and partitioning this drive will write small amount of metadata to disk.

Creating a Thick Provision Eager Zeroed Datastore will allocate all assigned space in thin pool. However it can be freed up after Datastore creation by SMC. I do be confused by  "free" and "reclaim" options in SMC "Start Allocate/Free/Reclaim". Reclaim only unallocate very few tracks and Free seems unallocate the whole disk zero written tracks. What do these 2 really mean?

Hi,

Jingyi, I too am slightly confused

I am not sure what version of Solutions Enabler/SMC you are on but it seems to have slightly changed in 7.3. In 7.3 it is not documented as well as previous versions IMO.

In previous versions there was a reclaim_type=unwritten (allocated but not used) or reclaim_type=reclaim (which reclaim unwritten and devs zeroed out). There is only a "start free" statement there is no start reclaim

In 7.3 the support for the clause reclaim_type= seems to be lapsed.

My reading is that instead there is "start free" or "start reclaim" where "start free" reclaims the equivalent to unwritten (allocated but not used) and also does not have persistent attribute set. "start reclaim" reclaims unwritten even if the persistent attribute set. I am assuming that "start reclaim" reclaims zeroed out space as well as unwritten even though I can not see this explicitly in the documentation.

The best practices documentation (P/N 300-006-718) references the symconfigure syntax with reclaim_type= so could it be clarified for SE/SMC 7.3? 

Regards,

Victor

Starting with Solutions Enabler V7.3, two new commands, start reclaim and stopreclaim are introduced to manage reclaiming thin device tracks. The start free and stop free commands will continue to be supported for unwritten allocations on a range of cylinders for a device, and for stopping the free operation on a thindevice in the De-allocating state (only).

To reclaim tracks on thin devices, use the following syntax:

start reclaim on tdev [allocate_type = persistent];

If the allocate_type=persistent is specified, this command frees up tracks that

are unwritten or zero-based, even if they are marked as persistent.

So start free only de-allocates tracks that are allocated but also set Not Written By Host (NBWH). Start reclaim will reclaim both allocated tracks with NWBH and zeroed extents.

Thanks for clarifying Cody, it is crystal clear now the way 7.3 handles it!

Regards,

Victor

Great, glad i could help!

Victor, I'm using SE version 7.3.2.2.  I actually did the operation on SMC instead of CLI. Below is the SMC reclaim screenshot.

After creating 8G eager zero thick device, Used CLI to display allocated and written tracks.

                       Pool          Pool             Total                 

   Sym          Total  Subs        Allocated         Written             

   Dev         Tracks   (%)     Tracks   (%)     Tracks   (%)  Status 

   -----------------------------------------------------------------------

   0756        982800    11     142008    14       9834     1  Bound     

           ---------- ----- ---------- ----- ---------- -----

   Tracks      982800    11     142008    14       9834     1

  }

Allocated tracks rise from 10920 to 142008.

Then I used SMC to start Reclaim, uncheck the persistent option.

                      Pool          Pool             Total                 

   Sym          Total  Subs        Allocated         Written             

   Dev         Tracks   (%)     Tracks   (%)     Tracks   (%)  Status 

   -----------------------------------------------------------------------

   0756        982800    11     141720    14       9518     1  Reclaiming

           ---------- ----- ---------- ----- ---------- -----

   Tracks      982800    11     141720    14       9518     1

  }

There were very few tracks being de-allocated.

Selectd Start Free, still uncheck the persistent option.

                       Pool          Pool             Total                 

   Sym          Total  Subs        Allocated         Written             

   Dev         Tracks   (%)     Tracks   (%)     Tracks   (%)  Status 

   -----------------------------------------------------------------------

   0756        982800    11      10608     1       9517     1  Bound     

           ---------- ----- ---------- ----- ---------- -----

   Tracks      982800    11      10608     1       9517     1

  }

You can see almost all 8G were de-allocated.

Cody, now you can see why I ask the difference between Start Reclaim and Start Free. I still don't undetstand the logic behind these 2 options.

If you are deploying an eagerzeroedthick virtual disk with ESX 4.1+ and Enginuity 5875 the ESX host will use write same and not actually write the zeros. It will only set the tracks Not Written By Host and then allocate the given extents (each extent =12 tracks=768KB). This is why there is no real difference in the written tracks after deployment. The written tracks that exist are simply just the VMFS metadata. Since no zeroes are written, there is no real difference in the way free or reclaim behave (or shouldn’t be). Free only de-allocates extents that are NWBH and since that is what the EZT deployment did, all of those tracks should be free after the start free. Start reclaim should query all of the allocated extents for extents that are filled with contiguous zeros or fully NWBH. Extents that are fully NWBH will be de-allocated and all-zero extents will be marked NWBH and then de-allocated.

The second pool show you have still shows the device as reclaiming. What was its final state?

Cody, thanks for your quick response. You are right about this, Free only de-allocates extents that are NWBH and  reclaim de-allocated extents that are filled with contiguous zeros or fully NWBH. The second pool show is still reclaiming, I must have captured the screen too early. So I redid the test and found reclaiming acted  exactly same as you said.

I also have another interesting finding. For all-zero tracks, reclaiming, will reduce total written tracks  by 316 tracks while reduce all allocated only by 288 tracks.

Did the free operation first,

Pool Bound Thin Devices(1):

  {

   -----------------------------------------------------------------------

                       Pool          Pool             Total                 

   Sym          Total  Subs        Allocated         Written             

   Dev         Tracks   (%)     Tracks   (%)     Tracks   (%)  Status 

   -----------------------------------------------------------------------

   0756        982800    11      11892     1       9919     1  Bound     

           ---------- ----- ---------- ----- ---------- -----

   Tracks      982800    11      11892     1       9919     1

  }

Start  reclaim

Pool Bound Thin Devices(1):

  {

   -----------------------------------------------------------------------

                       Pool          Pool             Total                 

   Sym          Total  Subs        Allocated         Written             

   Dev         Tracks   (%)     Tracks   (%)     Tracks   (%)  Status 

   -----------------------------------------------------------------------

   0756        982800    11      11604     1       9603     1  Bound     

           ---------- ----- ---------- ----- ---------- -----

   Tracks      982800    11      11604     1       9603     1

11892-11604=288

9919-9603-316

316-288=28 tracks

I tested twice and got same result. My thin meta device has 10 meta memebers. I don't know if the difference has something to do with Data structure, or it's the designed behavior for thin device allocate and reclaim.

Cody, I also read this primus emc277013, it mentioned:

When space is allocated the Symmetrix Enginuity code will fill the space with zeros. As the allocated space is used up it is overwrites the initial zeros. Therefore simply deleting the files from the allocated space does not meet the criteria for reclaiming space. The customer would have to fill the entire allocated space that was used with zeros, then they should be able to perform the reclaim. EMC has no procedure on how to fill a disk with zeros. The customer would have to write a script of his own.

As current Enginuity does not support UNMAP command, is there any setting or tool in vmware to write zeros to the dead space which was written by non-zero value before? If yes, these space then can be reclaimed later manually. I am also curious about the implementation of UNMAP command if it become supported later. Will the tracks de-allocated be written to all zeros before retuning to pool? or these tracks will be designed to zero out only on allocation phase?

I actually just updated the virtual provisioning with vSphere white paper this week to include a way to reclaim dead space by zeroing it in VMware and then reclaiming it. Check it out here:

http://www.emc.com/collateral/hardware/white-papers/h6813-implting-symmetrix-vrtl-prvsning-vsphere-wp.pdf

There is a section called reclaiming space on symmetrix thin devices and goes through the process.

UNMAP just sets the tracks NWBH and then de-allocates the extents as well. There is no zeroing involved. The write same with the UNMAP bit set will do this though--but this is not used often. The only thing that uses write same with the UNMAP bit set is the storreclaim utility that we offer through PSE.

I believe the difference you are seeing is that metamember require that a certain amount of allocation to remain even if they have no data (the act of binding does this). These extents cannot be reclaimed fully, just set NWBH.

Cody, you are amazing to answer all questions I might think of. Just downloaded the updated WP, it seems not be published in powerlink officially. I am eager to see UNMAP being supported by later enginuity, probably 5876?  Look forward to discussing with you later.