Highlighted
3 Argentum

Reclaim Not Working on Equallogic Volumes?

We have 3 different SAN vendors. On two of those SAN vendors the new esxcli unmap command with esxi 5.5 works fine. However, when running that command against volumes on the Equallogic arrays (thin provisioned volumes SAN side), the command never actually works. It completes without warning, but even after 24 hours, no volume space has been reclaimed.

I am hoping I can be provided with some insight?

The command I am using is the following:

esxcli --server=servername --username=root --password=password storage vmfs unmap --volume-label=san_volume

0 Kudos
14 Replies
Highlighted

RE: Reclaim Not Working on Equallogic Volumes?

There are a couple of requirements for SCSI UNMAP to work.

1.) EQL FW has to be 6.0.x or greater

2.)  The EQL volume can't be replicated.  That's either Point-In-Time or SyncRep.

3.)  If you have EQL snapshots, the space may not be returned until the snapshots still holding the unmapped blocks are deleted.

4.)  Try it on the ESX console directly.

       cd /vmfs/volumes/<datastore name>

       [tag:vmkfstools] -y %

VMware KB:

kb.vmware.com/.../search.do

You may have to try larger percentages.  Just be mindful of not using all free space on the VMFS datastore.  

The ESX unmap process first requires a "baloon" file to be created, then immediately deleted.  Then the unmap commands are sent for those LBAs.

Social Media and Community Professional
#IWork4Dell
Get Support on Twitter - @dellcarespro

Highlighted

RE: Reclaim Not Working on Equallogic Volumes?

I'm having the same issue.

ESXi 5.5 installed on Cisco B200 M3 blades

vCenter 5.5

EqualLogic PS6000 and PS6110VX (3.5) storage arrays running v7.0.1 firmware

I've ran both commands and it doesn't seem to work.

vmkfstools -y /vmfs/volumes/SAN01/

esxcli storage vmfs unamp -l SAN01

the "vmkfstools -y" command has been deprecated and you can't specify a percentage.  It automatically runs the unmap command in blocks of 200 (or says it does)

# vmkfstools -y /vmfs/volumes/SAN02/

Try to unmap 235104 blocks in units of 200 blocks from volume /vmfs/volumes/SAN02/.

Async unmapped 235085 blocks from volume /vmfs/volumes/SAN02/.

But EQL Group Mgr doesn't show the changes.  I have a 600 GB LUN that is showing 100% used, but on VMware its showing 50% used.  Do I need to expand the LUN so it can create the balloon file?

Thank you in advance.

0 Kudos
Highlighted

RE: Reclaim Not Working on Equallogic Volumes?

No you don't need to expand the volume.   The GUI showing 100% full just means that a lot of data has been written to that volume and the EQL pages have been allocated.   The ESXi OS knows what blocks are actually available.

Are there any EQL snapshots on that volume?   Is the volume replicated?

The array absolutely supports UNMAP.  Any version over 6.0.x supports it.  It's up to the OS to tell the array what blocks to UNMAP.

In the past I had to specify a large percentage to get good results.  Since it appears that the baloon file it creates grabs the next availalbe block on the Datastore.  Not always one that needs to be UNMAPped.

The EQL GUI can show 100% in use on an empty volume as far as the OS is considered.  

The EQL array is a block storage device, it can record how much has been written to it.  But without an UNMAP block command being sent from the host, it can't unallocate a block on its own.  Block storage is not filesystem aware.  

If you are worried that the EQL array says the volume is full, then ignore that.   Go by what the OS says is the current in use space.   The OS will reuse blocks as needed.  The array will then re-use the block that's associated with it on the volume.   You are not actually out of space until the OS tells you, you are.

Social Media and Community Professional
#IWork4Dell
Get Support on Twitter - @dellcarespro

0 Kudos
Highlighted

RE: Reclaim Not Working on Equallogic Volumes?

Thanks Don,

there are no snapshots and it is not replicated.

What I may do is move any virtual machines on that LUN to a newly created LUN and then just delete the older LUN that is showing 100% used.

Thanks again,

0 Kudos
Highlighted

RE: Reclaim Not Working on Equallogic Volumes?

You are welcome.  Also feel free to open a support case on this.

The VMFS volume is VMFS v5 correct?   As in formatted with V5, not upgraded from V3 to VMFS v5?  Upgraded Datastored don't support UNMAP.  (I believe)

I will also try this myself.   I use ESXi v5.0 / 5.1 more than 5.5 right now.  I know it works with those releases just fine.

Did you try specifying blocks when you did the reclaim?

VAAI UNMAP Improvements

vSphere 5.5 introduced a new simpler VAAI UNMAP/Reclaim command

# esxcli storage vmfs unmap

This is a big improvement from the older vmkfstools -y method. There are some additional enhancements to the reclaim mechanism too. The reclaim size now specified in blocks rather than a percentage value to make it more intuitive to calculate. Finally, dead space reclaimed in increments rather than all at once. Previously, trying to do reclaim in increments would have unmap trying to reclaim the same space over and over again. Lastly, with the introduction of 62TB VMDKs, unmap can now handle much larger dead space areas.

Social Media and Community Professional
#IWork4Dell
Get Support on Twitter - @dellcarespro

0 Kudos
Highlighted

RE: Reclaim Not Working on Equallogic Volumes?

ahh, that could be.  I believe these were originally VMFS3 datastores and I upgraded to VMFS5.

Will do some testing on my side to make sure unmap is working (will recreate the volumes).

If it still doesn't work then I'll open a support case.

Thanks Don!

0 Kudos
Highlighted

RE: Reclaim Not Working on Equallogic Volumes?

You are very welcome!   I hope that's all it is.  

There are some other features lost when you convert to V5 vs. reformat Datastores to V5.  The "sub block" performance enhancement is one of them.  Especially if the V3 was formatted with a greater than 1MB blocksize.

Good luck!

Social Media and Community Professional
#IWork4Dell
Get Support on Twitter - @dellcarespro

0 Kudos
Highlighted
3 Argentum

RE: Reclaim Not Working on Equallogic Volumes?

I have been delayed in getting back to this. These volumes are not replicated, no snapshots exist. This is firmware 6.0x. This used to work fine in previous versions of ESXi (4.5 and 5.1). Since going to 5.5 with new unmap command is where it is no longer working, since we dont specify percentages any longer. These volumes are also native vmfs5.

I am running the command with the defaults which I believe is 200.

0 Kudos
Highlighted

RE: Reclaim Not Working on Equallogic Volumes?

I would suggest trying a larger number than default.   With VMKFSTOOLS I noticed that I had to use a larger % to get good results.  

Social Media and Community Professional
#IWork4Dell
Get Support on Twitter - @dellcarespro

0 Kudos
Highlighted

RE: Reclaim Not Working on Equallogic Volumes?

Just re-tested this on a native VMFS5 datastore.  Works beautifully!

0 Kudos
Highlighted
2 Iron

RE: Reclaim Not Working on Equallogic Volumes?

I have a volume that is almost full. The traditional method is to map the VMDK files into another VM and zeros up them in that VM (run sdelete in Windows on these drives etc) which is very time consuming.

The recommended method (if you have storage vMotion license) is to migrate all the VMs into a new EQL volume and delete the previous volume.

Since I have recently upgraded our vSphere 4.1 to 5.5, I would like to try out the command: esxcli storage vmfs unmap.

However, I'm got quite confused over what has been discussed above. How does VMFS3 and VMFS5 influence the process of reclaiming the deleted space? And what is the impact for VMFS5 upgraded from VMFS3 vs fresh VMFS5 volume? My EQL volumes are still in VMFS3. Must I upgrade them to VMFS5 first before executing "esxcli storage vmfs unmap"?

0 Kudos
Highlighted

RE: Reclaim Not Working on Equallogic Volumes?

VMFS v3 does not support UNMAP.  If you upgrade a VMFS v3 to v5 that too does NOT support UNMAP.  It has to be a non EQL replicated volume, FW 6.0.x or greater, formatted natively with VMFS v5 format.

Social Media and Community Professional
#IWork4Dell
Get Support on Twitter - @dellcarespro

Highlighted

RE: Reclaim Not Working on Equallogic Volumes?

Hello. Thank you for the answer.

For me it doesn't work, and it could be because I have configured replication for the volume (i.e. requirement number 2 on your list is not satisfied). Must I remove replication for that volume temporarily and then re-configure it?

0 Kudos
Highlighted

RE: Reclaim Not Working on Equallogic Volumes?

Correct, if you have sync, or async replication enabled on a volume, reclaim will NOT work.  You would have to disable replication in order to support unmap.  However, then you would have to re-replicate the entire volume again when you re-enable replication

Social Media and Community Professional
#IWork4Dell
Get Support on Twitter - @dellcarespro

0 Kudos