3 Argentum

Reclaim Not Working on Equallogic Volumes?

We have 3 different SAN vendors. On two of those SAN vendors the new esxcli unmap command with esxi 5.5 works fine. However, when running that command against volumes on the Equallogic arrays (thin provisioned volumes SAN side), the command never actually works. It completes without warning, but even after 24 hours, no volume space has been reclaimed.

I am hoping I can be provided with some insight?

The command I am using is the following:

esxcli --server=servername --username=root --password=password storage vmfs unmap --volume-label=san_volume

0 Kudos
14 Replies

RE: Reclaim Not Working on Equallogic Volumes?

There are a couple of requirements for SCSI UNMAP to work.

1.) EQL FW has to be 6.0.x or greater

2.)  The EQL volume can't be replicated.  That's either Point-In-Time or SyncRep.

3.)  If you have EQL snapshots, the space may not be returned until the snapshots still holding the unmapped blocks are deleted.

4.)  Try it on the ESX console directly.

       cd /vmfs/volumes/<datastore name>

       [tag:vmkfstools] -y %

VMware KB:

kb.vmware.com/.../search.do

You may have to try larger percentages.  Just be mindful of not using all free space on the VMFS datastore.  

The ESX unmap process first requires a "baloon" file to be created, then immediately deleted.  Then the unmap commands are sent for those LBAs.

Social Media and Community Professional
#IWork4Dell
Get Support on Twitter - @dellcarespro

RE: Reclaim Not Working on Equallogic Volumes?

I'm having the same issue.

ESXi 5.5 installed on Cisco B200 M3 blades

vCenter 5.5

EqualLogic PS6000 and PS6110VX (3.5) storage arrays running v7.0.1 firmware

I've ran both commands and it doesn't seem to work.

vmkfstools -y /vmfs/volumes/SAN01/

esxcli storage vmfs unamp -l SAN01

the "vmkfstools -y" command has been deprecated and you can't specify a percentage.  It automatically runs the unmap command in blocks of 200 (or says it does)

# vmkfstools -y /vmfs/volumes/SAN02/

Try to unmap 235104 blocks in units of 200 blocks from volume /vmfs/volumes/SAN02/.

Async unmapped 235085 blocks from volume /vmfs/volumes/SAN02/.

But EQL Group Mgr doesn't show the changes.  I have a 600 GB LUN that is showing 100% used, but on VMware its showing 50% used.  Do I need to expand the LUN so it can create the balloon file?

Thank you in advance.

0 Kudos

RE: Reclaim Not Working on Equallogic Volumes?

No you don't need to expand the volume.   The GUI showing 100% full just means that a lot of data has been written to that volume and the EQL pages have been allocated.   The ESXi OS knows what blocks are actually available.

Are there any EQL snapshots on that volume?   Is the volume replicated?

The array absolutely supports UNMAP.  Any version over 6.0.x supports it.  It's up to the OS to tell the array what blocks to UNMAP.

In the past I had to specify a large percentage to get good results.  Since it appears that the baloon file it creates grabs the next availalbe block on the Datastore.  Not always one that needs to be UNMAPped.

The EQL GUI can show 100% in use on an empty volume as far as the OS is considered.  

The EQL array is a block storage device, it can record how much has been written to it.  But without an UNMAP block command being sent from the host, it can't unallocate a block on its own.  Block storage is not filesystem aware.  

If you are worried that the EQL array says the volume is full, then ignore that.   Go by what the OS says is the current in use space.   The OS will reuse blocks as needed.  The array will then re-use the block that's associated with it on the volume.   You are not actually out of space until the OS tells you, you are.

Social Media and Community Professional
#IWork4Dell
Get Support on Twitter - @dellcarespro

0 Kudos

RE: Reclaim Not Working on Equallogic Volumes?

Thanks Don,

there are no snapshots and it is not replicated.

What I may do is move any virtual machines on that LUN to a newly created LUN and then just delete the older LUN that is showing 100% used.

Thanks again,

0 Kudos

RE: Reclaim Not Working on Equallogic Volumes?

You are welcome.  Also feel free to open a support case on this.

The VMFS volume is VMFS v5 correct?   As in formatted with V5, not upgraded from V3 to VMFS v5?  Upgraded Datastored don't support UNMAP.  (I believe)

I will also try this myself.   I use ESXi v5.0 / 5.1 more than 5.5 right now.  I know it works with those releases just fine.

Did you try specifying blocks when you did the reclaim?

VAAI UNMAP Improvements

vSphere 5.5 introduced a new simpler VAAI UNMAP/Reclaim command

# esxcli storage vmfs unmap

This is a big improvement from the older vmkfstools -y method. There are some additional enhancements to the reclaim mechanism too. The reclaim size now specified in blocks rather than a percentage value to make it more intuitive to calculate. Finally, dead space reclaimed in increments rather than all at once. Previously, trying to do reclaim in increments would have unmap trying to reclaim the same space over and over again. Lastly, with the introduction of 62TB VMDKs, unmap can now handle much larger dead space areas.

Social Media and Community Professional
#IWork4Dell
Get Support on Twitter - @dellcarespro

0 Kudos

RE: Reclaim Not Working on Equallogic Volumes?

ahh, that could be.  I believe these were originally VMFS3 datastores and I upgraded to VMFS5.

Will do some testing on my side to make sure unmap is working (will recreate the volumes).

If it still doesn't work then I'll open a support case.

Thanks Don!

0 Kudos

RE: Reclaim Not Working on Equallogic Volumes?

You are very welcome!   I hope that's all it is.  

There are some other features lost when you convert to V5 vs. reformat Datastores to V5.  The "sub block" performance enhancement is one of them.  Especially if the V3 was formatted with a greater than 1MB blocksize.

Good luck!

Social Media and Community Professional
#IWork4Dell
Get Support on Twitter - @dellcarespro

0 Kudos
3 Argentum

RE: Reclaim Not Working on Equallogic Volumes?

I have been delayed in getting back to this. These volumes are not replicated, no snapshots exist. This is firmware 6.0x. This used to work fine in previous versions of ESXi (4.5 and 5.1). Since going to 5.5 with new unmap command is where it is no longer working, since we dont specify percentages any longer. These volumes are also native vmfs5.

I am running the command with the defaults which I believe is 200.

0 Kudos

RE: Reclaim Not Working on Equallogic Volumes?

I would suggest trying a larger number than default.   With VMKFSTOOLS I noticed that I had to use a larger % to get good results.  

Social Media and Community Professional
#IWork4Dell
Get Support on Twitter - @dellcarespro

0 Kudos