Deleted data not reflected in Volume/Physical Capacity

VMware does not run the unmap operation by default when you delete data on the datastores (or when you delete the datastore itself). In other words, the array does not know about the data being deleted from the host, unless the unmap command is seen by the array. This is not specific to XtremIO – but is applicable for any array that does thin provisioning.

You have the option of running the unmap operation manually or use the VSI plugin for XtremIO (free of cost) through which you can schedule (or run on an adhoc basis) the unmap commands for XtremIO storage.

When you delete the datastore in VMware, you are not telling the backend storage array that the corresponding space can be reclaimed. This is a manual operation that you need to do in VMware (VSI plugin automates this process).

so by deleting a datastore i am not deleting vmfs file system ? 

I have some worklflows where file system could be deleted but volumes could be re-used for other purposes.  So what would happen in my case if my 4 volumes would be presented to another ESX cluster and new datastores would be created ? Would the "ghost" data get overwritten or new data would be written somewhere else and my "ghost" data will be stranded forever ?

Also same scenario but with physical servers (EXT4, NTFS, XFS)

Thanks

The unmap operation has to be run before the datastores have been deleted. If you have nothing on the LUNs, why not delete the LUN itself?

datastores have been deleted, what are my options now (other then deleting volumes) ?

Any new info on this?  We are in the similar sitiuation. Is there a way to clean out old pointers from xtremio and reclaim space?

We moved our workloads onto xtremio last year but due to issues we had to move them back to Vmax. We have gotten the issues resolved and moved dev and test back but the xtremio showed as 64%  full physically. The same amount when we had Prod, dev and test workloads on it last prior to the issues. We have started migrating Prod workload the past couple of days and we are up to 77% physical capacity.

Do I simply go an unmap/reclaim all xtremio volumes in Vsphere? We are using Eagered Thick volumes.

Are you running in a VMware environment and is the XtremIO behind a VPLEX by any chance?

Assuming that the answer is 'yes' and 'no', you can run and schedule the unmap command through the VSI plugin for XtremIO. Have you tried that?

If this is a non-VMware environment - there are OS specific operations that you can do to take care of space reclamation issue. This is documented in the Host Configuration Guide for XtremIO.

No Vplex.  I do have VSI. I thought the unmap commands were specifically used for thin provisioned volumes?

I will try iterating through my XIO volumes and running the unmap command. hopefully it works or we have to start the process of adding capacity.

VSI plugins will allow you to run the UNMAP commands on the XtremIO datastores. You can also schedule those unmap operations from the VSI plugin if you wanted to. Make sure you are using the latest version of the VSI plugin.

VSI plugin has a neat functionality of being able to schedule the unmap operations if you wanted to. So you don’t need to remember running the script every so often.

Obviously, it is difficult to triage the problem with VSI plugin in this forum. Can you submit an SR for this issue and let me know what the SR number is?

I couldn't get the vsi plugin to load in my vsphere web client. I did however modify a community unmap script from Vmware  and was able to reclaim the dead space. I am now back to 65% physical capacity. Thanks for the help.

Also one thing I noticed when running the unmap commands was the latency went pretty high.We usually never saw anything higher than 3 ms on writes and maintained sub milliseconds on reads in xtremio with 22K write IOPs and 31K read iops. When running the unmap commands we saw latency spike to as high as 14 ms on writes and 18 ms on reads. Our IOPs were 14K on writes and 26K on reads during that time.  Luckily we did them in small batches with a limited number of volumes.

Is it because of the way Xtremio does garbage collection?

I had VSI running but I couldn't get it running after I deployed the latest version. When the projects slow down here I will attempt to get it back up and running and engage support if needed.

When you run the UNMAP command, there is work that the array needs to do – and it is indeed recommended that the UNMAP operations should be scheduled such that not all UNMAP operations are running at the same time.