Unsolved
This post is more than 5 years old
2 Intern
•
20.4K Posts
0
4227
Deleted data not reflected in Volume/Physical Capacity
Hello guys/gals,
I have a brand new cluster where i created 4 x 256G volumes. I presented these volumes to a vSphere cluster where i created 4 datastores. I then created a 40G VM on each datastore and ran some tests using IOmeter just to generate some workload.
Now i am done with my testing, i deleted my VMs from datastore (Volume Capacity in the GUI did not change, that's understandable). I then went ahead and deleted the actual datastores from vSphere cluster. Still Volume capacity nor Physical Capacity did not change, they are still displaying the same values as if my datastores were still in place and VMs were up and running. So next step i went ahead and unmapped volumes from vSphere cluster hoping that would trigger something but nada.
So the question is, does XtremIO run some kind of garbage collection job to actually reflect that data has been deleted from volumes ? Because there is nothing else on this XtremIO cluster but these 4 volumes.
Thank you
Kumar_A
727 Posts
0
July 28th, 2015 13:00
VMware does not run the unmap operation by default when you delete data on the datastores (or when you delete the datastore itself). In other words, the array does not know about the data being deleted from the host, unless the unmap command is seen by the array. This is not specific to XtremIO – but is applicable for any array that does thin provisioning.
You have the option of running the unmap operation manually or use the VSI plugin for XtremIO (free of cost) through which you can schedule (or run on an adhoc basis) the unmap commands for XtremIO storage.
Kumar_A
727 Posts
0
July 28th, 2015 14:00
When you delete the datastore in VMware, you are not telling the backend storage array that the corresponding space can be reclaimed. This is a manual operation that you need to do in VMware (VSI plugin automates this process).
dynamox
2 Intern
2 Intern
•
20.4K Posts
0
July 28th, 2015 14:00
so by deleting a datastore i am not deleting vmfs file system ?
dynamox
2 Intern
2 Intern
•
20.4K Posts
0
July 28th, 2015 14:00
I have some worklflows where file system could be deleted but volumes could be re-used for other purposes. So what would happen in my case if my 4 volumes would be presented to another ESX cluster and new datastores would be created ? Would the "ghost" data get overwritten or new data would be written somewhere else and my "ghost" data will be stranded forever ?
Also same scenario but with physical servers (EXT4, NTFS, XFS)
Thanks
Kumar_A
727 Posts
0
July 28th, 2015 14:00
The unmap operation has to be run before the datastores have been deleted. If you have nothing on the LUNs, why not delete the LUN itself?
dynamox
2 Intern
2 Intern
•
20.4K Posts
0
July 28th, 2015 14:00
datastores have been deleted, what are my options now (other then deleting volumes) ?
blubud
35 Posts
0
March 10th, 2016 06:00
Any new info on this? We are in the similar sitiuation. Is there a way to clean out old pointers from xtremio and reclaim space?
We moved our workloads onto xtremio last year but due to issues we had to move them back to Vmax. We have gotten the issues resolved and moved dev and test back but the xtremio showed as 64% full physically. The same amount when we had Prod, dev and test workloads on it last prior to the issues. We have started migrating Prod workload the past couple of days and we are up to 77% physical capacity.
Do I simply go an unmap/reclaim all xtremio volumes in Vsphere? We are using Eagered Thick volumes.
Kumar_A
727 Posts
0
March 10th, 2016 14:00
Are you running in a VMware environment and is the XtremIO behind a VPLEX by any chance?
Assuming that the answer is 'yes' and 'no', you can run and schedule the unmap command through the VSI plugin for XtremIO. Have you tried that?
If this is a non-VMware environment - there are OS specific operations that you can do to take care of space reclamation issue. This is documented in the Host Configuration Guide for XtremIO.
blubud
35 Posts
0
March 11th, 2016 05:00
No Vplex. I do have VSI. I thought the unmap commands were specifically used for thin provisioned volumes?
I will try iterating through my XIO volumes and running the unmap command. hopefully it works or we have to start the process of adding capacity.
Kumar_A
727 Posts
0
March 11th, 2016 08:00
VSI plugins will allow you to run the UNMAP commands on the XtremIO datastores. You can also schedule those unmap operations from the VSI plugin if you wanted to. Make sure you are using the latest version of the VSI plugin.
Kumar_A
727 Posts
0
March 15th, 2016 07:00
VSI plugin has a neat functionality of being able to schedule the unmap operations if you wanted to. So you don’t need to remember running the script every so often.
Obviously, it is difficult to triage the problem with VSI plugin in this forum. Can you submit an SR for this issue and let me know what the SR number is?
blubud
35 Posts
0
March 15th, 2016 07:00
I couldn't get the vsi plugin to load in my vsphere web client. I did however modify a community unmap script from Vmware and was able to reclaim the dead space. I am now back to 65% physical capacity. Thanks for the help.
blubud
35 Posts
0
March 15th, 2016 08:00
Also one thing I noticed when running the unmap commands was the latency went pretty high.We usually never saw anything higher than 3 ms on writes and maintained sub milliseconds on reads in xtremio with 22K write IOPs and 31K read iops. When running the unmap commands we saw latency spike to as high as 14 ms on writes and 18 ms on reads. Our IOPs were 14K on writes and 26K on reads during that time. Luckily we did them in small batches with a limited number of volumes.
Is it because of the way Xtremio does garbage collection?
blubud
35 Posts
0
March 15th, 2016 08:00
I had VSI running but I couldn't get it running after I deployed the latest version. When the projects slow down here I will attempt to get it back up and running and engage support if needed.
Kumar_A
727 Posts
0
March 15th, 2016 11:00
When you run the UNMAP command, there is work that the array needs to do – and it is indeed recommended that the UNMAP operations should be scheduled such that not all UNMAP operations are running at the same time.