Now that I can see the screenshot, it is what I thought. That graph indicates how much data has been written to that volume over time. When a file is deleted that is a WRITE to an allocation table that the OS maintains. On its own it does not tell the storage that space is no longer used. Newer OS's and filesystems support a feature known as "UNMAP" aka "Space reclaim". Then the OS will tell the backend storage that those blocks are now free. It requires support from both the OS and the backend storage device. EQL FW v6.+ supports UNMAP on NON replicated volumes. In Windows volumes replicated with DFS do NOT support UNMAP.
Here's a longer explanation from an earlier community forum post.
Solution Title ARRAY: GUI space usage differs from what the OS shows
Solution Details Q: Why is there a difference between what my file system shows as space used and what the PS array GUI shows for in-use for the volume?
A: The PS array is block-storage, and only knows about areas of a volume that have ever been written. The PS Series GUI reports this information for each volume. Volume allocation grows automatically due to application data writes. If later the application frees up space, the space is not marked as unused in the PS Series GUI. Hence the difference in views between the OS/file system and the PS Series GUI.
With thin provisioned volumes, this perception can be more pronounced.
Thin provisioning is a storage virtualization and provisioning model that allows administrators to logically allocate large addressable storage space to a volume, yet not physically commit storage resources to this space until it is used by an application. For example, using thin provisioning you can create a volume that an application views as 3 TB, while only allocating 300 GB of physical storage to it. As the operating system writes to the volume, physical disk space is allocated to the volume by the storage array. This physical disk space is taken from the available free space in the pool automatically and transparently. As a result, less physical storage is needed over time, and the stranded storage problem is eliminated. The administrator enjoys management benefits similar to over-provisioning, yet maintains the operational efficiencies of improved physical storage utilization. This more efficient use of physical storage resources typically allows an organization to defer or reduce storage purchases.
So Thin provisioning is a forward planning tool for storage allocation in which all the storage an application will need is allocated upfront, eliminating the trauma of expanding available storage in systems that do not support online expansion. Because the administrator initially provisions the application with all the storage it will need, repeated data growth operations are avoided.
Most important, because of the difference between reality and perception, anyone involved with thin provisioned storage must be aware of the duality in play. If all players are not vigilant someone could start drawing on the un-provisioned storage – exceeding capacity, disrupting operations, or requiring additional unplanned capital investments.
A thin-provisioned volume also grows automatically due to application data writes – the space is drawn from the pool free space (rather than having been pre-allocated in a normal volume). If later the application frees up space, the space is free in the file system but is not returned to the free space in the PS Series pool. The only way to reduce the physical allocation in the SAN is to create a new volume, copy the application data from the old volume to the new, and then delete the old volume.
A similar problem is when the initiator OS reports significantly more space in use than the array does. This can be pronounced in systems like VMWare that create large, sparse files. In VMWare, if you create yourself a 10GB disk for a VM as a VMDK file, VMWare does not write 10GB of zeros to the file. It creates an empty (sparse) 10GB file, and subtracts 10GB from free space. The act of creating the empty file only touches a few MB of actual sectors on the disk. So VMWare says 10GB missing, but the array says, perhaps, only 2MB written to.
Since the minimum volume reserve for any volume is 10%, the filesystem has a long way to go before the MB-scale writes catch up with the minimum reservation of a volume. For instance, a customer with a 100GB volume might create 5 VMs with 10GB disks. That's 50GB used according to VMWare, but only perhaps 5 x 2MB (10MB) written to the array. Until the customer starts filling the VMDK files with actual data, the array won't know anything is there. If has no idea what VMFS is; it only knows what's been written to the volume.
• Example: A file share is thin-provisioned with 1 TB logical size. Data is placed into the volume so that the physical allocation grows to 500 GB. Files are deleted from the file system, reducing the reported file system in use to 100 GB. The remaining 400 GB of physical storage remains allocated to this volume in the SAN.
• This issue can also occur with maintenance operations including defragmentation, database re-organization, and other application operations.
In most environments, file systems do not dramatically reduce in size, so this issue occurs infrequently. Also some file systems will not make efficient re-use of previously allocated space, and may not reuse deleted space until it runs out of unused space (this is not an issue for NTFS, VMFS).
A related question is how can this previously used, but now not-in-use space be reclaimed in the SAN?
Today this would require the creation of a new volume and copying the volume data from old to new. This likely would require the application/users that use the file system to be offline (a planned maintenance window). In the future, file systems such as NTFS in Windows Server 2008 will allow online shrink in addition to the existing online grow operations. The result with this procedure would be the ability to reclaim free space would be done online (perhaps during non-peak times), where the file system is shrunk online, then re-grown online to its original size. This would reclaim the volume space, however there may be a delay in gaining the free space back to the pool free space if snapshots are present, as they would hold the released space until the snapshots age (and are automatically deleted).
In some cases the amount of space used on the array will show LESS than what is shown by the OS. For example, VMware ESX. When ESX 3.x creates a 20GB VMDK, it doesn't actually write 20GB of data to the volume. A small file is created, then ESX tells the file allocation table that 20GB has been allocated. Over time as data is actually written and deleted, this will swing back the other way. Where ESX says there's less space allocated than what the array GUI indicates.
dwilliam62
1 Rookie
1 Rookie
•
1.5K Posts
0
November 20th, 2019 09:00
Hello,
The screenshot currently isn't being displayed, but in use space on any volume is the total amount of data written to that volume.
Regards,
Don
dwilliam62
1 Rookie
1 Rookie
•
1.5K Posts
0
November 20th, 2019 12:00
Hello,
Now that I can see the screenshot, it is what I thought. That graph indicates how much data has been written to that volume over time. When a file is deleted that is a WRITE to an allocation table that the OS maintains. On its own it does not tell the storage that space is no longer used. Newer OS's and filesystems support a feature known as "UNMAP" aka "Space reclaim". Then the OS will tell the backend storage that those blocks are now free. It requires support from both the OS and the backend storage device. EQL FW v6.+ supports UNMAP on NON replicated volumes. In Windows volumes replicated with DFS do NOT support UNMAP.
Here's a longer explanation from an earlier community forum post.
Solution Title ARRAY: GUI space usage differs from what the OS shows
Solution Details Q: Why is there a difference between what my file system shows as space used and what the PS array GUI shows for in-use for the volume?
A: The PS array is block-storage, and only knows about areas of a volume that have ever been written. The PS Series GUI reports this information for each volume. Volume allocation grows automatically due to application data writes. If later the application frees up space, the space is not marked as unused in the PS Series GUI. Hence the difference in views between the OS/file system and the PS Series GUI.
With thin provisioned volumes, this perception can be more pronounced.
Thin provisioning is a storage virtualization and provisioning model that allows administrators to logically allocate large addressable storage space to a volume, yet not physically commit storage resources to this space until it is used by an application. For example, using thin provisioning you can create a volume that an application views as 3 TB, while only allocating 300 GB of physical storage to it. As the operating system writes to the volume, physical disk space is allocated to the volume by the storage array. This physical disk space is taken from the available free space in the pool automatically and transparently. As a result, less physical storage is needed over time, and the stranded storage problem is eliminated. The administrator enjoys management benefits similar to over-provisioning, yet maintains the operational efficiencies of improved physical storage utilization. This more efficient use of physical storage resources typically allows an organization to defer or reduce storage purchases.
So Thin provisioning is a forward planning tool for storage allocation in which all the storage an application will need is allocated upfront, eliminating the trauma of expanding available storage in systems that do not support online expansion. Because the administrator initially provisions the application with all the storage it will need, repeated data growth operations are avoided.
Most important, because of the difference between reality and perception, anyone involved with thin provisioned storage must be aware of the duality in play. If all players are not vigilant someone could start drawing on the un-provisioned storage – exceeding capacity, disrupting operations, or requiring additional unplanned capital investments.
A thin-provisioned volume also grows automatically due to application data writes – the space is drawn from the pool free space (rather than having been pre-allocated in a normal volume). If later the application frees up space, the space is free in the file system but is not returned to the free space in the PS Series pool. The only way to reduce the physical allocation in the SAN is to create a new volume, copy the application data from the old volume to the new, and then delete the old volume.
A similar problem is when the initiator OS reports significantly more space in use than the array does. This can be pronounced in systems like VMWare that create large, sparse files. In VMWare, if you create yourself a 10GB disk for a VM as a VMDK file, VMWare does not write 10GB of zeros to the file. It creates an empty (sparse) 10GB file, and subtracts 10GB from free space. The act of creating the empty file only touches a few MB of actual sectors on the disk. So VMWare says 10GB missing, but the array says, perhaps, only 2MB written to.
Since the minimum volume reserve for any volume is 10%, the filesystem has a long way to go before the MB-scale writes catch up with the minimum reservation of a volume. For instance, a customer with a 100GB volume might create 5 VMs with 10GB disks. That's 50GB used according to VMWare, but only perhaps 5 x 2MB (10MB) written to the array. Until the customer starts filling the VMDK files with actual data, the array won't know anything is there. If has no idea what VMFS is; it only knows what's been written to the volume.
• Example: A file share is thin-provisioned with 1 TB logical size. Data is placed into the volume so that the physical allocation grows to 500 GB. Files are deleted from the file system, reducing the reported file system in use to 100 GB. The remaining 400 GB of physical storage remains allocated to this volume in the SAN.
• This issue can also occur with maintenance operations including defragmentation, database re-organization, and other application operations.
In most environments, file systems do not dramatically reduce in size, so this issue occurs infrequently. Also some file systems will not make efficient re-use of previously allocated space, and may not reuse deleted space until it runs out of unused space (this is not an issue for NTFS, VMFS).
A related question is how can this previously used, but now not-in-use space be reclaimed in the SAN?
Today this would require the creation of a new volume and copying the volume data from old to new. This likely would require the application/users that use the file system to be offline (a planned maintenance window). In the future, file systems such as NTFS in Windows Server 2008 will allow online shrink in addition to the existing online grow operations. The result with this procedure would be the ability to reclaim free space would be done online (perhaps during non-peak times), where the file system is shrunk online, then re-grown online to its original size. This would reclaim the volume space, however there may be a delay in gaining the free space back to the pool free space if snapshots are present, as they would hold the released space until the snapshots age (and are automatically deleted).
In some cases the amount of space used on the array will show LESS than what is shown by the OS. For example, VMware ESX. When ESX 3.x creates a 20GB VMDK, it doesn't actually write 20GB of data to the volume. A small file is created, then ESX tells the file allocation table that 20GB has been allocated. Over time as data is actually written and deleted, this will swing back the other way. Where ESX says there's less space allocated than what the array GUI indicates.
Regards,
Don