PowerVault ME4: What is Overcommitment and how does it work?

Table of Contents

1. What is Overcommitment and how does it work?
2. Where to see Overcommitment information or issues in the logs?
3. What is the Metadata Index size?
4. What is the 16GiB address table?
5. What is the maximum overcommitment limit?
6. What is the difference between Shared Data, Unique Data, and Snap Data?
7. Should the Unique Data (show snapshots type all) and Allocated Size (show snapshot-space) values match?
8. How to calculate how much space is committed, and where it is committed to?

 

---

Question: What is Overcommitment and how does it work?

Answer:

Overcommitted means that the total committed size of all virtual volumes, including snapshots, exceeds the physical space in the virtual pool.

This feature is specific to Virtual Storage. NOT applicable for Linear.

This feature allows overcommitting physical storage resources enabling the operating system to operate as though it has more storage space than is physically allocated.

Every Volume, including snapshots, created in the Array consumes Metadata Indexes from the Virtual Pool storage space.

In general, the FW supports a maximum of 1 PiB of Metadata per Virtual Pool.

 

For example, consider a storage array with 100 TB of usable pool space on Pool A.

With "Overcommitment" Disabled, its ONLY possible create Volumes (including Snapshots) to a max of 100 TB, the configuration cannot go beyond 100 TB of Metadata Index.

With "Overcommitment" Enabled, the FW allows the creation of Volumes (including Snapshots) beyond the 100 TB Physical Limit, as the FW supports a max of 1 PiB of Metadata Index.

Thus, the system can create Volumes (including Snapshots) to a max of 1 PiB per Pool. This is described in further detail in the question, "What is the metadata index size?"

---

Question: Where to see Overcommitment information/issues in the logs?

Answer:

When overcommitment information is seen in the GUI, event IDs show in the event logs. Below are some example event codes.

Event ID 473 - The indicated volume is using more than its threshold percentage of its virtual pool. This is an indication that the storage usage crossed the user-specified threshold for this volume.

Event ID 573 - Warning Allocated snapshot space for a virtual pool cannot be reduced because no snapshots are deletable.

Event ID 571 - Error Allocated snapshot space exceeded the configured percentage limit of the virtual pool.

 

 

 

---

Question: What is the Metadata Index size? What is the 16GiB address table?

Answer:

The maximum supported number of metadata index entries is 65312. When calculating physically addressable space, multiply the Metadata Index entries by the address table.

This equates to approx. 1 PiB, which is 1,048,576 GiB.

 

---

Question: What is the maximum overcommitment limit?

Answer:

In general, the FW supports a maximum of 1 PiB of Metadata per Virtual Pool.

With the "Overcommitment" feature Disabled, the Pool ONLY allows using Metadata to a max of the storage space available.

With the "Overcommitment" feature Enabled, the Pool allows using Metadata to a max of 1 PiB per Pool which is the FW limit.

 

---

Question: What is the difference between Shared Data, Unique Data, and Snap Data?

Answer:

Snap Data: The total amount of write data associated with the snapshot.

In other words: the Volume Allocation Information for the parent Volume of the Snapshot when the snapshot was originally taken. That is the committed size of the volume when the snapshot was taken.

Unique Data: The amount of write data that is unique to the snapshot.

In other words: this is the data that is changed or modified from the original volume

Shared Data: The amount of write data that is shared between this snapshot and other snapshots.

In other words: This is the data that is not changed or modified from the original volume when the snapshot is taken.

Also, "Shared Data + Unique Data" is approximately equal to the "Snap Data."

 

---

Question: Should the Unique Data (show snapshots type all) and Allocated Size (show snapshot-space) values match?

Answer:

As per the design, Unique Data (modified data) should match the Allocated Size value. The unique data is the snapshot’s physical data size, this unique data uses the snapshot space.

For Configurations with ONLY snapshots configured, the Sum of the Unique Data of all snapshots equals the Total Snapshot allocated size that is reported.

For Configurations with Replication configured, the Sum of the Unique Data of all Snapshots associated with standard snapshots plus the Unique Data of the snapshots associated with Replication tasks.

However, in general, the show snapshot output does not report the "Snap Data" or "Shared Data" or "Unique Data". This is why "show snapshots type all" command is used.

Example:

For configurations with Replication setup, the FW maintains three snapshots for each Replication Task (S1, S2, and S3).

The value for the snapshot "S3" is what is replicated after the initial replication.

On the subsequent Replication schedule, when S3 becomes S2 the "show snapshots" data reports all the associated data information in the Table.

While replication is occurring, the values of each "S3" are not reported in the logs. Also, there are no CLI commands to report this information.  This means that when replication is occurring, a discrepancy between Unique Data and Allocated Size, may be viewed.

This is the expected behavior.

 

---

Question: How to calculate how much space is committed, and where it is committed to?

Answer:

Committed space is the size of all virtual volumes + snapshot metadata size.

The GUI does not show where all the space is committed per volumes or snapshots. This must be calculated from logs and/or CLI commands.

The output from "show pools" and "show volumes" is needed.

 

Example:

Screenshot from GUI information showing overcommitment on Pool A but none on Pool B

 

Screenshot of more detail on Pool A

 

Example output from "show pools".

 

Example output from "show volumes" – truncated or modified to show where data comes from for following tables and calculations

 

Steps to calculate Overcommitment

Use the following values from the commands, setting up in a table or excel is suggested.

                show pools = Total Size, Avail, Snap Size

                show volumes = Total Size, Alloc Size

 

To break down the data into usable form to show where data is committed.  Below is a table of what fields to look at and where to pull the data from.

Name	CLI command	Column/Field	Modifier 1	Modifier 2	Calculation
Total Size	show pools	Total Size		Convert to a common denominator (MB/GB/TB)	N/A
Avail / Free	show pools	Avail		Convert to a common denominator (MB/GB/TB)	N/A
Snapshot Used	show pools	Snap Size		Convert to a common denominator (MB/GB/TB)	N/A
Volume Total Size	show volumes	Total Size	"base" items in Type Column	Convert to a common denominator (MB/GB/TB)	Add all tems together
Volume Allocated Size	show volumes	Alloc Size	"base" items in Type Column	Convert to a common denominator (MB/GB/TB)	Add all tems together
Total Snapshot Size	show volumes	Total Size	"snapshot" items in Type Column	Convert to a common denominator (MB/GB/TB)	Add all tems together
Snapshot Allocated	show volumes	Alloc Size	"snapshot" items in Type Column	Convert to a common denominator (MB/GB/TB)	Add all tems together

 

 

 

Example from the customer system

Title	Pool A (in GB)	Pool B (in GB)
Total Size	35200	35200
Avail / Free	29500	23900
Snapshot Used	1437.9	1305.3
Volume Total Size	12246.2	12578.4
Volume Allocated Size	4231.0952	9910.8845
Total Snapshot Size	29979.3	16762.9
Snapshot Allocated	16922.1	10974.2117

 

Now calculate the committed and/or overcommitted information per pool.

Committed:        Volume Total Size + Total Snapshot Size

Pool	Volume Total Size	Total Snapshot Size	Committed
A	12246.2	29979.3	42225.5
B	12578.4	16762.9	29341.3

 

Overcommitted:      Committed - Total Size

Pool	Committed	Total Size	Overcommitted
A	42225.5	35200	7025.5
B	29341.3	35200	-5858.7

 

 

 

From the example:

A is overcommitted by 7025.5 GB

B is not overcommitted as the number is negative.