PowerFlex: OpenStack: Volume size mismatch when the image_volume_cache_enabled is True

The following two symptoms are observed when the option "image_volume_cache_enabled" is used in an OpenStack/RHOSP (Red Hat OpenStack Platform) environment.

Symptom 1: 
The first time a bootable volume from a glance image is created, the created volume size always becomes 8 GB on the OpenStack side.

Symptom 2: 
The second time onwards of bootable volume creations from image-volume cache, the size of created volumes was not rounded to 8 GB on the OpenStack side.
 

In OpenStack/RHOSP with the PowerFlex Cinder backend storage

The option "image_volume_cache_enabled = True" is specified in cinder.conf, and bootable volumes are created.

Symptom 1: 
As an example, when a 33 GB bootable volume creation was initially requested, both OpenStack and PowerFlex expected to have the same 40 GB size volume. However, the 8 GB volume was created on the OpenStack side.

$ openstack volume list --long
$
$ openstack volume create --image cirros --type sio --size 33 33gb-1
$ openstack volume list --long
+--------------------------------------+--------------------------------------------+-------
| ID                                   | Name                                       | Status    | Size | Type | Bootable | Attached to | Properties      |
+--------------------------------------+--------------------------------------------+-------
| fed72292-fd84-4b33-bf63-063f1bfb9f75 | 33gb-1                                     | available |    8 | sio  | true     |             |                 | <<< 40 was expected but is 8.
| 364b78a7-0714-4b70-be16-8ec139564cb0 | image-b7d6350e-f835-4310-a5f4-6b6535ce86c4 | available |    8 | sio  | false    |             | readonly='True' | <<< This is the volume-image cache (snapshot).
+--------------------------------------+--------------------------------------------+-------

# scli --query_all_volumes
Volume ID: b582f85000000000 Name: mUnBlKjqSfur1KQillkMEw== Size: 40.0 GB (40960 MB) Not mapped Thin-provisioned <<< This is the corespoinding volume to "33gb-1".
Volume ID: b582f85100000001 Name: aWB3fINbQ7GV4BMlUXsOPQ== Size: 8.0 GB (8192 MB) Not mapped Snapshot of mUnBlKjqSfur1KQillkMEw==

Symptom 2:  
As an example, when a 50 GB bootable volume creation is requested while a volume-image cache is enabled, both OpenStack and PowerFlex expect to have the same 56 GB size volume. However, the 50 GB volume was created on the OpenStack side.

$ openstack volume create --image cirros --type sio --size 50 50gb-1
$ openstack volume list --long
+--------------------------------------+--------------------------------------------+-------
| ID                                   | Name                                       | Status    | Size | Type | Bootable | Attached to | Properties      |
+--------------------------------------+--------------------------------------------+-------
| 1f0a279f-9c3d-42a2-ba9c-f31fa160c92c | 50gb-1                                     | available |   50 | sio  | true     |             |                 | <<< 56 was expected but is 50.
| fed72292-fd84-4b33-bf63-063f1bfb9f75 | 33gb-1                                     | available |    8 | sio  | true     |             |                 |
| 364b78a7-0714-4b70-be16-8ec139564cb0 | image-b7d6350e-f835-4310-a5f4-6b6535ce86c4 | available |    8 | sio  | false    |             | readonly='True' |
+--------------------------------------+--------------------------------------------+-------

# scli --query_all_volumes
Volume ID: b582f85000000000 Name: mUnBlKjqSfur1KQillkMEw== Size: 40.0 GB (40960 MB) Not mapped Thin-provisioned
Volume ID: b582f85100000001 Name: aWB3fINbQ7GV4BMlUXsOPQ== Size: 8.0 GB (8192 MB) Not mapped Snapshot of mUnBlKjqSfur1KQillkMEw==
Volume ID: b582f85200000002 Name: cYLfBQJFRHKUjNjEtQpyAw== Size: 56.0 GB (57344 MB) Not mapped Snapshot of aWB3fINbQ7GV4BMlUXsOPQ== <<< This is the corresponding volume to "50gb-1".

Impact

Unable to manage backend storage capacity and volume size correctly from the OpenStack side.

Symptom 1: 
This is not a Cinder driver issue but an OpenStack Cinder internal mechanism. The issue is that Cinder does not expect storage to return a different size of disks than what a user specifies.

To resolve it, fix the Cinder cinder/volume/flows/manager/create_volume.py file. It must be aware of possible rounding by cinder driver or backend storage and to correct corresponding test suites.

Issue tracking #1915015 was opened for the OpenStack Cinder community to address the issue. cf. Bug #1915015 "PowerFlex volume has invalid size after creation f…”: Bugs: Cinder 

Symptom 2: 
This is a Cinder driver issue with the create_volume_from_snapshot function. The create_cloned_volume/extend_volume has the same issue.

There is an option to fix it in the Cinder driver by returning a real volume size, but that way could not pass the Tempest test cycles due to another issue on the Tempest side.

For now, the Tempest issue must be fixed first in order to implement a fix in the Cinder driver.

Issue tracking #1917299 was opened for the OpenStack Tempest community to address the issue. cf. Bug #1917299 "Need chunk size configuration for volume creation”: Bugs : tempest 

Note: This is not related to symptom 1. Users theoretically experience this issue even without the image_volume_cache_enabled option but when the above functions are called.

This is not a PowerFlex issue. Contact the OpenStack vendor and share this Dell article and mention the issue tracking numbers.

Impacted Versions: 
All

Fixed In Version: 
To be determined, no estimated time to resolution.

References: 
Cinder Image-Volume cache: 
This is an optional parameter implemented in the Cinder component to dramatically improve the performance of creating a volume from an image cache by using the backend storage's snapshot feature. See OpenStack document: https://docs.openstack.org/cinder/latest/admin/blockstorage-image-volume-cache.html

Tempest: 
The OpenStack Integration Test Suite is not only used for regression testing but for qualification too. See OpenStack document: https://docs.openstack.org/tempest/latest/overview.html

When this article was written: 

• Symptom 1: No workaround with the image_volume_cache_enabled option when enabled
• Symptom 2: The only possible workaround is to always specify a volume size multiple of 8 GB.