PowerFlex: OpenStack fails to delete volume following credential change
Summary: PowerFlex volume, associated with an OpenStack Instance (VM) deleted after a PowerFlex credential change, is not deleted from PowerFlex.
Symptoms
PowerFlex credentials used for OpenStack integration changed in the PowerFlex backend (MDM).
PowerFlex credentials are correct in the OpenStack Cinder configuration file, cinder.conf.
Cinder Volume, API, and Scheduler services restarted following the configuration change.
Existing OpenStack instance with PowerFlex volumes attached, created before the credential change.
Symptoms
Volumes associated with deleted instances remain provisioned in PowerFlex.
Volumes associated with deleted instances are marked as error_deleting in the output of openstack volume list, similar to the following:
+--------------------------------------+--------------+----------------+------+-------------+
| ID | Display Name | Status | Size | Attached to |
+--------------------------------------+--------------+----------------+------+-------------+
| 6b7ce738-5a28-40ff-8ffa-2c98f6a051ad | vol-1 | error_deleting | 24 | |
+--------------------------------------+--------------+----------------+------+-------------+
Impact
Volumes remain provisioned after instance deletion.
Volumes remain listed in Cinder after their associated instance has been removed.
Cause
Upon volume attachment to an instance, OpenStack Nova stores in its database Cinder-related information, including the ScaleIO credentials used at the time of attachment.
Nova does not update or refresh this information upon Cinder configuration changes.
After the credentials are changed in the ScaleIO backend, Nova attempts to interact with ScaleIO using the stored, old, credentials, which are rejected.
Instances created after the Cinder configuration change are not affected as, for these, Nova stores the updated credentials in its database.
Resolution
For volumes associated with deleted instances, manually unmap and delete from ScaleIO, using the ScaleIO CLI, UI, or other means.
To delete Cinder entries associated with such volumes, contact the OpenStack vendor.
For existing instances, created before the credential change, shelve and unshelve the instances.
This causes Nova to refresh its stored Cinder data.
Impacted Versions
OpenStack Newton and above
OpenStack distributions based on OpenStack Newton and above