VPLEX: Failed to unclaim storage-volume which was deleted from the backend array.
Summary: There was a firmware command error received when attempting to unclaim a storage-volume which had been deleted from the backend array.
Symptoms
- An array volume "storage_volume_1" initially with VPDID "VPD83T3:514f0c55892000abc" was already removed from the VPLEX and backend array.
- A new storage volume using the same name as the removed storage volume, that is "storage_volume_1", was created and presented to VPLEX. The "naa:" Identifier is "514f0c55892000xyz" on the backend array.
Example:
VPlexcli:/clusters/cluster-2/storage-elements/storage-arrays/XtremIO-XtremApp-CKMxxxxxxxxx69/logical-units> ll
Name Connectivity Active/AAO Passive/AAN Visibility LUNs ALUA
------------------------ Status Controllers Controllers ---------- ------------------ Support
------------------------ ------------ -------------- ----------- ---------- ------------------ -------
VPD83T3:xxxxxxxxxxxxxx54 ok CKMxxxxxxxxx69 active 0x0008000000000000 none
- The storage array became "unknown" after creating the new storage volume on VPLEX. The "naa:" Identifier became "VPD83T3:514f0c55892000abc" which was the identifier of the previously removed array volume named "storage_volume_1".
- However the new storage volume was mapped to "naa:" Identifier "514f0c55892000xyz" using the prior storage-volume name, "storage-volume_1", when seen from the backend array, and attempting to unmap it issued an error.
- The newly created storage volume has "VPD83T3:514f0c5589200xyz" as its NAA ID when seen from backend.
- Technically a volume with an identifier ending with "abc" doesn't exist. While the new volume with an identifier ending with "xyz" is fine.
When the unclaim command was used to unclaim a storage-volume that had been removed on the backend array it failed to unclaim Storage Volume.
When trying to unclaim a particular storage volume from VPLEX, we receive an error "Firmware command error." and the cause of the failure as reported by the firmware
Example:
VPlexcli:/clusters/cluster-2/storage-elements/storage-volumes> unclaim -d <storage_volume_name> storage-volume unclaim: Evaluation of <<unclaim -d storage_volume_name>> failed. cause: Failed to unclaim storage-volumes. cause: Firmware command error. cause: Valid storage-volume, but storage-volume not known by device. << reason of failure
Sample Output:
VPlexcli:/clusters/cluster-2/storage-elements/storage-volumes> unclaim -d INFxxxxxxxxxxxxDS01 storage-volume unclaim: Evaluation of <<unclaim -d INFXXXXXXXXXXXXDS01>> failed. cause: Failed to unclaim storage-volumes. cause: Firmware command error. cause: Valid storage-volume, but storage-volume not known by device.
Cause
A storage volume name having a particular VPDID, which is required to be used with the unclaim command, was deleted from the storage array but was not unclaimed from VPLEX. However a new storage volume with the same name was created.
Example:
VPlexcli:/clusters/cluster-2/storage-elements/storage-volumes> ll
Name VPD83 ID Capacity Use Vendor IO Type Thin Provision Thin
----------------------------------------- ---------------------------------------- -------- --------- ------- Status ----------- Rebuild Type Capable
----------------------------------------- ---------------------------------------- -------- --------- ------- ------ ----------- ------- --------- -------
INFxxxxxxxxxxxxDS01 VPD83T3:xxxxxxxxxxxxxx12 10T claimed - alive normal true legacy -
Resolution
- For verification, log in to VPlexcli and check the hierarchy of the storage volume to be unclaimed.
Example:
VPlexcli:/clusters/cluster-2/storage-elements/storage-volumes> show-use-hierarchy <storage_volume_name>
storage-volume: <storage_volume_name> (size)
show-use-hierarchy clusters/cluster-2/storage-elements/storage-volumes/<storage_volume_name>
Sample output:
VPlexcli:/clusters/cluster-2/storage-elements/storage-volumes> show-use-hierarchy INFxxxxxxxxxxxxDS01
storage-volume: INFxxxxxxxxxxxxDS01 (10T)
- Restart the VplexManagementConsole server as shown below,
service@<ManagemetServer>:~> sudo /etc/init.d/VPlexManagementConsole restart
Restarting EMC VPlex Management Console...
After restarting the management console, wait for 1-2 minutes to allow the system to stabilize before attempting a new login to the VPlexcli again.
- Log in to the Vplexcli console and re-issue the command "
unclaim -d <storage_volume_name>"
Example:
VPlexcli:/clusters/cluster-2/storage-elements/storage-volumes> unclaim -d <storage_volume_name>
Unclaimed 1 of 1 storage-volumes.
Sample Output:
VPlexcli:/clusters/cluster-2/storage-elements/storage-volumes> unclaim -d INFxxxxxxxxxxxxDS01
Unclaimed 1 of 1 storage-volumes.
- Do a long listing, ' ll ', at the storage volume context. The <storage_volume_name> should not show up. If it does not this means that the volume that you had just unclaimed has been successfully unclaimed.