PowerFlex 3.x Operations on vVol-Based VM Fail Due to Duplicated GUID

MDM events show that underlying operations for VASA operations, such as map_volume_to_sdc, are performed for an SDC other than the operation initiator.
For example, a clone operation was initiated from ESXi host to SDC ID 5f417bc300000005.
MDM events show a map_volume_to_sdc to SDC ID 5f417bbe00000000.

2020-11-23 15:34:27.656000:0033919:MDM_CLI_CONF_COMMAND_RECEIVED    INFO     Command map_volume_to_sdc received, User: 'admin'. [35392968] Volume: ID: d1e7e3ec00000029 mapped to SDC ID: 5f417bbe00000000 Multimapping allowed, Inform SDC
2020-11-23 15:34:27.660000:0033920:CLI_COMMAND_SUCCEEDED            INFO     Command map_volume_to_sdc succeeded. [35392968]

Scenario Two or more hosts have no hardware UUID set or have the same UUID. A host's hardware UUID can be examined using smbiosDump. The following output excerpt shows a host with no hardware UUID set.

  System Info: #1
    Manufacturer: "Inspur"
    Product: "SA5212M4"
    Version: "01"
    Serial: "123456789"
    UUID: undefined, but settable

Impact 

Operation on vVol-based VM occasionally fails.

As part of the VASA API, when hosts connect to a Storage Provider (PowerFlex VASA provider) a "context" is established.
To establish a context, the host sends a Globally Unique Identifier (GUID) to the provider and the provider responds with a session ID.
The session ID is then used in subsequent requests made by the host, such as createVirtualVolume (add volume) and bindVirtualVolume (map volume to SDC).
This session ID is used in the VASA protocol to determine which host initiated the request.

The host GUID sent when its context is established is generated, by ESXi, from the host's hardware UUID.

Later, when a VM operation is performed, the originating host sends the Provider a VASA API request, with the host's session ID.
The Provider then correlates the session ID to the corresponding host GUID, as it was received when the context was established.
The host GUID is used to determine the host's SDC GUID.

When the host ID is not unique, the Provider may perform the underlying operation on one of the other hosts which originally provided the same GUID as the operation originator.
In the case the host the operation was performed on is not the originating host, the result is not reflected on the originating host, which causes the VM operation to fail.

Example flow:

1. Host A has no hardware UUID set and, therefore, sends GUID FFFFFFFF-FFFF-FFFF-FFFF-FFFFFFFFFFFF to the Provider, when the context is established.
   It receives a session ID aaa.
2. Host B also has no hardware UUID set, sends the same GUID, FFFFFFFF-FFFF-FFFF-FFFF-FFFFFFFFFFFF and receives a session ID bbb.
3. Later, the user initiates a VM clone operation, which causes a bindVirtualVolume VASA API calls to be sent from host A.
   Host A sends the session ID aaa.
4. The Provider correlates session ID aaa with GUID FFFFFFFF-FFFF-FFFF-FFFF-FFFFFFFFFFFF
   GUID FFFFFFFF-FFFF-FFFF-FFFF-FFFFFFFFFFFF may occasionally be associated with host B, since both A and B provided this GUID.
5. The underlying operation for bindVirtualVolume is map volume to SDC.
   The Provider successfully maps the volume to host B and responds to host A that the operation was successful.
6. Host A does not have the volume mapped to it, which causes the VM clone operation to fail.

• Remove the Storage Provider configuration from vCenter.
• Set a unique hardware UUID for each host.
  For instructions, contact the hardware vendor.
• Re-register the PowerFlex VASA Provider.

Impacted Versions

VxFlex OS VASA Provider 3.0.x
PowerFlex VASA Provider 3.5.x

Fixed in Version

PowerFlex VASA Provider 3.6