VPLEX Reports Too Many Paths to VNX Despite Correct Pathing
Summary: Connectivity validate-be states storage-volumes have more than supported (4) active paths. No configuration changes have been made and the zoning, and mapping/masking are confirmed to be good. ...
Symptoms
Generally this is seen issue following a VNX SP reboot, NDU, or other activity which results in a change in the path states. Following the change, VPLEX volumes log scsi/27 (06/2a/06): Unit Attention, ASYMMETRIC ACCESS STATE CHANGED which resulted in target port refreshes, followed by scsi/146 indicating volume having more than four active paths.
Impacted Hardware:
Hardware: VPLEX Series
Hardware: VPLEX VS1
Hardware: VPLEX VS2
Hardware: VPLEX Local
Hardware: VPLEX Metro
Hardware: VPLEX Geo
Impacted VPLEX GeoSynchrony:
Software: GeoSynchrony 4.0
Software: GeoSynchrony 4.0.1
Software: GeoSynchrony 4.1
Software: GeoSynchrony 4.2
Software: GeoSynchrony 4.2 Patch 1
Software: GeoSynchrony 5.0
Software: GeoSynchrony 5.0.1
Software: GeoSynchrony 5.0.1 Patch 1
Software: GeoSynchrony 5.0.1 Patch 2
Software: GeoSynchrony 5.1
Software: GeoSynchrony 5.1 Patch 1
Software: GeoSynchrony 5.1 Patch 2
Software: GeoSynchrony 5.1 Patch 3
Software: GeoSynchrony 5.1 Patch 4
Software: GeoSynchrony 5.2
Software: GeoSynchrony 5.2 Patch 1
Software: GeoSynchrony 5.2 Service Pack 1
Software: GeoSynchrony 5.2 Service Pack 1 HF 1
Software: GeoSynchrony 5.2 Service Pack 1 Patch 1
Software: GeoSynchrony 5.2 Service Pack 1 Patch 2
Software: GeoSynchrony 5.2 Service Pack 1 Patch 3
Software: GeoSynchrony 5.3
Software: GeoSynchrony 5.3 Patch 1
Software: GeoSynchrony 5.3 Patch 2
Software: GeoSynchrony 5.3 Patch 3
Software: GeoSynchrony 5.3 Patch 4
Software: GeoSynchrony 5.4 Patch 1
Software: GeoSynchrony 5.4 Service Pack 1
Software: GeoSynchrony 5.4 Service Pack 1 Patch 1
Software: GeoSynchrony 5.4 Service Pack 1 Patch 3
Software: GeoSynchrony 5.4 Service Pack 1 Patch 4
Backend connectivity checks find that there are more than the supported (4) active paths from the same director to VNX volumes. This issue generally does not affect all LUNs on an array and may present itself asymmetrically across the VPLEX directors.
Example:
Executing connectivity validate-be --detailed --verbose... WARNING: Storage volumes which have more than supported (4) active paths from the same director: Cluster Director Array Storage volumes which have more than --------- -------------- --------------------------------------------- supported (4) active paths from the same --------- -------------- --------------------------------------------- director --------- -------------- --------------------------------------------- ---------------------------------------- cluster-1 director-1-1-A (APM00xxxxxxxxx, EMC~CLARiiON~APM00xxxxxxxxx) VPD83T3:60060160xxxxxx0076675fbab698e311 VPD83T3:60060160xxxxxx0052a54dea9999e311 VPD83T3:60060160xxxxxx003264e853e4afe111 VPD83T3:60060160xxxxxx00a897d6399a99e311 VPD83T3:60060160xxxxxx0072675fbab698e311 VPD83T3:60060160xxxxxx0026a0820b2069e411 VPD83T3:60060160xxxxxx00d23db51bd191e211 VPD83T3:60060160xxxxxx00803a45d92403e211 VPD83T3:60060160xxxxxx003664e853e4afe111 VPD83T3:60060160xxxxxx0022a0820b2069e411 VPD83T3:60060160xxxxxx0006a11a122069e411 VPD83T3:60060160xxxxxx0060e43fe739d7e511 VPD83T3:60060160xxxxxx00783a45d92403e211 VPD83T3:60060160xxxxxx001ea0820b2069e411 cluster-1 director-1-1-B (APM00xxxxxxxxx, EMC~CLARiiON~APM00xxxxxxxxx) VPD83T3:60060160xxxxxx0076675fbab698e311 VPD83T3:60060160xxxxxx0052a54dea9999e311 VPD83T3:60060160xxxxxx003264e853e4afe111 VPD83T3:60060160xxxxxx00a897d6399a99e311 VPD83T3:60060160xxxxxx0072675fbab698e311 VPD83T3:60060160xxxxxx0026a0820b2069e411 VPD83T3:60060160xxxxxx00d23db51bd191e211 VPD83T3:60060160xxxxxx00803a45d92403e211 VPD83T3:60060160xxxxxx003664e853e4afe111 VPD83T3:60060160xxxxxx0022a0820b2069e411 VPD83T3:60060160xxxxxx0006a11a122069e411 VPD83T3:60060160xxxxxx0060e43fe739d7e511 VPD83T3:60060160xxxxxx00783a45d92403e211 VPD83T3:60060160xxxxxx001ea0820b2069e411 cluster-1 director-1-2-A (APM00xxxxxxxxx, EMC~CLARiiON~APM00xxxxxxxxx) VPD83T3:60060160xxxxxx0076675fbab698e311 VPD83T3:60060160xxxxxx0052a54dea9999e311 VPD83T3:60060160xxxxxx00a897d6399a99e311 VPD83T3:60060160xxxxxx0072675fbab698e311 VPD83T3:60060160xxxxxx0026a0820b2069e411 VPD83T3:60060160xxxxxx00d23db51bd191e211 VPD83T3:60060160xxxxxx00803a45d92403e211 VPD83T3:60060160xxxxxx003664e853e4afe111 VPD83T3:60060160xxxxxx0022a0820b2069e411 VPD83T3:60060160xxxxxx0006a11a122069e411 VPD83T3:60060160xxxxxx0060e43fe739d7e511 VPD83T3:60060160xxxxxx00783a45d92403e211 VPD83T3:60060160xxxxxx001ea0820b2069e411 cluster-1 director-1-2-B (APM00xxxxxxxxx, EMC~CLARiiON~APM00xxxxxxxxx) VPD83T3:60060160xxxxxx0076675fbab698e311 VPD83T3:60060160xxxxxx0052a54dea9999e311 VPD83T3:60060160xxxxxx003264e853e4afe111 VPD83T3:60060160xxxxxx00a897d6399a99e311 VPD83T3:60060160xxxxxx0072675fbab698e311 VPD83T3:60060160xxxxxx0026a0820b2069e411 VPD83T3:60060160xxxxxx00d23db51bd191e211 VPD83T3:60060160xxxxxx00803a45d92403e211 VPD83T3:60060160xxxxxx003664e853e4afe111 VPD83T3:60060160xxxxxx0022a0820b2069e411 VPD83T3:60060160xxxxxx0006a11a122069e411 VPD83T3:60060160xxxxxx0060e43fe739d7e511 VPD83T3:60060160xxxxxx00783a45d92403e211 VPD83T3:60060160xxxxxx001ea0820b2069e411 cluster-1 director-1-3-A (APM00xxxxxxxxx, EMC~CLARiiON~APM00xxxxxxxxx) VPD83T3:60060160xxxxxx003264e853e4afe111 VPD83T3:60060160xxxxxx002e64e853e4afe111 VPD83T3:60060160xxxxxx003664e853e4afe111 VPD83T3:60060160xxxxxx00b6708fc4cd9fe311 VPD83T3:60060160xxxxxx0060e43fe739d7e511 cluster-1 director-1-3-B (APM00xxxxxxxxx, EMC~CLARiiON~APM00xxxxxxxxx) VPD83T3:60060160xxxxxx003264e853e4afe111 VPD83T3:60060160xxxxxx002e64e853e4afe111 VPD83T3:60060160xxxxxx003664e853e4afe111 VPD83T3:60060160xxxxxx00b6708fc4cd9fe311 VPD83T3:60060160xxxxxx0060e43fe739d7e511 cluster-1 director-1-4-A (APM00xxxxxxxxx, EMC~CLARiiON~APM00xxxxxxxxx) VPD83T3:60060160xxxxxx003264e853e4afe111 VPD83T3:60060160xxxxxx002e64e853e4afe111 VPD83T3:60060160xxxxxx003664e853e4afe111 VPD83T3:60060160xxxxxx00b6708fc4cd9fe311 VPD83T3:60060160xxxxxx0060e43fe739d7e511 cluster-1 director-1-4-B (APM00xxxxxxxxx, EMC~CLARiiON~APM00xxxxxxxxx) VPD83T3:60060160xxxxxx003264e853e4afe111 VPD83T3:60060160xxxxxx002e64e853e4afe111 VPD83T3:60060160xxxxxx003664e853e4afe111 VPD83T3:60060160xxxxxx00b6708fc4cd9fe311 VPD83T3:60060160xxxxxx0060e43fe739d7e511
Cause
If I/O is not flowing to a device when the path changes state, the state of the path is not updated by VPLEX.
Detailed example:
One of these four paths from tpg 2 triggered a "Request Target Port Group" (RTPG) refresh.
- RTPG refresh causes a request to fetch updated RTPG information for the LU
- However earlier, the LU access state through tpg 2 was UNAVAILABLE, but now it is AAO
- So therefore these four paths become marked as AAO
- Also earlier, the LU access state through tpg 1 was AAO, but now it is AAN
- The four paths from tpg 1 were never refreshed
- LU then ends up with eight AAO paths (four from first tpg and four from second tpg)
- A LU refresh here would have brought in the updated path types
Resolution
Workaround:
The workaround for this issue is to sequentially bounce the VPLEX backend ports zoned to the affected array. After each port bounce, a few minutes should be allowed for the backend port to log back in to its targets prior to bouncing the next port. Bouncing the VPLEX backend ports triggers a fresh login. As part of the login, VPLEX sends an INQ to the array and pulls the correct path state.
Resolution:
A permanent fix for this issue is in 5.5.0.00.00.11 and newer code.