VPLEX: Disk marked hardware dead due to SCSI check condition 3/11/0 from storage-array

This event is triggered when VPLEX performs a read request to the underlying storage array LUN, and the array is unable to service I/O on that block of the LUN, which triggers the 3/11/0 check condition (bad block on the array)

This is commonly seen in situations for times of heavy read-I/O such as:

• VPLEX Extent/Device Migration
• Backup operations
• Database Integrity Checks

VPLEX storage-volume is marked "hardware-dead", but shows healthy on the storage-array interface.

Sample output from cli command ll /clusters/cluster-2/storage-elements/storage-volumes/storage-volume name>
 

VPLEX device/extent migration (mobility job) gets stuck at a certain percent.

Sample output from cli command ll data-migrations/device-migrations/<device_migration_name>
 

Host sees VPLEX storage go offline or marked dead, and VPLEX storage-volume is also marked critical-failure or hardware-dead.

Sample data as noted in the firmware log,
amf/45 disk VPD83T3:xxxxxxxxxxxxxxx: read failure: marking this in-use disk dead

VPLEX firmware logs show streaming or intermittent scsi/27 (Check Condition) with SCSI Sense Code entries for 3/11/0 which translates to "Medium Error - unrecovered read error"

Sample output as noted in firmware log during incident,

2016/06/09 02:46:23.67: scsi/27 tgt VPD83T3:6006016011663200b058c25a984de511 cmd 0x28 status 0x2 valid 0 resp 0x70 seg 0x0 bits 0x0 key 0x3 info 0x0 alen 10 csi 0x0 asc 0x11 ascq 0x0 fru 0x0 sks 0x0
2016/06/09 02:46:23.68: scsi/27 tgt VPD83T3:6006016011663200b058c25a984de511 cmd 0x28 status 0x2 valid 0 resp 0x70 seg 0x0 bits 0x0 key 0x3 info 0x0 alen 10 csi 0x0 asc 0x11 ascq 0x0 fru 0x0 sks 0x0
2016/06/09 02:46:23.69: scsi/27 tgt VPD83T3:6006016011663200b058c25a984de511 cmd 0x28 status 0x2 valid 0 resp 0x70 seg 0x0 bits 0x0 key 0x3 info 0x0 alen 10 csi 0x0 asc 0x11 ascq 0x0 fru 0x0 sks 0x0

To confirm this issue, the following will always be true:
key   = 0x3
asc   =  0x11
ascq = 0x0

When VPLEX sends an I/O read request (0x28) to the storage-array, the array is not able to successfully service the I/O request and responds with check-condition 3/11/0 for "unrecovered read error".

VPLEX attempts to read from a bad block on the storage-array, and since the storage-array is unable to service this I/O VPLEX marks the storage as dead.

This is not array or array-code specific.

The cause for this is external to VPLEX and is a problem on the storage-array with LUN.

The storage-array that is sending the scsi check condition, 3/11/0, to VPLEX must be investigated by the respective array vendor. This issue is triggered by the array not being able to service the read I/O request due to an "unrecovered read" issue on the storage-array.

VNX Support must be engaged.

The following cli command can be run on the VPLEX Management-Server to get a list of the top 50 logical-units impacted by the 3/11/0 check conditions:

grep "key 0x3 " /var/log/VPlex/cli/firmware.log_* | awk '{print $3,$5,$18,$19,$26,$27,$28,$29}' | sort | uniq -c | sort -nr | head -50


Example:

This is a block-layer issue on the storage-array and can only be resolved by taking action on the storage-array itself.

This is not a VPLEX issue, but the VPLEX reporting a symptom seen from the backend array.

The use of "storage-volume resurrect --force" is not valid here.
This command forces the dead storage-volume to appear as "alive" in VPLEX regardless of its current IO-status or issues on the underlying storage-array.
This command forces the storage-volume to come back online until the next IO fails to the underlying storage-array.
When the host requests the same block of data that has the 3/11/0 issue on the underlying storage-array, the storage-volume will be marked dead again.
This is expected behavior and not an indication of a VPLEX issue.


Presenting the problematic storage-volume directly from the storage-array to the host (bypassing VPLEX) may let the host use some of the data. However this action is directly presenting possible data corruption to the host. The host continues to have issues reading from the specific blocks with the 3/11/0 check condition issue.