VPLEX: Storage-Volume in critical-failure state due to scsi check condition B/44/00 from storage-array
Summary: VPLEX marks disk dead due to scsi sense code B/44/00 from the underlying storage-array.
Symptoms
VPLEX storage-volume is in a critical-failure state, and the underlying volume is healthy on the Array side.
Sample output from cli command: ll /clusters/cluster-1/storage-elements/storage-volumes/<storage-volume name>
VPlexcli:/> ll /clusters/cluster-1/storage-elements/storage-volumes/Symm1234_0000 /clusters/cluster-1/storage-elements/storage-volumes/Symm1234_0000: Name Value ----------------------------- ------------------------------------------------ application-consistent false block-count 65798400 block-size 4K capacity 251G description - free-chunks [] health-indications [hardware dead] health-state critical-failure <<-- io-status dead itls 0x50001442607de710/0x5000097378028844/80, 0x50001442607de710/0x50000973780288c4/80, 0x50001442607de711/0x5000097378028844/80, 0x50001442607de711/0x50000973780288c4/80, 0x50001442707de710/0x5000097378028844/80, 0x50001442707de710/0x50000973780288c4/80, 0x50001442707de711/0x5000097378028844/80, 0x50001442707de711/0x50000973780288c4/80, 0x50001442a0100410/0x5000097378028844/80, 0x50001442a0100410/0x50000973780288c4/80, ... (16 total) largest-free-chunk 0B locality - operational-status error provision-type legacy storage-array-name EMC-SYMMETRIX-XXXXXXXXX storage-volumetype normal system-id VPD83T3:60000097000000000000000000000000 thin-capable false thin-rebuild true total-free-space 0B underlying-storage-block-size 512 use unusable used-by [extent_Symm1234_0000_1] vendor-specific-name EMC
Storage Volume in critical-failure state
VPlexcli:/> storage-volume summary SUMMARY (cluster-1) StorageVolume Name IO Status Operational Status Health State ---------------------------------------- --------- ------------------ ---------------- Symm1234_0000 dead error critical-failure <<--
A sample excerpt from the Dial Home that highlights the logical unit that died:
<EventData><![CDATA[disk VPD83T3:60000970000000000000000000000000: disk died, no mirror exists
If this storage-volume is not part of a distributed device, then host loses access to this Virtual Volume causing a Data Unavailability situation.
VPlex firmware logs portray streaming scsi/27 events with SCSI Sense Code key 0xb asc 0x44 ascq 0x0 (B/44/00) which translates to INTERNAL TARGET FAILURE.
Sample scsi/27 instances from firmware.log
128.221.253.37/cpu0/log:5988:W/"0060166fcbfe192045-1":3298782:<6>2016/11/11 06:50:57.94: scsi/27 tgt VPD83T3:60000097000000000000000000000000 cmd 0x2a status 0x2 valid 0 resp 0x70 seg 0x0 bits 0x0 key 0xb info 0x0 alen 10 csi 0x0 asc 0x44 ascq 0x0 fru 0x0 sks 0x0 128.221.253.37/cpu0/log:5988:W/"0060166fcbfe192045-1":3298783:<6>2016/11/11 06:50:58.00: scsi/27 tgt VPD83T3:60000097000000000000000000000000 cmd 0x2a status 0x2 valid 0 resp 0x70 seg 0x0 bits 0x0 key 0xb info 0x0 alen 10 csi 0x0 asc 0x44 ascq 0x0 fru 0x0 sks 0x0 128.221.253.37/cpu0/log:5988:W/"0060166fcbfe192045-1":3298784:<6>2016/11/11 06:50:58.16: scsi/27 tgt VPD83T3:60000097000000000000000000000000 cmd 0x2a status 0x2 valid 0 resp 0x70 seg 0x0 bits 0x0 key 0xb info 0x0 alen 10 csi 0x0 asc 0x44 ascq 0x0 fru 0x0 sks 0x0 128.221.253.37/cpu0/log:5988:W/"0060166fcbfe192045-1":3298785:<6>2016/11/11 06:50:58.30: scsi/27 tgt VPD83T3:60000097000000000000000000000000 cmd 0x2a status 0x2 valid 0 resp 0x70 seg 0x0 bits 0x0 key 0xb info 0x0 alen 10 csi 0x0 asc 0x44 ascq 0x0 fru 0x0 sks 0x0
Twenty-one retries performed and the storage-volume are marked dead due to continuous write failures.
128.221.253.37/cpu0/log:5988:W/"0060166fcbfe192045-1":3298797:<4>2016/11/11 06:51:01.42: amf/45 disk VPD83T3:60000097000000000000000000000000: write failure: marking this in-use disk dead 128.221.253.37/cpu0/log:5988:W/"0060166fcbfe192045-1":3298798:<2>2016/11/11 06:51:01.42: amf/97 disk VPD83T3:60000097000000000000000000000000: disk died, no mirror exists
Cause
Storage-Array returns B/44/00 for a Write I/O request (0x2a), the array is not able to serve the I/O request and responds with check condition B/44/00 (INTERNAL TARGET FAILURE)
If the underlying storage-volume is a VMAX volume, then I/O failures can occur for reasons given below:
The reason why the VMAX aborted the IO is not given to the VPlex. There are many reasons why an internal target failure will occur, and could be caused by a very broad range of sources.
Resolution
The storage-array that is sending the scsi check condition, B/44/00, to VPLEX must be investigated by the respective array vendor. This issue is triggered by the array not being able to service the write I/O request due to an "Internal Target Failure" issue on the storage-array.
If the underlying Array is a VMAX Array, then an array can send scsi check condition "B/44/00" for several reasons.
The following cli command can be run on the VPLEX Management-Server to get a list of logical-units impacted by the B/44/00 check condition:
grep "scsi/27" firmware.log* | grep 'key 0xb' |awk '{print $3, $5, $18, $19, $26,$27,$28,$29}'| sort | uniq -c | sort -nr
Example:
service@ManagementServer:~> grep "scsi/27" firmware.log* | grep 'key 0xb' | awk '{print $3, $5, $18, $19, $26,$27,$28,$29}'| sort | uniq -c | sort -nr
103474 scsi/27 VPD83T3:60000097000000000000000000000000 key 0xb asc 0x44 ascq 0x0
45254 scsi/27 VPD83T3:60000097000000000000000000000001 key 0xb asc 0x44 ascq 0x0
32908 scsi/27 VPD83T3:60000097000000000000000000000002 key 0xb asc 0x44 ascq 0x0
25581 scsi/27 VPD83T3:60000097000000000000000000000003 key 0xb asc 0x44 ascq 0x0
21043 scsi/27 VPD83T3:60000097000000000000000000000004 key 0xb asc 0x44 ascq 0x0
10809 scsi/27 VPD83T3:60000097000000000000000000000005 key 0xb asc 0x44 ascq 0x0
10651 scsi/27 VPD83T3:60000097000000000000000000000006 key 0xb asc 0x44 ascq 0x0
If this is a non-EMC array, engage the respective array vendor in order to resolve the issue that exists on the storage-array.
Additional Information
This is not a VPlex issue, this could either be a Back-end Array or a Fabric issue.
Storage-Array team, Connectivity, or both must be engaged.