VPLEX: Storage Volume in critical failure state due to SCSI check condition 07/27/07 from storage-array
Summary: VPLEX marks disk dead due to scsi sense code 07/27/07 from underlying storage-array after 21 attempted retires.
Ez a cikk a következő(k)re vonatkozik:
Ez a cikk nem vonatkozik a következő(k)re:
Ez a cikk nem kapcsolódik egyetlen konkrét termékhez sem.
Ez a cikk nem azonosítja az összes termékverziót.
Symptoms
VPLEX storage-volume is in a 'critical-failure' state, and the underlying volume is healthy on the Array side.
Sample output from cli command "ll /clusters/cluster-1/storage-elements/storage-volumes/<storage-volume name>"
VPlexcli:/> ll /clusters/cluster-1/storage-elements/storage-volumes/Symm1234_0000
/clusters/cluster-1/storage-elements/storage-volumes/Symm1234_0000:
Name Value
----------------------------- ------------------------------------------------
application-consistent false
block-count 65798400
block-size 4K
capacity 251G
description -
free-chunks []
health-indications [hardware dead]
health-state critical-failure <<--
io-status dead
itls 0x5000144200000010/0x5000097000000000/80,
0x5000144200000010/0x5000097000000000/80,
0x5000144200000011/0x5000097000000000/80,
0x5000144200000011/0x5000097000000000/80,
0x5000144200000110/0x5000097000000000/80,
0x5000144200000110/0x5000097000000000/80,
0x5000144200000111/0x5000097000000000/80,
0x5000144200000111/0x5000097000000000/80,
0x5000144200000110/0x5000097000000000/80,
0x5000144200000110/0x5000097000000000/80, ...
(16 total)
largest-free-chunk 0B
locality -
operational-status error
provision-type legacy
storage-array-name EMC-SYMMETRIX-XXXXXXXXX
storage-volumetype normal
system-id VPD83T3:60000097000000000000000000000000
thin-capable false
thin-rebuild true
total-free-space 0B
underlying-storage-block-size 512
use unusable
used-by [extent_Symm1234_0000_1]
vendor-specific-name EMC
Name Value
----------------------------- ------------------------------------------------
application-consistent false
block-count 65798400
block-size 4K
capacity 251G
description -
free-chunks []
health-indications [hardware dead]
health-state critical-failure <<--
io-status dead
itls 0x5000144200000010/0x5000097000000000/80,
0x5000144200000010/0x5000097000000000/80,
0x5000144200000011/0x5000097000000000/80,
0x5000144200000011/0x5000097000000000/80,
0x5000144200000110/0x5000097000000000/80,
0x5000144200000110/0x5000097000000000/80,
0x5000144200000111/0x5000097000000000/80,
0x5000144200000111/0x5000097000000000/80,
0x5000144200000110/0x5000097000000000/80,
0x5000144200000110/0x5000097000000000/80, ...
(16 total)
largest-free-chunk 0B
locality -
operational-status error
provision-type legacy
storage-array-name EMC-SYMMETRIX-XXXXXXXXX
storage-volumetype normal
system-id VPD83T3:60000097000000000000000000000000
thin-capable false
thin-rebuild true
total-free-space 0B
underlying-storage-block-size 512
use unusable
used-by [extent_Symm1234_0000_1]
vendor-specific-name EMC
Storage Volume in critical-failure state
VPlexcli:/> storage-volume summary
SUMMARY (cluster-1)
StorageVolume Name IO Status Operational Status Health State
---------------------------------------- --------- ------------------ ----------------
Symm1234_0000 dead error critical-failure <<--
Storage Array in ok state
VPlexcli:/> ll **/storage-arrays
/clusters/cluster-1/storage-elements/storage-arrays:
Name Connectivity Auto Ports Logical
--------------------------- Status Switch ------------------- Unit
--------------------------- ------------ ------ ------------------- Count
--------------------------- ------------ ------ ------------------- -------
EMC-SYMMETRIX-XXXXXXXXX ok true 0x5000097000000000, X
0x5000097000000000,
0x5000097000000000,
0x5000097000000000,
A sample excerpt from the Dial Home that highlights the logical unit that died:
<EventData><![CDATA[disk VPD83T3:60000970000000000000000000000000: disk died, no mirror exists
If this storage-volume is not part of a distributed device, then the host loses access to this Virtual Volume causing a Data Unavailability (DU) situation.
VPLEX firmware logs show streaming 'scsi/27' events with SCSI Sense Code key 0x7 asc 0x27 ascq 0x7 (07/27/07) which translates to SPACE ALLOCATION FAILED WRITE PROTECT.
VPLEX also records a scsi/169 event in firmware.log which means "The reported thin-enabled storage-volume has run out of space."
Sample scsi/27 instances from firmware.log
<EventData><![CDATA[disk VPD83T3:60000970000000000000000000000000: disk died, no mirror exists
If this storage-volume is not part of a distributed device, then the host loses access to this Virtual Volume causing a Data Unavailability (DU) situation.
VPLEX firmware logs show streaming 'scsi/27' events with SCSI Sense Code key 0x7 asc 0x27 ascq 0x7 (07/27/07) which translates to SPACE ALLOCATION FAILED WRITE PROTECT.
VPLEX also records a scsi/169 event in firmware.log which means "The reported thin-enabled storage-volume has run out of space."
|
128.221.252.37/cpu0/log:5988:W/"00601636f400114833-1":113877:<2>2016/07/27 22:07:22.05: scsi/169 Space allocation failed on scsi vol VPD83T3:6000009700000000000000000000000
|
Sample scsi/27 instances from firmware.log
128.221.253.37/cpu0/log:5988:W/"0060166fcbfe192045-1":3298782:<6>2016/11/11 06:50:57.94: scsi/27 tgt VPD83T3:60000097000000000000000000000000 cmd 0x2a status 0x2 valid 0 resp 0x70 seg 0x0 bits 0x0 key 0x7 info 0x0 alen 10 csi 0x0 asc 0x27 ascq 0x7 fru 0x0 sks 0x0
128.221.253.37/cpu0/log:5988:W/"0060166fcbfe192045-1":3298783:<6>2016/11/11 06:50:58.00: scsi/27 tgt VPD83T3:60000097000000000000000000000000 cmd 0x2a status 0x2 valid 0 resp 0x70 seg 0x0 bits 0x0 key 0x7 info 0x0 alen 10 csi 0x0 asc 0x27 ascq 0x7 fru 0x0 sks 0x0
128.221.253.37/cpu0/log:5988:W/"0060166fcbfe192045-1":3298784:<6>2016/11/11 06:50:58.16: scsi/27 tgt VPD83T3:60000097000000000000000000000000 cmd 0x2a status 0x2 valid 0 resp 0x70 seg 0x0 bits 0x0 key 0x7 info 0x0 alen 10 csi 0x0 asc 0x27 ascq 0x7 fru 0x0 sks 0x0
128.221.253.37/cpu0/log:5988:W/"0060166fcbfe192045-1":3298785:<6>2016/11/11 06:50:58.30: scsi/27 tgt VPD83T3:60000097000000000000000000000000 cmd 0x2a status 0x2 valid 0 resp 0x70 seg 0x0 bits 0x0 key 0x7 info 0x0 alen 10 csi 0x0 asc 0x27 ascq 0x7 fru 0x0 sks 0x0
128.221.253.37/cpu0/log:5988:W/"0060166fcbfe192045-1":3298783:<6>2016/11/11 06:50:58.00: scsi/27 tgt VPD83T3:60000097000000000000000000000000 cmd 0x2a status 0x2 valid 0 resp 0x70 seg 0x0 bits 0x0 key 0x7 info 0x0 alen 10 csi 0x0 asc 0x27 ascq 0x7 fru 0x0 sks 0x0
128.221.253.37/cpu0/log:5988:W/"0060166fcbfe192045-1":3298784:<6>2016/11/11 06:50:58.16: scsi/27 tgt VPD83T3:60000097000000000000000000000000 cmd 0x2a status 0x2 valid 0 resp 0x70 seg 0x0 bits 0x0 key 0x7 info 0x0 alen 10 csi 0x0 asc 0x27 ascq 0x7 fru 0x0 sks 0x0
128.221.253.37/cpu0/log:5988:W/"0060166fcbfe192045-1":3298785:<6>2016/11/11 06:50:58.30: scsi/27 tgt VPD83T3:60000097000000000000000000000000 cmd 0x2a status 0x2 valid 0 resp 0x70 seg 0x0 bits 0x0 key 0x7 info 0x0 alen 10 csi 0x0 asc 0x27 ascq 0x7 fru 0x0 sks 0x0
After 21 retries are performed, the storage-volume is marked dead due to continuous write failures.
|
128.221.253.37/cpu0/log:5988:W/"0060166fcbfe192045-1":3298797:<4>2016/11/11 06:51:01.42: amf/45 disk VPD83T3:60000097000000000000000000000000: write failure: marking this in-use disk dead
128.221.253.37/cpu0/log:5988:W/"0060166fcbfe192045-1":3298798:<2>2016/11/11 06:51:01.42: amf/97 disk VPD83T3:60000097000000000000000000000000: disk died, no mirror exists |
|
Cause
Space allocation has failed on the thin-enabled storage-volumes. There are no more available storage blocks on the backend array to map to the address to which the VPLEX has issued a write command.
Resolution
The storage-array that is sending the scsi check condition, 7/27/07, to VPLEX must be investigated by the respective array vendor. This issue is triggered by the array not being able to service the write I/O request due to an "SPACE ALLOCATION FAILED WRITE PROTECT" issue on the storage-array.
The following cli grep command may be run on the VPLEX Management-Server to get a list of logical-units impacted by the 7/27/07 check condition:
grep "scsi/27" firmware.log* | grep 'key 0x#' | grep 'asc 0x##' | grep 'ascq 0x#' | awk '{print $3, $5, $18, $19, $26,$27,$28,$29}'| sort | uniq -c | sort -nr
Example:
If this is a non-Dell array, have the end user engage the respective array vendor. In order to resolve the issue that exists on the storage-array.
The following cli grep command may be run on the VPLEX Management-Server to get a list of logical-units impacted by the 7/27/07 check condition:
grep "scsi/27" firmware.log* | grep 'key 0x#' | grep 'asc 0x##' | grep 'ascq 0x#' | awk '{print $3, $5, $18, $19, $26,$27,$28,$29}'| sort | uniq -c | sort -nr
Example:
service@ManagementServer:/var/log/VPlex/cli> grep "scsi/27" firmware.log* | grep 'key 0x7' | grep 'asc 0x27' | grep 'ascq 0x7' | awk '{print $3, $5, $18, $19, $26,$27,$28,$29}'| sort | uniq -c | sort -nr
193474 scsi/27 VPD83T3:60000097000000000000000000000000 key 0x7 asc 0x27 ascq 0x7
42254 scsi/27 VPD83T3:60000097000000000000000000000001 key 0x7 asc 0x27 ascq 0x7
32408 scsi/27 VPD83T3:60000097000000000000000000000002 key 0x7 asc 0x27 ascq 0x7
26581 scsi/27 VPD83T3:60000097000000000000000000000003 key 0x7 asc 0x27 ascq 0x7
21643 scsi/27 VPD83T3:60000097000000000000000000000004 key 0x7 asc 0x27 ascq 0x7
19409 scsi/27 VPD83T3:60000097000000000000000000000005 key 0x7 asc 0x27 ascq 0x7
17651 scsi/27 VPD83T3:60000097000000000000000000000006 key 0x7 asc 0x27 ascq 0x7
193474 scsi/27 VPD83T3:60000097000000000000000000000000 key 0x7 asc 0x27 ascq 0x7
42254 scsi/27 VPD83T3:60000097000000000000000000000001 key 0x7 asc 0x27 ascq 0x7
32408 scsi/27 VPD83T3:60000097000000000000000000000002 key 0x7 asc 0x27 ascq 0x7
26581 scsi/27 VPD83T3:60000097000000000000000000000003 key 0x7 asc 0x27 ascq 0x7
21643 scsi/27 VPD83T3:60000097000000000000000000000004 key 0x7 asc 0x27 ascq 0x7
19409 scsi/27 VPD83T3:60000097000000000000000000000005 key 0x7 asc 0x27 ascq 0x7
17651 scsi/27 VPD83T3:60000097000000000000000000000006 key 0x7 asc 0x27 ascq 0x7
If this is a non-Dell array, have the end user engage the respective array vendor. In order to resolve the issue that exists on the storage-array.
Additional Information
- After the stated backend storage pool space has been freed up then from the VPlexcli run the command 'storage-volume resurrect' to resurrect the status of any storage-volumes showing as 'hardware dead'.
- The host administrator may try reclaiming storage using the scsi UNMAP command from the host, provided the Back-end Array supports UNMAP operation from VPLEX.
- If reclaimed storage is not sufficient, the storage administrator must add free block storage to the backend storage array to increase the space available to the thin-enabled storage-volumes.
Érintett termékek
VPLEX for All Flash, VPLEX SeriesTermékek
VPLEX Series, VPLEX VS2, VPLEX VS6Termék tulajdonságai
Article Number: 000167470
Article Type: Solution
Utoljára módosítva: 28 okt. 2025
Version: 5
Választ kaphat kérdéseire más Dell-felhasználóktól
Támogatási szolgáltatások
Ellenőrizze, hogy a készüléke rendelkezik-e támogatási szolgáltatással.