VPLEX: Storage Volume in critical failure state due to SCSI check condition 07/27/07 from storage-array

Summary: VPLEX marks disk dead due to scsi sense code 07/27/07 from underlying storage-array after 21 attempted retires.

Ez a cikk a következő(k)re vonatkozik: Ez a cikk nem vonatkozik a következő(k)re: Ez a cikk nem kapcsolódik egyetlen konkrét termékhez sem. Ez a cikk nem azonosítja az összes termékverziót.
További források megtekintése

Symptoms



VPLEX storage-volume is in a 'critical-failure' state, and the underlying volume is healthy on the Array side.
 
Sample output from cli command "ll /clusters/cluster-1/storage-elements/storage-volumes/<storage-volume name>"
 
VPlexcli:/> ll /clusters/cluster-1/storage-elements/storage-volumes/Symm1234_0000
 
/clusters/cluster-1/storage-elements/storage-volumes/Symm1234_0000:
Name                           Value
-----------------------------  ------------------------------------------------
application-consistent         false
block-count                    65798400
block-size                     4K
capacity                       251G
description                    -
free-chunks                    []
health-indications             [hardware dead]
health-state                   critical-failure <<--
io-status                      dead
itls                           0x5000144200000010/0x5000097000000000/80,
                               0x5000144200000010/0x5000097000000000/80,
                               0x5000144200000011/0x5000097000000000/80,
                               0x5000144200000011/0x5000097000000000/80,
                               0x5000144200000110/0x5000097000000000/80,
                               0x5000144200000110/0x5000097000000000/80,
                               0x5000144200000111/0x5000097000000000/80,
                               0x5000144200000111/0x5000097000000000/80,
                               0x5000144200000110/0x5000097000000000/80,
                               0x5000144200000110/0x5000097000000000/80, ...
                               (16 total)
largest-free-chunk             0B
locality                       -
operational-status             error
provision-type                 legacy
storage-array-name             EMC-SYMMETRIX-XXXXXXXXX
storage-volumetype             normal
system-id                      VPD83T3:60000097000000000000000000000000
thin-capable                   false
thin-rebuild                   true
total-free-space               0B
underlying-storage-block-size  512
use                            unusable
used-by                        [extent_Symm1234_0000_1]
vendor-specific-name           EMC

 
Storage Volume in critical-failure state
 
VPlexcli:/> storage-volume summary
SUMMARY (cluster-1)
StorageVolume Name                        IO Status  Operational Status  Health State
----------------------------------------  ---------  ------------------  ----------------
Symm1234_0000                             dead       error               critical-failure <<--

 
Storage Array in ok state

VPlexcli:/> ll **/storage-arrays

/clusters/cluster-1/storage-elements/storage-arrays:
Name                         Connectivity  Auto    Ports                Logical
---------------------------  Status        Switch  -------------------  Unit
---------------------------  ------------  ------  -------------------  Count
---------------------------  ------------  ------  -------------------  -------
EMC-SYMMETRIX-XXXXXXXXX      ok            true    0x5000097000000000,  X
                                                   0x5000097000000000,
                                                   0x5000097000000000,
                                                   0x5000097000000000,
 
A sample excerpt from the Dial Home that highlights the logical unit that died:
 
 <EventData><![CDATA[disk VPD83T3:60000970000000000000000000000000: disk died, no mirror exists

If this storage-volume is not part of a distributed device, then the host loses access to this Virtual Volume causing a Data Unavailability (DU) situation.

VPLEX firmware logs show streaming 'scsi/27' events with SCSI Sense Code key 0x7 asc 0x27 ascq 0x7 (07/27/07) which translates to SPACE ALLOCATION FAILED WRITE PROTECT.

VPLEX also records a scsi/169 event in firmware.log which means "The reported thin-enabled storage-volume has run out of space."
 
128.221.252.37/cpu0/log:5988:W/"00601636f400114833-1":113877:<2>2016/07/27 22:07:22.05: scsi/169 Space allocation failed on scsi vol VPD83T3:6000009700000000000000000000000

Sample scsi/27 instances from firmware.log
 
128.221.253.37/cpu0/log:5988:W/"0060166fcbfe192045-1":3298782:<6>2016/11/11 06:50:57.94: scsi/27 tgt VPD83T3:60000097000000000000000000000000 cmd 0x2a status 0x2 valid 0 resp 0x70 seg 0x0 bits 0x0 key 0x7 info 0x0 alen 10 csi 0x0 asc 0x27 ascq 0x7 fru 0x0 sks 0x0
128.221.253.37/cpu0/log:5988:W/"0060166fcbfe192045-1":3298783:<6>2016/11/11 06:50:58.00: scsi/27 tgt VPD83T3:60000097000000000000000000000000 cmd 0x2a status 0x2 valid 0 resp 0x70 seg 0x0 bits 0x0 key 0x7 info 0x0 alen 10 csi 0x0 asc 0x27 ascq 0x7 fru 0x0 sks 0x0
128.221.253.37/cpu0/log:5988:W/"0060166fcbfe192045-1":3298784:<6>2016/11/11 06:50:58.16: scsi/27 tgt VPD83T3:60000097000000000000000000000000 cmd 0x2a status 0x2 valid 0 resp 0x70 seg 0x0 bits 0x0 key 0x7 info 0x0 alen 10 csi 0x0 asc 0x27 ascq 0x7 fru 0x0 sks 0x0
128.221.253.37/cpu0/log:5988:W/"0060166fcbfe192045-1":3298785:<6>2016/11/11 06:50:58.30: scsi/27 tgt VPD83T3:60000097000000000000000000000000 cmd 0x2a status 0x2 valid 0 resp 0x70 seg 0x0 bits 0x0 key 0x7 info 0x0 alen 10 csi 0x0 asc 0x27 ascq 0x7 fru 0x0 sks 0x0


After 21 retries are performed, the storage-volume is marked dead due to continuous write failures.
 
128.221.253.37/cpu0/log:5988:W/"0060166fcbfe192045-1":3298797:<4>2016/11/11 06:51:01.42: amf/45 disk VPD83T3:60000097000000000000000000000000: write failure: marking this in-use disk dead
128.221.253.37/cpu0/log:5988:W/"0060166fcbfe192045-1":3298798:<2>2016/11/11 06:51:01.42: amf/97 disk VPD83T3:60000097000000000000000000000000: disk died, no mirror exists
 

Cause

Space allocation has failed on the thin-enabled storage-volumes. There are no more available storage blocks on the backend array to map to the address to which the VPLEX has issued a write command.

Resolution

The storage-array that is sending the scsi check condition, 7/27/07, to VPLEX must be investigated by the respective array vendor. This issue is triggered by the array not being able to service the write I/O request due to an "SPACE ALLOCATION FAILED WRITE PROTECT" issue on the storage-array.

The following cli grep command may be run on the VPLEX Management-Server to get a list of logical-units impacted by the 7/27/07 check condition:

grep "scsi/27" firmware.log* | grep 'key 0x#' | grep 'asc 0x##' | grep 'ascq 0x#' | awk '{print $3, $5, $18, $19, $26,$27,$28,$29}'| sort | uniq -c | sort -nr

Example:
service@ManagementServer:/var/log/VPlex/cli> grep "scsi/27" firmware.log* | grep 'key 0x7' | grep 'asc 0x27' | grep 'ascq 0x7' | awk '{print $3, $5, $18, $19, $26,$27,$28,$29}'| sort | uniq -c | sort -nr

 193474 scsi/27 VPD83T3:60000097000000000000000000000000 key 0x7 asc 0x27 ascq 0x7
  42254 scsi/27 VPD83T3:60000097000000000000000000000001 key 0x7 asc 0x27 ascq 0x7
  32408 scsi/27 VPD83T3:60000097000000000000000000000002 key 0x7 asc 0x27 ascq 0x7
  26581 scsi/27 VPD83T3:60000097000000000000000000000003 key 0x7 asc 0x27 ascq 0x7
  21643 scsi/27 VPD83T3:60000097000000000000000000000004 key 0x7 asc 0x27 ascq 0x7
  19409 scsi/27 VPD83T3:60000097000000000000000000000005 key 0x7 asc 0x27 ascq 0x7
  17651 scsi/27 VPD83T3:60000097000000000000000000000006 key 0x7 asc 0x27 ascq 0x7

If this is a non-Dell array, have the end user engage the respective array vendor. In order to resolve the issue that exists on the storage-array.


Additional Information

  • After the stated backend storage pool space has been freed up then from the VPlexcli run the command 'storage-volume resurrect' to resurrect the status of any storage-volumes showing as 'hardware dead'.
  • The host administrator may try reclaiming storage using the scsi UNMAP command from the host, provided the Back-end Array supports UNMAP operation from VPLEX.
  • If reclaimed storage is not sufficient, the storage administrator must add free block storage to the backend storage array to increase the space available to the thin-enabled storage-volumes. 

Érintett termékek

VPLEX for All Flash, VPLEX Series

Termékek

VPLEX Series, VPLEX VS2, VPLEX VS6
Termék tulajdonságai
Article Number: 000167470
Article Type: Solution
Utoljára módosítva: 28 okt. 2025
Version:  5
Választ kaphat kérdéseire más Dell-felhasználóktól
Támogatási szolgáltatások
Ellenőrizze, hogy a készüléke rendelkezik-e támogatási szolgáltatással.