VPLEX: Storage Volume in critical failure state due to SCSI check condition 07/27/07 from storage-array

Summary: VPLEX marks disk dead due to scsi sense code 07/27/07 from underlying storage-array after 21 attempted retires.

This article applies to This article does not apply to This article is not tied to any specific product. Not all product versions are identified in this article.
Check out other resources

Symptoms



VPLEX storage-volume is in a 'critical-failure' state, and the underlying volume is healthy on the Array side.
 
Sample output from cli command "ll /clusters/cluster-1/storage-elements/storage-volumes/<storage-volume name>"
 
VPlexcli:/> ll /clusters/cluster-1/storage-elements/storage-volumes/Symm1234_0000
 
/clusters/cluster-1/storage-elements/storage-volumes/Symm1234_0000:
Name                           Value
-----------------------------  ------------------------------------------------
application-consistent         false
block-count                    65798400
block-size                     4K
capacity                       251G
description                    -
free-chunks                    []
health-indications             [hardware dead]
health-state                   critical-failure <<--
io-status                      dead
itls                           0x5000144200000010/0x5000097000000000/80,
                               0x5000144200000010/0x5000097000000000/80,
                               0x5000144200000011/0x5000097000000000/80,
                               0x5000144200000011/0x5000097000000000/80,
                               0x5000144200000110/0x5000097000000000/80,
                               0x5000144200000110/0x5000097000000000/80,
                               0x5000144200000111/0x5000097000000000/80,
                               0x5000144200000111/0x5000097000000000/80,
                               0x5000144200000110/0x5000097000000000/80,
                               0x5000144200000110/0x5000097000000000/80, ...
                               (16 total)
largest-free-chunk             0B
locality                       -
operational-status             error
provision-type                 legacy
storage-array-name             EMC-SYMMETRIX-XXXXXXXXX
storage-volumetype             normal
system-id                      VPD83T3:60000097000000000000000000000000
thin-capable                   false
thin-rebuild                   true
total-free-space               0B
underlying-storage-block-size  512
use                            unusable
used-by                        [extent_Symm1234_0000_1]
vendor-specific-name           EMC

 
Storage Volume in critical-failure state
 
VPlexcli:/> storage-volume summary
SUMMARY (cluster-1)
StorageVolume Name                        IO Status  Operational Status  Health State
----------------------------------------  ---------  ------------------  ----------------
Symm1234_0000                             dead       error               critical-failure <<--

 
Storage Array in ok state

VPlexcli:/> ll **/storage-arrays

/clusters/cluster-1/storage-elements/storage-arrays:
Name                         Connectivity  Auto    Ports                Logical
---------------------------  Status        Switch  -------------------  Unit
---------------------------  ------------  ------  -------------------  Count
---------------------------  ------------  ------  -------------------  -------
EMC-SYMMETRIX-XXXXXXXXX      ok            true    0x5000097000000000,  X
                                                   0x5000097000000000,
                                                   0x5000097000000000,
                                                   0x5000097000000000,
 
A sample excerpt from the Dial Home that highlights the logical unit that died:
 
 <EventData><![CDATA[disk VPD83T3:60000970000000000000000000000000: disk died, no mirror exists

If this storage-volume is not part of a distributed device, then the host loses access to this Virtual Volume causing a Data Unavailability (DU) situation.

VPLEX firmware logs show streaming 'scsi/27' events with SCSI Sense Code key 0x7 asc 0x27 ascq 0x7 (07/27/07) which translates to SPACE ALLOCATION FAILED WRITE PROTECT.

VPLEX also records a scsi/169 event in firmware.log which means "The reported thin-enabled storage-volume has run out of space."
 
128.221.252.37/cpu0/log:5988:W/"00601636f400114833-1":113877:<2>2016/07/27 22:07:22.05: scsi/169 Space allocation failed on scsi vol VPD83T3:6000009700000000000000000000000

Sample scsi/27 instances from firmware.log
 
128.221.253.37/cpu0/log:5988:W/"0060166fcbfe192045-1":3298782:<6>2016/11/11 06:50:57.94: scsi/27 tgt VPD83T3:60000097000000000000000000000000 cmd 0x2a status 0x2 valid 0 resp 0x70 seg 0x0 bits 0x0 key 0x7 info 0x0 alen 10 csi 0x0 asc 0x27 ascq 0x7 fru 0x0 sks 0x0
128.221.253.37/cpu0/log:5988:W/"0060166fcbfe192045-1":3298783:<6>2016/11/11 06:50:58.00: scsi/27 tgt VPD83T3:60000097000000000000000000000000 cmd 0x2a status 0x2 valid 0 resp 0x70 seg 0x0 bits 0x0 key 0x7 info 0x0 alen 10 csi 0x0 asc 0x27 ascq 0x7 fru 0x0 sks 0x0
128.221.253.37/cpu0/log:5988:W/"0060166fcbfe192045-1":3298784:<6>2016/11/11 06:50:58.16: scsi/27 tgt VPD83T3:60000097000000000000000000000000 cmd 0x2a status 0x2 valid 0 resp 0x70 seg 0x0 bits 0x0 key 0x7 info 0x0 alen 10 csi 0x0 asc 0x27 ascq 0x7 fru 0x0 sks 0x0
128.221.253.37/cpu0/log:5988:W/"0060166fcbfe192045-1":3298785:<6>2016/11/11 06:50:58.30: scsi/27 tgt VPD83T3:60000097000000000000000000000000 cmd 0x2a status 0x2 valid 0 resp 0x70 seg 0x0 bits 0x0 key 0x7 info 0x0 alen 10 csi 0x0 asc 0x27 ascq 0x7 fru 0x0 sks 0x0


After 21 retries are performed, the storage-volume is marked dead due to continuous write failures.
 
128.221.253.37/cpu0/log:5988:W/"0060166fcbfe192045-1":3298797:<4>2016/11/11 06:51:01.42: amf/45 disk VPD83T3:60000097000000000000000000000000: write failure: marking this in-use disk dead
128.221.253.37/cpu0/log:5988:W/"0060166fcbfe192045-1":3298798:<2>2016/11/11 06:51:01.42: amf/97 disk VPD83T3:60000097000000000000000000000000: disk died, no mirror exists
 

Cause

Space allocation has failed on the thin-enabled storage-volumes. There are no more available storage blocks on the backend array to map to the address to which the VPLEX has issued a write command.

Resolution

The storage-array that is sending the scsi check condition, 7/27/07, to VPLEX must be investigated by the respective array vendor. This issue is triggered by the array not being able to service the write I/O request due to an "SPACE ALLOCATION FAILED WRITE PROTECT" issue on the storage-array.

The following cli grep command may be run on the VPLEX Management-Server to get a list of logical-units impacted by the 7/27/07 check condition:

grep "scsi/27" firmware.log* | grep 'key 0x#' | grep 'asc 0x##' | grep 'ascq 0x#' | awk '{print $3, $5, $18, $19, $26,$27,$28,$29}'| sort | uniq -c | sort -nr

Example:
service@ManagementServer:/var/log/VPlex/cli> grep "scsi/27" firmware.log* | grep 'key 0x7' | grep 'asc 0x27' | grep 'ascq 0x7' | awk '{print $3, $5, $18, $19, $26,$27,$28,$29}'| sort | uniq -c | sort -nr

 193474 scsi/27 VPD83T3:60000097000000000000000000000000 key 0x7 asc 0x27 ascq 0x7
  42254 scsi/27 VPD83T3:60000097000000000000000000000001 key 0x7 asc 0x27 ascq 0x7
  32408 scsi/27 VPD83T3:60000097000000000000000000000002 key 0x7 asc 0x27 ascq 0x7
  26581 scsi/27 VPD83T3:60000097000000000000000000000003 key 0x7 asc 0x27 ascq 0x7
  21643 scsi/27 VPD83T3:60000097000000000000000000000004 key 0x7 asc 0x27 ascq 0x7
  19409 scsi/27 VPD83T3:60000097000000000000000000000005 key 0x7 asc 0x27 ascq 0x7
  17651 scsi/27 VPD83T3:60000097000000000000000000000006 key 0x7 asc 0x27 ascq 0x7

If this is a non-Dell array, have the end user engage the respective array vendor. In order to resolve the issue that exists on the storage-array.


Additional Information

  • After the stated backend storage pool space has been freed up then from the VPlexcli run the command 'storage-volume resurrect' to resurrect the status of any storage-volumes showing as 'hardware dead'.
  • The host administrator may try reclaiming storage using the scsi UNMAP command from the host, provided the Back-end Array supports UNMAP operation from VPLEX.
  • If reclaimed storage is not sufficient, the storage administrator must add free block storage to the backend storage array to increase the space available to the thin-enabled storage-volumes. 

Affected Products

VPLEX for All Flash, VPLEX Series

Products

VPLEX Series, VPLEX VS2, VPLEX VS6
Article Properties
Article Number: 000167470
Article Type: Solution
Last Modified: 28 Oct 2025
Version:  5
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.