PowerFlex: ESXi I/O errors and possible host hang (FastSlab_Alloc)

Under certain loads I/O errors are returned to the application, an I/O request aborted by the OS, the bus resets, and possibly the ESXi host may hang.

ESXi SDC host

SDC configured with the default queue depth

Data unavailable I/O errors returned to the application.

Vmkernel.log:

2019-09-01T04:35:16.313Z cpu33:66292)scini: blkDev_HandleRequest:2656: ScaleIO R2_6:Error: FastSlab_Alloc for Mit Command. TargetID 16, LUN 16. CDB Opcode 0x8a.

The queue depth can be a number ranging from 32 -16384, the default being 32.  

It is recommended to change the queue depth to 512. 

When the SDC's queue length approaches its upper limit, the SDC may not be able to allocate sufficient memory resources from the ESXi kernel to support additional I/O requests.

When the SDC is unable to allocate the necessary memory under the above-mentioned circumstances, a software issue in the SDC causes it to not respond to the associated I/O requests.

To resolve this issue, reduce the SDC queue depth.

• Set the ESXi scini module's blkDev_QDepth parameter. The recommended value is 512.
  No application performance impact is expected due to the change.
  See the documentation for the steps required to modify ESXi SDC configuration parameters.
  For example, the following command adds blkDev_QDepth parameter to existing parameters only if it is not set. A reboot is required for the parameters to take effect.

  grep scini /etc/vmware/esx.conf | grep -qw blkDev_QDepth || esxcli system module parameters set -m scini -p "$(awk -F'"' '/scini\/options/{print $2}' /etc/vmware/esx.conf) blkDev_QDepth=512"
  	

Impacted Versions

PowerFlex 3.x

PowerFlex 4.x

Fixed in Version

TBD