PowerFlex Power-On Reset Errors Streaming In Vmkernel.Log

Scenario
The problem can occur during normal operations in a vSphere environment with shared datastores.

Symptoms
Errors in the vmkernel.log:

2020-02-26T06:37:08.094Z cpu20:2109794)WARNING: ScaleIO mapVolIO_ReportIOErrorIfNeeded:491 :[7975953082] IO-ERROR Type READ. comb: 69a3801e01c3. offsetInComb 11008920. SizeInLB 8. SDS_ID facbf26700000018. Comb Gen 1. Head Gen 12b. StartLB 19fe123
2020-02-26T06:37:08.094Z cpu20:2109794)scini: mapVolIO_ReportIOErrorIfNeeded:512: ScaleIO:Vol ID 0xecadc19600000034. Last vol TGT fault status SUCCESS(65) Reason (reservation conflict) RC (IO_FAULT_RESERVATION_CONFLICT) Retry count (0) chan (1)
2020-02-26T06:37:08.094Z cpu20:2109794)WARNING: ScaleIO blkScsi_PrintIOInfo:3333 :hCmd 0x459afefab140, OpCode 0x88, rc 53 scsiStat 24, senseCode 5, asc 0, ascq 0
2020-02-26T06:37:08.094Z cpu20:2109975)NMP: nmp_ThrottleLogForDevice:3734: last error status from device eui.36b1f928611380beecadc19600000034 repeated 1 times
2020-02-26T06:37:08.115Z cpu20:2109975)NMP: nmp_ThrottleLogForDevice:3788: Cmd 0x88 (0x459afefab140, 3042871) to dev "eui.36b1f928611380beecadc19600000034" on path "vmhba64:C0:T0:L52" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x6 0x29 0x2. Act:NONE
2020-02-26T06:37:08.115Z cpu20:2109975)ScsiDeviceIO: 3414: Cmd(0x459afefab140) 0x88, CmdSN 0xac from world 3042871 to dev "eui.36b1f928611380beecadc19600000034" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x6 0x29 0x2.
2020-02-26T06:37:08.115Z cpu20:2109975)ScsiCore: 1714: Power-on Reset occurred on eui.36b1f928611380beecadc19600000034

OR

2020-02-26T06:33:25.752Z cpu23:2109794)WARNING: ScaleIO mapVolIO_ReportIOErrorIfNeeded:491 :[7975730761] IO-ERROR Type READ. comb: 69a3801e0324. offsetInComb 10939880. SizeInLB 224. SDS_ID facbf26d0000001e. Comb Gen 1. Head Gen 12b. StartLB 19ba$
2020-02-26T06:33:25.752Z cpu23:2109794)scini: mapVolIO_ReportIOErrorIfNeeded:512: ScaleIO:Vol ID 0xecadc19600000034. Last vol TGT fault status SUCCESS(65) Reason (old SCSI gen without retry) RC (IO_FAULT_OLD_SCSI_GEN) Retry count (0) chan (1)
2020-02-26T06:33:25.752Z cpu23:2109794)WARNING: ScaleIO blkScsi_PrintIOInfo:3333 :hCmd 0x459af4cbe300, OpCode 0x88, rc 51 scsiStat 2, senseCode 6, asc 41, ascq 2
2020-02-26T06:33:25.752Z cpu13:2109975)ScsiDeviceIO: 3414: Cmd(0x459af4cbe300) 0x88, CmdSN 0x8f from world 3042871 to dev "eui.36b1f928611380beecadc19600000034" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x6 0x29 0x2.
2020-02-26T06:33:25.752Z cpu13:2109975)ScsiCore: 1714: Power-on Reset occurred on eui.36b1f928611380beecadc19600000034
2020-02-26T06:33:26.557Z cpu23:2109794)WARNING: ScaleIO mapVolIO_ReportIOErrorIfNeeded:491 :[7975731566] IO-ERROR Type WRITE. comb: 69a3001d80b6. offsetInComb 2749760. SizeInLB 8. SDS_ID facbf26e0000001f. Comb Gen 1. Head Gen 136. StartLB a7c07d4
2020-02-26T06:33:26.557Z cpu23:2109794)scini: mapVolIO_ReportIOErrorIfNeeded:512: ScaleIO:Vol ID 0xecadc19500000033. Last vol TGT fault status IO_FAULT_NOT_PRI(12) Reason (old SCSI gen without retry) RC (IO_FAULT_OLD_SCSI_GEN) Retry count (0) chan (1)
2020-02-26T06:33:26.557Z cpu23:2109794)WARNING: ScaleIO blkScsi_PrintIOInfo:3333 :hCmd 0x459ac1ce5bc0, OpCode 0x2a, rc 51 scsiStat 2, senseCode 6, asc 41, ascq 2
2020-02-26T06:33:26.557Z cpu13:2109975)ScsiDeviceIO: 3414: Cmd(0x459ac1ce5bc0) 0x2a, CmdSN 0xffffe001ee83fad0 from world 3042834 to dev "eui.36b1f928611380beecadc19500000033" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x6 0x29 0x2.
2020-02-26T06:33:26.557Z cpu13:2109975)ScsiCore: 1714: Power-on Reset occurred on eui.36b1f928611380beecadc19500000033
2020-02-26T06:33:26.557Z cpu13:2109975)ScsiDeviceIO: 3414: Cmd(0x459ac1d63980) 0x2a, CmdSN 0xffffe00205830340 from world 3042834 to dev "eui.36b1f928611380beecadc19500000033" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x6 0x29 0x2.

Impact

IOs are not affected, these errors are retriable. Potential performance deterioration with peak I/O traffic

One of the ESXi hosts in the cluster was misconfigured - VMFS.HardwareAcceleratedLocking was not set.

In vSphere, ATS locking can be enabled either per file system or globally. In this case, all the hosts in the cluster had matching settings (ATS-only) for all their VMFS datastores, but one host had it globally disabled:

[root@ESXI01:~] cat /etc/vmware/esx.conf |grep HardwareAcceleratedLocking
/adv/VMFS3/HardwareAcceleratedLocking = "0"

For comparison, per-datastore locking was set correctly (ATS):

root@ESX01:~] esxcli storage vmfs lockmode list
Volume Name                UUID                                 Type    Locking Mode  ATS Compatible  ATS Upgrade Modes  ATS Incompatibility Reason
-------------------------  -----------------------------------  ------  ------------  --------------  -----------------  ---------------------------
DAS149                     5dc98f40-5c2f58a8-f5e3-e4434b90b4ca  VMFS-6  ATS+SCSI               false  None               Device does not support ATS
PD1_00015         5df78825-074d4806-6836-e4434b90b4ca  VMFS-6  ATS                     true  No upgrade needed
PD2_00016         5df788a4-91dd451e-f732-e4434b421db8  VMFS-6  ATS                     true  No upgrade needed
PD1_00017         5df87e5f-b80e78fe-0c4d-e4434b904750  VMFS-6  ATS                     true  No upgrade needed
PD1_00018_NOVADP  5df87fad-2a18dd10-3c38-e4434b913086  VMFS-6  ATS                     true  No upgrade needed

Workaround

Enable HardwareAcceleratedLocking in Advanced settings.

Impacted Versions

All versions

Fixed In Version

N/A - not a VxFlex OS issue