PowerFlex Power-On Reset Errors Streaming In Vmkernel.Log

Cenário
O problema pode ocorrer durante operações normais em um ambiente vSphere com datastores compartilhados.

Sintomas
Erros no vmkernel.log:

2020-02-26T06:37:08.094Z cpu20:2109794)WARNING: ScaleIO mapVolIO_ReportIOErrorIfNeeded:491 :[7975953082] IO-ERROR Type READ. comb: 69a3801e01c3. offsetInComb 11008920. SizeInLB 8. SDS_ID facbf26700000018. Comb Gen 1. Head Gen 12b. StartLB 19fe123
2020-02-26T06:37:08.094Z cpu20:2109794)scini: mapVolIO_ReportIOErrorIfNeeded:512: ScaleIO:Vol ID 0xecadc19600000034. Last vol TGT fault status SUCCESS(65) Reason (reservation conflict) RC (IO_FAULT_RESERVATION_CONFLICT) Retry count (0) chan (1)
2020-02-26T06:37:08.094Z cpu20:2109794)WARNING: ScaleIO blkScsi_PrintIOInfo:3333 :hCmd 0x459afefab140, OpCode 0x88, rc 53 scsiStat 24, senseCode 5, asc 0, ascq 0
2020-02-26T06:37:08.094Z cpu20:2109975)NMP: nmp_ThrottleLogForDevice:3734: last error status from device eui.36b1f928611380beecadc19600000034 repeated 1 times
2020-02-26T06:37:08.115Z cpu20:2109975)NMP: nmp_ThrottleLogForDevice:3788: Cmd 0x88 (0x459afefab140, 3042871) to dev "eui.36b1f928611380beecadc19600000034" on path "vmhba64:C0:T0:L52" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x6 0x29 0x2. Act:NONE
2020-02-26T06:37:08.115Z cpu20:2109975)ScsiDeviceIO: 3414: Cmd(0x459afefab140) 0x88, CmdSN 0xac from world 3042871 to dev "eui.36b1f928611380beecadc19600000034" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x6 0x29 0x2.
2020-02-26T06:37:08.115Z cpu20:2109975)ScsiCore: 1714: Power-on Reset occurred on eui.36b1f928611380beecadc19600000034

OU

2020-02-26T06:33:25.752Z cpu23:2109794)WARNING: ScaleIO mapVolIO_ReportIOErrorIfNeeded:491 :[7975730761] IO-ERROR Type READ. comb: 69a3801e0324. offsetInComb 10939880. SizeInLB 224. SDS_ID facbf26d0000001e. Comb Gen 1. Head Gen 12b. StartLB 19ba$
2020-02-26T06:33:25.752Z cpu23:2109794)scini: mapVolIO_ReportIOErrorIfNeeded:512: ScaleIO:Vol ID 0xecadc19600000034. Last vol TGT fault status SUCCESS(65) Reason (old SCSI gen without retry) RC (IO_FAULT_OLD_SCSI_GEN) Retry count (0) chan (1)
2020-02-26T06:33:25.752Z cpu23:2109794)WARNING: ScaleIO blkScsi_PrintIOInfo:3333 :hCmd 0x459af4cbe300, OpCode 0x88, rc 51 scsiStat 2, senseCode 6, asc 41, ascq 2
2020-02-26T06:33:25.752Z cpu13:2109975)ScsiDeviceIO: 3414: Cmd(0x459af4cbe300) 0x88, CmdSN 0x8f from world 3042871 to dev "eui.36b1f928611380beecadc19600000034" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x6 0x29 0x2.
2020-02-26T06:33:25.752Z cpu13:2109975)ScsiCore: 1714: Power-on Reset occurred on eui.36b1f928611380beecadc19600000034
2020-02-26T06:33:26.557Z cpu23:2109794)WARNING: ScaleIO mapVolIO_ReportIOErrorIfNeeded:491 :[7975731566] IO-ERROR Type WRITE. comb: 69a3001d80b6. offsetInComb 2749760. SizeInLB 8. SDS_ID facbf26e0000001f. Comb Gen 1. Head Gen 136. StartLB a7c07d4
2020-02-26T06:33:26.557Z cpu23:2109794)scini: mapVolIO_ReportIOErrorIfNeeded:512: ScaleIO:Vol ID 0xecadc19500000033. Last vol TGT fault status IO_FAULT_NOT_PRI(12) Reason (old SCSI gen without retry) RC (IO_FAULT_OLD_SCSI_GEN) Retry count (0) chan (1)
2020-02-26T06:33:26.557Z cpu23:2109794)WARNING: ScaleIO blkScsi_PrintIOInfo:3333 :hCmd 0x459ac1ce5bc0, OpCode 0x2a, rc 51 scsiStat 2, senseCode 6, asc 41, ascq 2
2020-02-26T06:33:26.557Z cpu13:2109975)ScsiDeviceIO: 3414: Cmd(0x459ac1ce5bc0) 0x2a, CmdSN 0xffffe001ee83fad0 from world 3042834 to dev "eui.36b1f928611380beecadc19500000033" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x6 0x29 0x2.
2020-02-26T06:33:26.557Z cpu13:2109975)ScsiCore: 1714: Power-on Reset occurred on eui.36b1f928611380beecadc19500000033
2020-02-26T06:33:26.557Z cpu13:2109975)ScsiDeviceIO: 3414: Cmd(0x459ac1d63980) 0x2a, CmdSN 0xffffe00205830340 from world 3042834 to dev "eui.36b1f928611380beecadc19500000033" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x6 0x29 0x2.

Impacto

As E/Ss não são afetadas, esses erros são recuperáveis. Possível deterioração do desempenho com o tráfego de E/S de pico

Um dos hosts do ESXi no cluster foi configurado incorretamente — VMFS. HardwareAcceleratedLocking não foi definido.

No vSphere, o bloqueio ATS pode ser ativado por sistema de arquivos ou globalmente. Nesse caso, todos os hosts no cluster tinham configurações correspondentes (somente ATS) para todos os seus datastores VMFS, mas um host tinha isso globalmente desabilitado:

[root@ESXI01:~] cat /etc/vmware/esx.conf |grep HardwareAcceleratedLocking
/adv/VMFS3/HardwareAcceleratedLocking = "0"

Para fins de comparação, o bloqueio por datastore foi definido corretamente (ATS):

root@ESX01:~] esxcli storage vmfs lockmode list
Volume Name                UUID                                 Type    Locking Mode  ATS Compatible  ATS Upgrade Modes  ATS Incompatibility Reason
-------------------------  -----------------------------------  ------  ------------  --------------  -----------------  ---------------------------
DAS149                     5dc98f40-5c2f58a8-f5e3-e4434b90b4ca  VMFS-6  ATS+SCSI               false  None               Device does not support ATS
PD1_00015         5df78825-074d4806-6836-e4434b90b4ca  VMFS-6  ATS                     true  No upgrade needed
PD2_00016         5df788a4-91dd451e-f732-e4434b421db8  VMFS-6  ATS                     true  No upgrade needed
PD1_00017         5df87e5f-b80e78fe-0c4d-e4434b904750  VMFS-6  ATS                     true  No upgrade needed
PD1_00018_NOVADP  5df87fad-2a18dd10-3c38-e4434b913086  VMFS-6  ATS                     true  No upgrade needed

Solução temporária

Ative HardwareAcceleratedLocking em Advanced settings.

Versões afetadas

Todas as versões

Correção feita na versão

N/D – Não é um problema do VxFlex OS