Is this problem with only one ScaleIO volume or all the ScaleIO volumes on which you have created datastores?

SanjeevMalhotra написал(а): Is this problem with only one ScaleIO volume or all the ScaleIO volumes on which you have created datastores?

One volume in ScaleIO and one ScaleIO datastore for ESXi on all nodes (5 nodes).

Problem with IO ERROR on one node from all nodes (4 nodes working fine).

Hi Amama,

Try to check you SDC->SDS connectivity status with 'esxcli ip neighbor list' and ping from the ESXi console every SDS IP in your environment - it most cases one (or more) SDS IP addresses are not accessible via the SDC, so it tries to connect to some SDS to retrieve a particular chunk and fails. Since all the other SDCs work fine, I would presume there are no errors on the SDS side, that would suggest this particular SDC->SDS communication problems.

Cheers,

Pawel

All connection are working.

ARP table in actually state.

Log:

2017-04-02T06:29:32.948Z cpu16:32831)ScsiDeviceIO: 2595: Cmd(0x439d959c81c0) 0x2a, CmdSN 0x80000008 from world 1065468 to dev "eui.7b850322485d5c3051e837c300000000" failed H:0x8 D:0x0 P:0x0

....

2017-04-02T06:29:32.948Z cpu20:32835)ScsiDeviceIO: 2595: Cmd(0x439d8ff28f40) 0x2a, CmdSN 0x80000025 from world 37109 to dev "eui.7b850322485d5c3051e837c300000000" failed H:0x8 D:0x0 P:0x0

2017-04-02T06:29:32.948Z cpu9:32824)ScsiDeviceIO: 2651: Cmd(0x43a593ad2d40) 0x89, CmdSN 0x889a75 from world 32806 to dev "eui.7b850322485d5c3051e837c300000000" failed H:0x5 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.

2017-04-02T06:29:32.948Z cpu20:32835)ScsiDeviceIO: 2595: Cmd(0x439d80011e40) 0x2a, CmdSN 0x889a72 from world 32806 to dev "eui.7b850322485d5c3051e837c300000000" failed H:0x8 D:0x0 P:0x0

2017-04-02T06:29:32.948Z cpu20:32835)ScsiDeviceIO: 2595: Cmd(0x439d9a78dd40) 0x2a, CmdSN 0x889a73 from world 32806 to dev "eui.7b850322485d5c3051e837c300000000" failed H:0x8 D:0x0 P:0x0

2017-04-02T06:29:32.948Z cpu20:32835)ScsiDeviceIO: 2595: Cmd(0x439d8096aa40) 0x2a, CmdSN 0x889a74 from world 32806 to dev "eui.7b850322485d5c3051e837c300000000" failed H:0x8 D:0x0 P:0x0

2017-04-02T06:29:32.948Z cpu20:32835)ScsiDeviceIO: 2595: Cmd(0x439d8ff64240) 0x2a, CmdSN 0x800e000c from world 512010 to dev "eui.7b850322485d5c3051e837c300000000" failed H:0x8 D:0x0 P:0x0

2017-04-02T06:29:32.949Z cpu20:32835)ScsiDeviceIO: 2595: Cmd(0x43a5808f39c0) 0x28, CmdSN 0x889a76 from world 34151 to dev "eui.7b850322485d5c3051e837c300000000" failed H:0x8 D:0x0 P:0x0

2017-04-02T06:29:33.888Z cpu15:88644)HBX: 2802: 'scaleio01': HB at offset 3313664 - Waiting for timed out HB:

2017-04-02T06:29:33.888Z cpu15:88644)  [HB state abcdef02 offset 3313664 gen 39 stampUS 1541732280530 uuid 58c913fc-7eaac31d-56ea-002590f469b6 jrnl drv 14.61 lockImpl 4]

2017-04-02T06:29:35.492Z cpu16:34151)HBX: 2802: 'scaleio01': HB at offset 3313664 - Waiting for timed out HB:

2017-04-02T06:29:35.492Z cpu16:34151)  [HB state abcdef02 offset 3313664 gen 39 stampUS 1541732280530 uuid 58c913fc-7eaac31d-56ea-002590f469b6 jrnl drv 14.61 lockImpl 4]

2017-04-02T06:29:41.689Z cpu29:33353)WARNING: [1541753022] IO-ERROR comb: 2da880000321. offsetInComb 2690048. SizeInLB 128. SDS_ID 9a2ad19900000002. Comb Gen 25. Head Gen 5565.

2017-04-02T06:29:41.689Z cpu29:33353)WARNING: Vol ID 0x51e837c300000000. Last fault Status IO_HARD_ERROR(20).Last error Status SUCCESS(65) Reason (ERROR) Retry count (20) chan (1)

2017-04-02T06:29:41.689Z cpu29:33353)scini: blkScsi_PrintIOInfo:3274: ScaleIO R2_01:hCmd 0x439d939658c0, OpCode 0x2a, rc 1 scsiStat 2, senseCode 4, asc 0, ascq 0

2017-04-02T06:29:41.689Z cpu1:33507)ScsiDeviceIO: 2651: Cmd(0x439d939658c0) 0x2a, CmdSN 0x889934 from world 36004 to dev "eui.7b850322485d5c3051e837c300000000" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x4 0x0 0x0.

.....

2017-04-02T06:29:41.691Z cpu1:33507)ScsiDeviceIO: 2651: Cmd(0x439d96158ac0) 0x2a, CmdSN 0x889938 from world 36004 to dev "eui.7b850322485d5c3051e837c300000000" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x4 0x0 0x0.

2017-04-02T06:29:41.691Z cpu1:33507)NMP: nmp_ThrottleLogForDevice:3231: last error status from device eui.7b850322485d5c3051e837c300000000 repeated 40 times

2017-04-02T06:29:41.691Z cpu1:33507)ScsiDeviceIO: 2651: Cmd(0x439d93534b80) 0x2a, CmdSN 0x889927 from world 36004 to dev "eui.7b850322485d5c3051e837c300000000" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x4 0x0 0x0.

.....

2017-04-02T06:29:41.730Z cpu1:33507)ScsiDeviceIO: 2651: Cmd(0x439d80972bc0) 0x2a, CmdSN 0x889a6d from world 36004 to dev "eui.7b850322485d5c3051e837c300000000" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x4 0x0 0x0.

2017-04-02T06:29:41.730Z cpu0:36004)HBX: 2802: 'scaleio01': HB at offset 3313664 - Waiting for timed out HB:

2017-04-02T06:29:41.730Z cpu0:36004)  [HB state abcdef02 offset 3313664 gen 39 stampUS 1541732280530 uuid 58c913fc-7eaac31d-56ea-002590f469b6 jrnl drv 14.61 lockImpl 4]

2017-04-02T06:29:41.757Z cpu20:32857)HBX: 276: 'scaleio01': HB at offset 3313664 - Reclaimed heartbeat [Timeout]:

2017-04-02T06:29:41.757Z cpu20:32857)  [HB state abcdef02 offset 3313664 gen 39 stampUS 1541753089373 uuid 58c913fc-7eaac31d-56ea-002590f469b6 jrnl drv 14.61 lockImpl 4]

2017-04-02T06:29:49.948Z cpu9:32824)ScsiDeviceIO: 2651: Cmd(0x43a594d8d800) 0x89, CmdSN 0x889bf9 from world 32806 to dev "eui.7b850322485d5c3051e837c300000000" failed H:0x5 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.

How to localize and fix problem?

After reboot SVM are working fine.

How to localize and fix problem?

Try to see if there is anything in the ESXi logs and/or SVM messages - there might have been a disk problem on that SVM?