Skip to main content
Start a Conversation

This post is more than 5 years old

Solved!

Go to Solution

ndjake

5 Posts

225650

May 16th, 2013 12:00

iSCSI connection issues or not?

Hello,

I'm encountering some issues with iscsi connections in my vSphere environment and my Equallogic group and wondering if I could get some input from you all. I'm curious to know if what I'm seeing is an Equallogic MEM or NLB event or if I actually have some type of issue. Here's a bit about my environment:

3-ESXi Hosts - 5.0 U1 (623860)
Each host has four 1Gbps cards dedicated to the iSCSI network (2 Intel and 2 Broadcom).
Storage Heartbeat VMK configured and is the lowest vmk on vSwitch.
Delayed ack is disabled.
Dell MEM 1.1.2 installed

2-HP 2910al-24g switches
4 ports are dedicated to a LACP trunk group for the interswitch connection
jumbo frames enabled
flow control enabled
iSCSI VLAN is not VLAN 1
Switch logs look good with very few drops or errors.

One Equallogic group with two PS4100's (One PS4100X and One PS4100XV).
Both arrays on FW 5.2.2
Dedicated management network.
Latency looks good across the group.
TCP Retransmits very low <0.1%

Should I expect to see the re-balance event noted in the vmkernel.log file? From other logs I've seen online, it appears as if they all show an entry that states a re-balance request has been received from the array, but I'm not seeing that in my logs. I've provided the relevant entries from both the Equallogic group and vSphere host below. Thanks.

Jake

The informational events that have me somewhat concerned are logged on the group as:

Severity Date and Time Member Message 
-------- ------------------- --------------- ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 
 INFO 5/16/13 12:08:41 PM PS4100X-0 iSCSI session to target '192.168.90.103:3260, iqn.2001-05.com.equallogic:4-52aed6-56c15d168-d6c0021a85c4f71c-eql-esxi-vol-f' from initiator '192.168.90.53:56744, iqn.1998-01.com.vmware:VHOST05-484f029e' was closed. Logout request was received from the initiator. 
 INFO 5/16/13 12:08:40 PM PS4100X-0 iSCSI login to target '192.168.90.102:3260, iqn.2001-05.com.equallogic:4-52aed6-56c15d168-d6c0021a85c4f71c-eql-esxi-vol-f' from initiator '192.168.90.53:60923, iqn.1998-01.com.vmware:VHOST05-484f029e' successful, using Jumbo Frame length. 
 INFO 5/16/13 12:08:40 PM PS4100X-0 iSCSI login to target '192.168.90.101:3260, iqn.2001-05.com.equallogic:4-52aed6-56c15d168-d6c0021a85c4f71c-eql-esxi-vol-f' from initiator '192.168.90.51:55351, iqn.1998-01.com.vmware:VHOST05-484f029e' successful, using Jumbo Frame length. 
 INFO 5/16/13 12:08:40 PM PS4100X-0 iSCSI login to target '192.168.90.104:3260, iqn.2001-05.com.equallogic:4-52aed6-56c15d168-d6c0021a85c4f71c-eql-esxi-vol-f' from initiator '192.168.90.50:56018, iqn.1998-01.com.vmware:VHOST05-484f029e' successful, using Jumbo Frame length. 
 INFO 5/16/13 12:08:40 PM PS4100X-0 iSCSI login to target '192.168.90.104:3260, iqn.2001-05.com.equallogic:4-52aed6-56c15d168-d6c0021a85c4f71c-eql-esxi-vol-f' from initiator '192.168.90.52:62320, iqn.1998-01.com.vmware:VHOST05-484f029e' successful, using Jumbo Frame length. 
 INFO 5/16/13 12:08:38 PM PS4100X-0 iSCSI session to target '192.168.90.102:3260, iqn.2001-05.com.equallogic:4-52aed6-56c15d168-d6c0021a85c4f71c-eql-esxi-vol-f' from initiator '192.168.90.50:53740, iqn.1998-01.com.vmware:VHOST05-484f029e' was closed. Logout request was received from the initiator. 
 INFO 5/16/13 12:08:38 PM PS4100X-0 iSCSI session to target '192.168.90.101:3260, iqn.2001-05.com.equallogic:4-52aed6-56c15d168-d6c0021a85c4f71c-eql-esxi-vol-f' from initiator '192.168.90.52:62640, iqn.1998-01.com.vmware:VHOST05-484f029e' was closed. Logout request was received from the initiator. 
 INFO 5/16/13 12:08:38 PM PS4100X-0 iSCSI session to target '192.168.90.103:3260, iqn.2001-05.com.equallogic:4-52aed6-56c15d168-d6c0021a85c4f71c-eql-esxi-vol-f' from initiator '192.168.90.51:53550, iqn.1998-01.com.vmware:VHOST05-484f029e' was closed. Logout request was received from the initiator. 
 INFO 5/16/13 12:08:27 PM PS4100X-0 iSCSI session to target '192.168.90.104:3260, iqn.2001-05.com.equallogic:4-52aed6-56c15d168-d6c0021a85c4f71c-eql-esxi-vol-f' from initiator '192.168.90.50:51627, iqn.1998-01.com.vmware:VHOST05-484f029e' was closed. Logout request was received from the initiator. 
 INFO 5/16/13 12:08:26 PM PS4100X-0 iSCSI login to target '192.168.90.102:3260, iqn.2001-05.com.equallogic:4-52aed6-56c15d168-d6c0021a85c4f71c-eql-esxi-vol-f' from initiator '192.168.90.50:53740, iqn.1998-01.com.vmware:VHOST05-484f029e' successful, using Jumbo Frame length. 
 INFO 5/16/13 12:08:26 PM PS4100X-0 iSCSI login to target '192.168.90.101:3260, iqn.2001-05.com.equallogic:4-52aed6-56c15d168-d6c0021a85c4f71c-eql-esxi-vol-f' from initiator '192.168.90.52:62640, iqn.1998-01.com.vmware:VHOST05-484f029e' successful, using Jumbo Frame length. 
 INFO 5/16/13 12:08:26 PM PS4100X-0 iSCSI login to target '192.168.90.103:3260, iqn.2001-05.com.equallogic:4-52aed6-56c15d168-d6c0021a85c4f71c-eql-esxi-vol-f' from initiator '192.168.90.51:53550, iqn.1998-01.com.vmware:VHOST05-484f029e' successful, using Jumbo Frame length. 
 INFO 5/16/13 12:08:26 PM PS4100X-0 iSCSI login to target '192.168.90.103:3260, iqn.2001-05.com.equallogic:4-52aed6-56c15d168-d6c0021a85c4f71c-eql-esxi-vol-f' from initiator '192.168.90.53:56744, iqn.1998-01.com.vmware:VHOST05-484f029e' successful, using Jumbo Frame length. 
 INFO 5/16/13 12:08:24 PM PS4100X-0 iSCSI session to target '192.168.90.101:3260, iqn.2001-05.com.equallogic:4-52aed6-56c15d168-d6c0021a85c4f71c-eql-esxi-vol-f' from initiator '192.168.90.53:53109, iqn.1998-01.com.vmware:VHOST05-484f029e' was closed. Logout request was received from the initiator. 
 INFO 5/16/13 12:08:24 PM PS4100X-0 iSCSI session to target '192.168.90.102:3260, iqn.2001-05.com.equallogic:4-52aed6-56c15d168-d6c0021a85c4f71c-eql-esxi-vol-f' from initiator '192.168.90.51:53339, iqn.1998-01.com.vmware:VHOST05-484f029e' was closed. Logout request was received from the initiator. 
 INFO 5/16/13 12:08:24 PM PS4100X-0 iSCSI session to target '192.168.90.104:3260, iqn.2001-05.com.equallogic:4-52aed6-56c15d168-d6c0021a85c4f71c-eql-esxi-vol-f' from initiator '192.168.90.52:53108, iqn.1998-01.com.vmware:VHOST05-484f029e' was closed. Logout request was received from the initiator. 

The host logs these events in vmkernel.log:

2013-05-16T16:08:24.658Z cpu1:2724)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic: vmhba37:CH:1 T:6 CN:0: Failed to receive data: Connection closed by peer
2013-05-16T16:08:24.658Z cpu1:2724)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic: Sess [ISID: 00023d000006 TARGET: iqn.2001-05.com.equallogic:4-52aed6-56c15d168-d6c0021a85c4f71c-eql-esxi-vol-f TPGT: 1 TSIH: 0]
2013-05-16T16:08:24.658Z cpu3:2997)NMP: nmp_ThrottleLogForDevice:2318: Cmd 0x28 (0x41244101e280, 4863) to dev "naa.64ed2a85165dc1561cf7c4851a02c0d6" on path "vmhba37:C1:T6:L0" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
2013-05-16T16:08:24.658Z cpu1:2724)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic: Conn [CID: 0 L: 192.168.90.51:53339 R: 192.168.90.102:3260]
2013-05-16T16:08:24.658Z cpu3:2997)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.64ed2a85165dc1561cf7c4851a02c0d6" state in doubt; requested fast path state update...
2013-05-16T16:08:24.658Z cpu1:2724)iscsi_vmk: iscsivmk_ConnRxNotifyFailure: vmhba37:CH:1 T:6 CN:0: Connection rx notifying failure: Failed to Receive. State=Online
2013-05-16T16:08:24.658Z cpu3:2997)ScsiDeviceIO: 2309: Cmd(0x41244101e280) 0x28, CmdSN 0x800000d3 from world 4863 to dev "naa.64ed2a85165dc1561cf7c4851a02c0d6" failed H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2013-05-16T16:08:24.658Z cpu1:2724)iscsi_vmk: iscsivmk_ConnRxNotifyFailure: Sess [ISID: 00023d000006 TARGET: iqn.2001-05.com.equallogic:4-52aed6-56c15d168-d6c0021a85c4f71c-eql-esxi-vol-f TPGT: 1 TSIH: 0]
2013-05-16T16:08:24.658Z cpu1:2724)iscsi_vmk: iscsivmk_ConnRxNotifyFailure: Conn [CID: 0 L: 192.168.90.51:53339 R: 192.168.90.102:3260]
2013-05-16T16:08:24.658Z cpu1:2724)WARNING: iscsi_vmk: iscsivmk_StopConnection: vmhba37:CH:1 T:6 CN:0: iSCSI connection is being marked "OFFLINE" (Event:6)
2013-05-16T16:08:24.658Z cpu1:2724)WARNING: iscsi_vmk: iscsivmk_StopConnection: Sess [ISID: 00023d000006 TARGET: iqn.2001-05.com.equallogic:4-52aed6-56c15d168-d6c0021a85c4f71c-eql-esxi-vol-f TPGT: 1 TSIH: 0]
2013-05-16T16:08:24.658Z cpu1:2724)WARNING: iscsi_vmk: iscsivmk_StopConnection: Conn [CID: 0 L: 192.168.90.51:53339 R: 192.168.90.102:3260]
2013-05-16T16:08:24.658Z cpu5:2724)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic: vmhba37:CH:2 T:6 CN:0: Failed to receive data: Connection closed by peer
2013-05-16T16:08:24.658Z cpu5:2724)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic: Sess [ISID: 00023d000007 TARGET: iqn.2001-05.com.equallogic:4-52aed6-56c15d168-d6c0021a85c4f71c-eql-esxi-vol-f TPGT: 1 TSIH: 0]
2013-05-16T16:08:24.658Z cpu5:2724)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic: Conn [CID: 0 L: 192.168.90.52:53108 R: 192.168.90.104:3260]
2013-05-16T16:08:24.658Z cpu5:2724)iscsi_vmk: iscsivmk_ConnRxNotifyFailure: vmhba37:CH:2 T:6 CN:0: Connection rx notifying failure: Failed to Receive. State=Online
2013-05-16T16:08:24.659Z cpu5:2724)iscsi_vmk: iscsivmk_ConnRxNotifyFailure: Sess [ISID: 00023d000007 TARGET: iqn.2001-05.com.equallogic:4-52aed6-56c15d168-d6c0021a85c4f71c-eql-esxi-vol-f TPGT: 1 TSIH: 0]
2013-05-16T16:08:24.659Z cpu5:2724)iscsi_vmk: iscsivmk_ConnRxNotifyFailure: Conn [CID: 0 L: 192.168.90.52:53108 R: 192.168.90.104:3260]
2013-05-16T16:08:24.659Z cpu5:2724)WARNING: iscsi_vmk: iscsivmk_StopConnection: vmhba37:CH:2 T:6 CN:0: iSCSI connection is being marked "OFFLINE" (Event:6)
2013-05-16T16:08:24.659Z cpu5:2724)WARNING: iscsi_vmk: iscsivmk_StopConnection: Sess [ISID: 00023d000007 TARGET: iqn.2001-05.com.equallogic:4-52aed6-56c15d168-d6c0021a85c4f71c-eql-esxi-vol-f TPGT: 1 TSIH: 0]
2013-05-16T16:08:24.659Z cpu5:2724)WARNING: iscsi_vmk: iscsivmk_StopConnection: Conn [CID: 0 L: 192.168.90.52:53108 R: 192.168.90.104:3260]
2013-05-16T16:08:24.659Z cpu3:2997)NMP: nmp_ThrottleLogForDevice:2318: Cmd 0x28 (0x412441cb8500, 4863) to dev "naa.64ed2a85165dc1561cf7c4851a02c0d6" on path "vmhba37:C2:T6:L0" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x5 0x24 0x0. Act:EVAL
2013-05-16T16:08:24.659Z cpu0:2724)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic: vmhba37:CH:3 T:6 CN:0: Failed to receive data: Connection closed by peer
2013-05-16T16:08:24.659Z cpu0:2724)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic: Sess [ISID: 00023d000008 TARGET: iqn.2001-05.com.equallogic:4-52aed6-56c15d168-d6c0021a85c4f71c-eql-esxi-vol-f TPGT: 1 TSIH: 0]
2013-05-16T16:08:24.659Z cpu0:2724)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic: Conn [CID: 0 L: 192.168.90.53:53109 R: 192.168.90.101:3260]
2013-05-16T16:08:24.659Z cpu0:2724)iscsi_vmk: iscsivmk_ConnRxNotifyFailure: vmhba37:CH:3 T:6 CN:0: Connection rx notifying failure: Failed to Receive. State=Online
2013-05-16T16:08:24.659Z cpu0:2724)iscsi_vmk: iscsivmk_ConnRxNotifyFailure: Sess [ISID: 00023d000008 TARGET: iqn.2001-05.com.equallogic:4-52aed6-56c15d168-d6c0021a85c4f71c-eql-esxi-vol-f TPGT: 1 TSIH: 0]
2013-05-16T16:08:24.659Z cpu0:2724)iscsi_vmk: iscsivmk_ConnRxNotifyFailure: Conn [CID: 0 L: 192.168.90.53:53109 R: 192.168.90.101:3260]
2013-05-16T16:08:24.659Z cpu0:2724)WARNING: iscsi_vmk: iscsivmk_StopConnection: vmhba37:CH:3 T:6 CN:0: iSCSI connection is being marked "OFFLINE" (Event:6)
2013-05-16T16:08:24.659Z cpu0:2724)WARNING: iscsi_vmk: iscsivmk_StopConnection: Sess [ISID: 00023d000008 TARGET: iqn.2001-05.com.equallogic:4-52aed6-56c15d168-d6c0021a85c4f71c-eql-esxi-vol-f TPGT: 1 TSIH: 0]
2013-05-16T16:08:24.659Z cpu0:2724)WARNING: iscsi_vmk: iscsivmk_StopConnection: Conn [CID: 0 L: 192.168.90.53:53109 R: 192.168.90.101:3260]
2013-05-16T16:08:24.771Z cpu15:5735)NMP: nmp_ThrottleLogForDevice:2318: Cmd 0x28 (0x412441018980, 4863) to dev "naa.64ed2a85165dc1561cf7c4851a02c0d6" on path "vmhba37:C1:T6:L0" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
2013-05-16T16:08:24.860Z cpu4:4828)NMP: nmp_ThrottleLogForDevice:2318: Cmd 0x2a (0x4124010a25c0, 2064) to dev "naa.64ed2a85165dc1561cf7c4851a02c0d6" on path "vmhba37:C3:T6:L0" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
2013-05-16T16:08:24.908Z cpu10:2777)ScsiDeviceIO: 2309: Cmd(0x412440982980) 0x2a, CmdSN 0x80000059 from world 4863 to dev "naa.64ed2a85165dc1561cf7c4851a02c0d6" failed H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2013-05-16T16:08:24.912Z cpu15:2063)NMP: nmp_ThrottleLogForDevice:2318: Cmd 0x28 (0x412441018980, 4863) to dev "naa.64ed2a85165dc1561cf7c4851a02c0d6" on path "vmhba37:C3:T6:L0" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
2013-05-16T16:08:24.921Z cpu10:10448)WARNING: NMP: nmp_SelectPathAndIssueCommand:2693:PSP selected path "vmhba37:C1:T6:L0" in a bad state (dead)on device "naa.64ed2a85165dc1561cf7c4851a02c0d6".
2013-05-16T16:08:24.921Z cpu10:10448)NMP: nmp_ThrottleLogForDevice:2318: Cmd 0x2a (0x412440b242c0, 4863) to dev "naa.64ed2a85165dc1561cf7c4851a02c0d6" on path "vmhba37:C1:T6:L0" Failed: H:0x1 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:FAILOVER
2013-05-16T16:08:24.921Z cpu10:10448)WARNING: NMP: nmp_DeviceRetryCommand:133:Device "naa.64ed2a85165dc1561cf7c4851a02c0d6": awaiting fast path state update for failover with I/O blocked. No prior reservation exists on the device.
2013-05-16T16:08:24.921Z cpu10:10448)WARNING: NMP: nmp_DeviceStartLoop:721:NMP Device "naa.64ed2a85165dc1561cf7c4851a02c0d6" is blocked. Not starting I/O from device.
2013-05-16T16:08:24.964Z cpu7:2088)WARNING: dell_psp_eql_routed: psp_eqlUpdatePathSelection:1014:Path is reported as inactive (0x50475304)
2013-05-16T16:08:24.964Z cpu7:2088)WARNING: dell_psp_eql_routed: psp_eqlUpdatePathSelection:1014:Path is reported as inactive (0x50475304)
2013-05-16T16:08:24.964Z cpu7:2088)WARNING: dell_psp_eql_routed: psp_eqlUpdatePathSelection:1014:Path is reported as inactive (0x50475304)
2013-05-16T16:08:24.965Z cpu0:2088)WARNING: dell_psp_eql_routed: psp_eqlUpdatePathSelection:1014:Path is reported as inactive (0x50475304)
2013-05-16T16:08:24.965Z cpu0:2088)WARNING: dell_psp_eql_routed: psp_eqlUpdatePathSelection:1014:Path is reported as inactive (0x50475304)
2013-05-16T16:08:24.965Z cpu0:2088)WARNING: dell_psp_eql_routed: psp_eqlUpdatePathSelection:1014:Path is reported as inactive (0x50475304)
2013-05-16T16:08:24.967Z cpu6:2209)ScsiPath: 4963: DeletePath : adapter=vmhba37, channel=1, target=6, lun=0
2013-05-16T16:08:24.967Z cpu6:2209)WARNING: dell_psp_eql_routed: psp_eqlUpdatePathSelection:1014:Path is reported as inactive (0x50475304)
2013-05-16T16:08:24.967Z cpu6:2209)WARNING: dell_psp_eql_routed: psp_eqlUpdatePathSelection:1014:Path is reported as inactive (0x50475304)
2013-05-16T16:08:24.967Z cpu6:2209)WARNING: ScsiPath: 5022: Remove path: vmhba37:C1:T6:L0
2013-05-16T16:08:24.967Z cpu6:2206)ScsiPath: 4963: DeletePath : adapter=vmhba37, channel=2, target=6, lun=0
2013-05-16T16:08:24.967Z cpu6:2206)WARNING: dell_psp_eql_routed: psp_eqlUpdatePathSelection:1014:Path is reported as inactive (0x50475304)
2013-05-16T16:08:24.967Z cpu6:2206)WARNING: ScsiPath: 5022: Remove path: vmhba37:C2:T6:L0
2013-05-16T16:08:24.967Z cpu6:2206)ScsiPath: 4963: DeletePath : adapter=vmhba37, channel=3, target=6, lun=0
2013-05-16T16:08:24.968Z cpu2:2206)WARNING: ScsiPath: 5022: Remove path: vmhba37:C3:T6:L0
2013-05-16T16:08:25.232Z cpu12:2698)WARNING: NMP: nmpDeviceAttemptFailover:599:Retry world failover device "naa.64ed2a85165dc1561cf7c4851a02c0d6" - issuing command 0x412440b242c0
2013-05-16T16:08:25.233Z cpu15:2063)NMP: nmpCompleteRetryForPath:321: Retry world recovered device "naa.64ed2a85165dc1561cf7c4851a02c0d6"
2013-05-16T16:08:25.271Z cpu1:2724)iscsi_vmk: iscsivmk_ConnNetRegister: socket 0x410023555110 network resource pool netsched.pools.persist.iscsi associated
2013-05-16T16:08:25.271Z cpu1:2724)iscsi_vmk: iscsivmk_ConnNetRegister: socket 0x410023555110 network tracker id 1 tracker.iSCSI.192.168.90.100 associated
2013-05-16T16:08:25.272Z cpu1:2724)iscsi_vmk: iscsivmk_ConnNetRegister: socket 0x410023490210 network resource pool netsched.pools.persist.iscsi associated
2013-05-16T16:08:25.272Z cpu1:2724)iscsi_vmk: iscsivmk_ConnNetRegister: socket 0x410023490210 network tracker id 1 tracker.iSCSI.192.168.90.100 associated
2013-05-16T16:08:25.272Z cpu1:2724)iscsi_vmk: iscsivmk_ConnNetRegister: socket 0x41002cdc23b0 network resource pool netsched.pools.persist.iscsi associated
2013-05-16T16:08:25.272Z cpu1:2724)iscsi_vmk: iscsivmk_ConnNetRegister: socket 0x41002cdc23b0 network tracker id 1 tracker.iSCSI.192.168.90.100 associated
2013-05-16T16:08:25.272Z cpu1:2724)iscsi_vmk: iscsivmk_ConnNetRegister: socket 0x41002cdc2a50 network resource pool netsched.pools.persist.iscsi associated
2013-05-16T16:08:25.272Z cpu1:2724)iscsi_vmk: iscsivmk_ConnNetRegister: socket 0x41002cdc2a50 network tracker id 1 tracker.iSCSI.192.168.90.100 associated
2013-05-16T16:08:25.280Z cpu1:2724)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic: vmhba37:CH:1 T:6 CN:0: Failed to receive data: Connection closed by peer
2013-05-16T16:08:25.280Z cpu1:2724)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic: Sess [ISID: TARGET: (null) TPGT: 0 TSIH: 0]
2013-05-16T16:08:25.280Z cpu1:2724)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic: Conn [CID: 0 L: 192.168.90.53:49974 R: 192.168.90.100:3260]
2013-05-16T16:08:25.280Z cpu1:2724)iscsi_vmk: iscsivmk_ConnRxNotifyFailure: vmhba37:CH:1 T:6 CN:0: Connection rx notifying failure: Failed to Receive. State=Bound
2013-05-16T16:08:25.280Z cpu1:2724)iscsi_vmk: iscsivmk_ConnRxNotifyFailure: Sess [ISID: TARGET: (null) TPGT: 0 TSIH: 0]
2013-05-16T16:08:25.280Z cpu1:2724)iscsi_vmk: iscsivmk_ConnRxNotifyFailure: Conn [CID: 0 L: 192.168.90.53:49974 R: 192.168.90.100:3260]
2013-05-16T16:08:25.289Z cpu1:2724)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic: vmhba37:CH:2 T:6 CN:0: Failed to receive data: Connection closed by peer
2013-05-16T16:08:25.289Z cpu1:2724)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic: Sess [ISID: TARGET: (null) TPGT: 0 TSIH: 0]
2013-05-16T16:08:25.289Z cpu1:2724)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic: Conn [CID: 0 L: 192.168.90.51:59154 R: 192.168.90.100:3260]
2013-05-16T16:08:25.289Z cpu1:2724)iscsi_vmk: iscsivmk_ConnRxNotifyFailure: vmhba37:CH:2 T:6 CN:0: Connection rx notifying failure: Failed to Receive. State=Bound
2013-05-16T16:08:25.289Z cpu1:2724)iscsi_vmk: iscsivmk_ConnRxNotifyFailure: Sess [ISID: TARGET: (null) TPGT: 0 TSIH: 0]
2013-05-16T16:08:25.289Z cpu1:2724)iscsi_vmk: iscsivmk_ConnRxNotifyFailure: Conn [CID: 0 L: 192.168.90.51:59154 R: 192.168.90.100:3260]
2013-05-16T16:08:25.298Z cpu1:2724)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic: vmhba37:CH:3 T:6 CN:0: Failed to receive data: Connection closed by peer
2013-05-16T16:08:25.298Z cpu1:2724)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic: Sess [ISID: TARGET: (null) TPGT: 0 TSIH: 0]
2013-05-16T16:08:25.298Z cpu1:2724)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic: Conn [CID: 0 L: 192.168.90.52:53935 R: 192.168.90.100:3260]
2013-05-16T16:08:25.298Z cpu1:2724)iscsi_vmk: iscsivmk_ConnRxNotifyFailure: vmhba37:CH:3 T:6 CN:0: Connection rx notifying failure: Failed to Receive. State=Bound
2013-05-16T16:08:25.298Z cpu1:2724)iscsi_vmk: iscsivmk_ConnRxNotifyFailure: Sess [ISID: TARGET: (null) TPGT: 0 TSIH: 0]
2013-05-16T16:08:25.298Z cpu1:2724)iscsi_vmk: iscsivmk_ConnRxNotifyFailure: Conn [CID: 0 L: 192.168.90.52:53935 R: 192.168.90.100:3260]
2013-05-16T16:08:25.306Z cpu1:2724)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic: vmhba37:CH:4 T:6 CN:0: Failed to receive data: Connection closed by peer
2013-05-16T16:08:25.306Z cpu1:2724)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic: Sess [ISID: TARGET: (null) TPGT: 0 TSIH: 0]
2013-05-16T16:08:25.306Z cpu1:2724)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic: Conn [CID: 0 L: 192.168.90.50:49401 R: 192.168.90.100:3260]
2013-05-16T16:08:25.306Z cpu1:2724)iscsi_vmk: iscsivmk_ConnRxNotifyFailure: vmhba37:CH:4 T:6 CN:0: Connection rx notifying failure: Failed to Receive. State=Bound
2013-05-16T16:08:25.306Z cpu1:2724)iscsi_vmk: iscsivmk_ConnRxNotifyFailure: Sess [ISID: TARGET: (null) TPGT: 0 TSIH: 0]
2013-05-16T16:08:25.306Z cpu1:2724)iscsi_vmk: iscsivmk_ConnRxNotifyFailure: Conn [CID: 0 L: 192.168.90.50:49401 R: 192.168.90.100:3260]
2013-05-16T16:08:25.507Z cpu7:2055)NMP: nmp_ThrottleLogForDevice:2318: Cmd 0x1a (0x412401cfc340, 0) to dev "mpx.vmhba1:C0:T0:L0" on path "vmhba1:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE
2013-05-16T16:08:25.507Z cpu7:2055)ScsiDeviceIO: 2322: Cmd(0x412401cfc340) 0x1a, CmdSN 0x1175b from world 0 to dev "mpx.vmhba1:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2013-05-16T16:08:25.560Z cpu1:2724)WARNING: iscsi_vmk: iscsivmk_StopConnection: vmhba37:CH:1 T:6 CN:0: iSCSI connection is being marked "OFFLINE" (Event:4)
2013-05-16T16:08:25.560Z cpu1:2724)WARNING: iscsi_vmk: iscsivmk_StopConnection: Sess [ISID: TARGET: (null) TPGT: 0 TSIH: 0]
2013-05-16T16:08:25.560Z cpu1:2724)WARNING: iscsi_vmk: iscsivmk_StopConnection: Conn [CID: 0 L: 192.168.90.53:49974 R: 192.168.90.100:3260]
2013-05-16T16:08:25.561Z cpu1:2724)iscsi_vmk: iscsivmk_ConnNetRegister: socket 0x41002cdd7210 network resource pool netsched.pools.persist.iscsi associated
2013-05-16T16:08:25.561Z cpu1:2724)iscsi_vmk: iscsivmk_ConnNetRegister: socket 0x41002cdd7210 network tracker id 1 tracker.iSCSI.192.168.90.103 associated
2013-05-16T16:08:25.561Z cpu1:2724)WARNING: iscsi_vmk: iscsivmk_StopConnection: vmhba37:CH:2 T:6 CN:0: iSCSI connection is being marked "OFFLINE" (Event:4)
2013-05-16T16:08:25.561Z cpu1:2724)WARNING: iscsi_vmk: iscsivmk_StopConnection: Sess [ISID: TARGET: (null) TPGT: 0 TSIH: 0]
2013-05-16T16:08:25.561Z cpu1:2724)WARNING: iscsi_vmk: iscsivmk_StopConnection: Conn [CID: 0 L: 192.168.90.51:59154 R: 192.168.90.100:3260]
2013-05-16T16:08:25.562Z cpu1:2724)iscsi_vmk: iscsivmk_ConnNetRegister: socket 0x41002347f910 network resource pool netsched.pools.persist.iscsi associated
2013-05-16T16:08:25.562Z cpu1:2724)iscsi_vmk: iscsivmk_ConnNetRegister: socket 0x41002347f910 network tracker id 1 tracker.iSCSI.192.168.90.103 associated
2013-05-16T16:08:25.562Z cpu1:2724)WARNING: iscsi_vmk: iscsivmk_StopConnection: vmhba37:CH:3 T:6 CN:0: iSCSI connection is being marked "OFFLINE" (Event:4)
2013-05-16T16:08:25.562Z cpu1:2724)WARNING: iscsi_vmk: iscsivmk_StopConnection: Sess [ISID: TARGET: (null) TPGT: 0 TSIH: 0]
2013-05-16T16:08:25.562Z cpu1:2724)WARNING: iscsi_vmk: iscsivmk_StopConnection: Conn [CID: 0 L: 192.168.90.52:53935 R: 192.168.90.100:3260]
2013-05-16T16:08:25.563Z cpu1:2724)iscsi_vmk: iscsivmk_ConnNetRegister: socket 0x41002cdc23b0 network resource pool netsched.pools.persist.iscsi associated
2013-05-16T16:08:25.563Z cpu1:2724)iscsi_vmk: iscsivmk_ConnNetRegister: socket 0x41002cdc23b0 network tracker id 1 tracker.iSCSI.192.168.90.101 associated
2013-05-16T16:08:25.563Z cpu1:2724)WARNING: iscsi_vmk: iscsivmk_StopConnection: vmhba37:CH:4 T:6 CN:0: iSCSI connection is being marked "OFFLINE" (Event:4)
2013-05-16T16:08:25.563Z cpu1:2724)WARNING: iscsi_vmk: iscsivmk_StopConnection: Sess [ISID: TARGET: (null) TPGT: 0 TSIH: 0]
2013-05-16T16:08:25.563Z cpu1:2724)WARNING: iscsi_vmk: iscsivmk_StopConnection: Conn [CID: 0 L: 192.168.90.50:49401 R: 192.168.90.100:3260]
2013-05-16T16:08:25.564Z cpu1:2724)iscsi_vmk: iscsivmk_ConnNetRegister: socket 0x41002cdc2a50 network resource pool netsched.pools.persist.iscsi associated
2013-05-16T16:08:25.564Z cpu1:2724)iscsi_vmk: iscsivmk_ConnNetRegister: socket 0x41002cdc2a50 network tracker id 1 tracker.iSCSI.192.168.90.102 associated
2013-05-16T16:08:26.869Z cpu11:2724)WARNING: iscsi_vmk: iscsivmk_StartConnection: vmhba37:CH:1 T:6 CN:0: iSCSI connection is being marked "ONLINE"
2013-05-16T16:08:26.869Z cpu11:2724)WARNING: iscsi_vmk: iscsivmk_StartConnection: Sess [ISID: 00023d000008 TARGET: iqn.2001-05.com.equallogic:4-52aed6-56c15d168-d6c0021a85c4f71c-eql-esxi-vol-f TPGT: 1 TSIH: 0]
2013-05-16T16:08:26.869Z cpu11:2724)WARNING: iscsi_vmk: iscsivmk_StartConnection: Conn [CID: 0 L: 192.168.90.53:56744 R: 192.168.90.103:3260]
2013-05-16T16:08:26.870Z cpu11:2724)WARNING: iscsi_vmk: iscsivmk_StartConnection: vmhba37:CH:2 T:6 CN:0: iSCSI connection is being marked "ONLINE"
2013-05-16T16:08:26.870Z cpu11:2724)WARNING: iscsi_vmk: iscsivmk_StartConnection: Sess [ISID: 00023d000006 TARGET: iqn.2001-05.com.equallogic:4-52aed6-56c15d168-d6c0021a85c4f71c-eql-esxi-vol-f TPGT: 1 TSIH: 0]
2013-05-16T16:08:26.870Z cpu11:2724)WARNING: iscsi_vmk: iscsivmk_StartConnection: Conn [CID: 0 L: 192.168.90.51:53550 R: 192.168.90.103:3260]
2013-05-16T16:08:26.871Z cpu11:2724)WARNING: iscsi_vmk: iscsivmk_StartConnection: vmhba37:CH:3 T:6 CN:0: iSCSI connection is being marked "ONLINE"
2013-05-16T16:08:26.871Z cpu11:2724)WARNING: iscsi_vmk: iscsivmk_StartConnection: Sess [ISID: 00023d000007 TARGET: iqn.2001-05.com.equallogic:4-52aed6-56c15d168-d6c0021a85c4f71c-eql-esxi-vol-f TPGT: 1 TSIH: 0]
2013-05-16T16:08:26.871Z cpu11:2724)WARNING: iscsi_vmk: iscsivmk_StartConnection: Conn [CID: 0 L: 192.168.90.52:62640 R: 192.168.90.101:3260]
2013-05-16T16:08:26.872Z cpu11:2724)WARNING: iscsi_vmk: iscsivmk_StartConnection: vmhba37:CH:4 T:6 CN:0: iSCSI connection is being marked "ONLINE"
2013-05-16T16:08:26.872Z cpu11:2724)WARNING: iscsi_vmk: iscsivmk_StartConnection: Sess [ISID: 00023d010005 TARGET: iqn.2001-05.com.equallogic:4-52aed6-56c15d168-d6c0021a85c4f71c-eql-esxi-vol-f TPGT: 1 TSIH: 0]
2013-05-16T16:08:26.872Z cpu11:2724)WARNING: iscsi_vmk: iscsivmk_StartConnection: Conn [CID: 0 L: 192.168.90.50:53740 R: 192.168.90.102:3260]
2013-05-16T16:08:27.085Z cpu6:3533)ScsiScan: 1098: Path 'vmhba37:C1:T6:L0': Vendor: 'EQLOGIC ' Model: '100E-00 ' Rev: '5.2 '
2013-05-16T16:08:27.085Z cpu6:3533)ScsiScan: 1101: Path 'vmhba37:C1:T6:L0': Type: 0x0, ANSI rev: 5, TPGS: 1 (implicit only)
2013-05-16T16:08:27.085Z cpu4:3533)ScsiScan: 1582: Add path: vmhba37:C1:T6:L0
2013-05-16T16:08:27.085Z cpu4:3533)ScsiScan: 1098: Path 'vmhba37:C2:T6:L0': Vendor: 'EQLOGIC ' Model: '100E-00 ' Rev: '5.2 '
2013-05-16T16:08:27.085Z cpu4:3533)ScsiScan: 1101: Path 'vmhba37:C2:T6:L0': Type: 0x0, ANSI rev: 5, TPGS: 1 (implicit only)
2013-05-16T16:08:27.086Z cpu4:3533)ScsiScan: 1582: Add path: vmhba37:C2:T6:L0
2013-05-16T16:08:27.087Z cpu4:3533)ScsiScan: 1098: Path 'vmhba37:C3:T6:L0': Vendor: 'EQLOGIC ' Model: '100E-00 ' Rev: '5.2 '
2013-05-16T16:08:27.087Z cpu4:3533)ScsiScan: 1101: Path 'vmhba37:C3:T6:L0': Type: 0x0, ANSI rev: 5, TPGS: 1 (implicit only)
2013-05-16T16:08:27.091Z cpu0:3533)ScsiScan: 1582: Add path: vmhba37:C3:T6:L0
2013-05-16T16:08:27.091Z cpu0:3533)ScsiScan: 1098: Path 'vmhba37:C4:T6:L0': Vendor: 'EQLOGIC ' Model: '100E-00 ' Rev: '5.2 '
2013-05-16T16:08:27.091Z cpu0:3533)ScsiScan: 1101: Path 'vmhba37:C4:T6:L0': Type: 0x0, ANSI rev: 5, TPGS: 1 (implicit only)
2013-05-16T16:08:27.092Z cpu0:3533)ScsiScan: 1582: Add path: vmhba37:C4:T6:L0
2013-05-16T16:08:27.092Z cpu0:3533)ScsiPath: 4541: Plugin 'NMP' claimed path 'vmhba37:C4:T6:L0'
2013-05-16T16:08:27.092Z cpu0:3533)ScsiPath: 4541: Plugin 'NMP' claimed path 'vmhba37:C3:T6:L0'
2013-05-16T16:08:27.092Z cpu0:3533)ScsiPath: 4541: Plugin 'NMP' claimed path 'vmhba37:C2:T6:L0'
2013-05-16T16:08:27.092Z cpu0:3533)ScsiPath: 4541: Plugin 'NMP' claimed path 'vmhba37:C1:T6:L0'
2013-05-16T16:08:27.189Z cpu13:2724)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic: vmhba37:CH:0 T:6 CN:0: Failed to receive data: Connection closed by peer
2013-05-16T16:08:27.189Z cpu13:2724)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic: Sess [ISID: 00023d000005 TARGET: iqn.2001-05.com.equallogic:4-52aed6-56c15d168-d6c0021a85c4f71c-eql-esxi-vol-f TPGT: 1 TSIH: 0]
2013-05-16T16:08:27.189Z cpu13:2724)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic: Conn [CID: 0 L: 192.168.90.50:51627 R: 192.168.90.104:3260]
2013-05-16T16:08:27.189Z cpu13:2724)iscsi_vmk: iscsivmk_ConnRxNotifyFailure: vmhba37:CH:0 T:6 CN:0: Connection rx notifying failure: Failed to Receive. State=Online
2013-05-16T16:08:27.189Z cpu13:2724)iscsi_vmk: iscsivmk_ConnRxNotifyFailure: Sess [ISID: 00023d000005 TARGET: iqn.2001-05.com.equallogic:4-52aed6-56c15d168-d6c0021a85c4f71c-eql-esxi-vol-f TPGT: 1 TSIH: 0]
2013-05-16T16:08:27.189Z cpu13:2724)iscsi_vmk: iscsivmk_ConnRxNotifyFailure: Conn [CID: 0 L: 192.168.90.50:51627 R: 192.168.90.104:3260]
2013-05-16T16:08:27.189Z cpu10:2724)WARNING: iscsi_vmk: iscsivmk_StopConnection: vmhba37:CH:0 T:6 CN:0: iSCSI connection is being marked "OFFLINE" (Event:6)
2013-05-16T16:08:27.189Z cpu10:2724)WARNING: iscsi_vmk: iscsivmk_StopConnection: Sess [ISID: 00023d000005 TARGET: iqn.2001-05.com.equallogic:4-52aed6-56c15d168-d6c0021a85c4f71c-eql-esxi-vol-f TPGT: 1 TSIH: 0]
2013-05-16T16:08:27.189Z cpu10:2724)WARNING: iscsi_vmk: iscsivmk_StopConnection: Conn [CID: 0 L: 192.168.90.50:51627 R: 192.168.90.104:3260]
2013-05-16T16:08:27.189Z cpu14:2677)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.64ed2a85165dc1561cf7c4851a02c0d6" state in doubt; requested fast path state update...
2013-05-16T16:08:27.189Z cpu14:2677)ScsiDeviceIO: 2309: Cmd(0x412440985380) 0x28, CmdSN 0x8000003a from world 4863 to dev "naa.64ed2a85165dc1561cf7c4851a02c0d6" failed H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2013-05-16T16:08:27.238Z cpu10:2058)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.64ed2a85165dc1561cf7c4851a02c0d6" state in doubt; requested fast path state update...
2013-05-16T16:08:27.278Z cpu15:2063)NMP: nmp_ThrottleLogForDevice:2318: Cmd 0x28 (0x412441114a40, 4863) to dev "naa.64ed2a85165dc1561cf7c4851a02c0d6" on path "vmhba37:C0:T6:L0" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
2013-05-16T16:08:27.314Z cpu14:2062)ScsiDeviceIO: 2309: Cmd(0x41244136e280) 0x28, CmdSN 0x800000d9 from world 4863 to dev "naa.64ed2a85165dc1561cf7c4851a02c0d6" failed H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2013-05-16T16:08:27.314Z cpu14:2062)ScsiDeviceIO: 2309: Cmd(0x412441832f00) 0x28, CmdSN 0x800000ca from world 4863 to dev "naa.64ed2a85165dc1561cf7c4851a02c0d6" failed H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2013-05-16T16:08:27.368Z cpu10:2058)NMP: nmp_ThrottleLogForDevice:2318: Cmd 0x28 (0x412440f813c0, 4863) to dev "naa.64ed2a85165dc1561cf7c4851a02c0d6" on path "vmhba37:C0:T6:L0" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
2013-05-16T16:08:27.456Z cpu9:2057)NMP: nmp_ThrottleLogForDevice:2318: Cmd 0x28 (0x412440f50600, 4863) to dev "naa.64ed2a85165dc1561cf7c4851a02c0d6" on path "vmhba37:C0:T6:L0" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
2013-05-16T16:08:27.456Z cpu9:2057)ScsiDeviceIO: 2309: Cmd(0x412440f50600) 0x28, CmdSN 0x80000044 from world 4863 to dev "naa.64ed2a85165dc1561cf7c4851a02c0d6" failed H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2013-05-16T16:08:27.478Z cpu11:2059)NMP: nmp_ThrottleLogForDevice:2318: Cmd 0x28 (0x412440f813c0, 4863) to dev "naa.64ed2a85165dc1561cf7c4851a02c0d6" on path "vmhba37:C0:T6:L0" Failed: H:0x1 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:FAILOVER
2013-05-16T16:08:27.478Z cpu11:2059)WARNING: NMP: nmp_DeviceRetryCommand:133:Device "naa.64ed2a85165dc1561cf7c4851a02c0d6": awaiting fast path state update for failover with I/O blocked. No prior reservation exists on the device.
2013-05-16T16:08:27.478Z cpu11:2059)NMP: nmp_ThrottleLogForDevice:2318: Cmd 0x28 (0x41244054a180, 4863) to dev "naa.64ed2a85165dc1561cf7c4851a02c0d6" on path "vmhba37:C0:T6:L0" Failed: H:0x1 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:FAILOVER
2013-05-16T16:08:27.478Z cpu11:2059)NMP: nmp_ThrottleLogForDevice:2318: Cmd 0x28 (0x41244136d180, 4863) to dev "naa.64ed2a85165dc1561cf7c4851a02c0d6" on path "vmhba37:C0:T6:L0" Failed: H:0x1 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:FAILOVER
2013-05-16T16:08:27.488Z cpu2:2196)WARNING: dell_psp_eql_routed: psp_eqlUpdatePathSelection:1014:Path is reported as inactive (0x50475304)
2013-05-16T16:08:27.488Z cpu6:2698)WARNING: NMP: nmpDeviceAttemptFailover:599:Retry world failover device "naa.64ed2a85165dc1561cf7c4851a02c0d6" - issuing command 0x412440f813c0
2013-05-16T16:08:27.497Z cpu9:2057)NMP: nmpCompleteRetryForPath:321: Retry world recovered device "naa.64ed2a85165dc1561cf7c4851a02c0d6"
2013-05-16T16:08:28.196Z cpu7:2206)ScsiPath: 4963: DeletePath : adapter=vmhba37, channel=0, target=6, lun=0
2013-05-16T16:08:28.197Z cpu14:2062)WARNING: NMP: nmp_DeviceStartLoop:721:NMP Device "naa.64ed2a85165dc1561cf7c4851a02c0d6" is blocked. Not starting I/O from device.
2013-05-16T16:08:28.198Z cpu14:2062)WARNING: NMP: nmp_DeviceStartLoop:721:NMP Device "naa.64ed2a85165dc1561cf7c4851a02c0d6" is blocked. Not starting I/O from device.
2013-05-16T16:08:28.201Z cpu14:2062)WARNING: NMP: nmp_DeviceStartLoop:721:NMP Device "naa.64ed2a85165dc1561cf7c4851a02c0d6" is blocked. Not starting I/O from device.
2013-05-16T16:08:28.201Z cpu10:2058)WARNING: NMP: nmp_DeviceStartLoop:721:NMP Device "naa.64ed2a85165dc1561cf7c4851a02c0d6" is blocked. Not starting I/O from device.
2013-05-16T16:08:28.229Z cpu14:2062)WARNING: NMP: nmp_DeviceStartLoop:721:NMP Device "naa.64ed2a85165dc1561cf7c4851a02c0d6" is blocked. Not starting I/O from device.
2013-05-16T16:08:28.229Z cpu4:2206)WARNING: ScsiPath: 5022: Remove path: vmhba37:C0:T6:L0
2013-05-16T16:08:38.588Z cpu5:2724)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic: vmhba37:CH:2 T:6 CN:0: Failed to receive data: Connection closed by peer
2013-05-16T16:08:38.589Z cpu5:2724)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic: Sess [ISID: 00023d000006 TARGET: iqn.2001-05.com.equallogic:4-52aed6-56c15d168-d6c0021a85c4f71c-eql-esxi-vol-f TPGT: 1 TSIH: 0]
2013-05-16T16:08:38.589Z cpu5:2724)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic: Conn [CID: 0 L: 192.168.90.51:53550 R: 192.168.90.103:3260]
2013-05-16T16:08:38.589Z cpu5:2724)iscsi_vmk: iscsivmk_ConnRxNotifyFailure: vmhba37:CH:2 T:6 CN:0: Connection rx notifying failure: Failed to Receive. State=Online
2013-05-16T16:08:38.589Z cpu5:2724)iscsi_vmk: iscsivmk_ConnRxNotifyFailure: Sess [ISID: 00023d000006 TARGET: iqn.2001-05.com.equallogic:4-52aed6-56c15d168-d6c0021a85c4f71c-eql-esxi-vol-f TPGT: 1 TSIH: 0]
2013-05-16T16:08:38.589Z cpu5:2724)iscsi_vmk: iscsivmk_ConnRxNotifyFailure: Conn [CID: 0 L: 192.168.90.51:53550 R: 192.168.90.103:3260]
2013-05-16T16:08:38.589Z cpu5:2724)WARNING: iscsi_vmk: iscsivmk_StopConnection: vmhba37:CH:2 T:6 CN:0: iSCSI connection is being marked "OFFLINE" (Event:6)
2013-05-16T16:08:38.589Z cpu5:2724)WARNING: iscsi_vmk: iscsivmk_StopConnection: Sess [ISID: 00023d000006 TARGET: iqn.2001-05.com.equallogic:4-52aed6-56c15d168-d6c0021a85c4f71c-eql-esxi-vol-f TPGT: 1 TSIH: 0]
2013-05-16T16:08:38.589Z cpu5:2724)WARNING: iscsi_vmk: iscsivmk_StopConnection: Conn [CID: 0 L: 192.168.90.51:53550 R: 192.168.90.103:3260]
2013-05-16T16:08:38.589Z cpu2:2701)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.64ed2a85165dc1561cf7c4851a02c0d6" state in doubt; requested fast path state update...
2013-05-16T16:08:38.589Z cpu2:2701)ScsiDeviceIO: 2309: Cmd(0x412441780f40) 0x28, CmdSN 0x80000047 from world 4863 to dev "naa.64ed2a85165dc1561cf7c4851a02c0d6" failed H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2013-05-16T16:08:38.590Z cpu5:2724)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic: vmhba37:CH:3 T:6 CN:0: Failed to receive data: Connection closed by peer
2013-05-16T16:08:38.590Z cpu5:2724)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic: Sess [ISID: 00023d000007 TARGET: iqn.2001-05.com.equallogic:4-52aed6-56c15d168-d6c0021a85c4f71c-eql-esxi-vol-f TPGT: 1 TSIH: 0]
2013-05-16T16:08:38.590Z cpu5:2724)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic: Conn [CID: 0 L: 192.168.90.52:62640 R: 192.168.90.101:3260]
2013-05-16T16:08:38.590Z cpu5:2724)iscsi_vmk: iscsivmk_ConnRxNotifyFailure: vmhba37:CH:3 T:6 CN:0: Connection rx notifying failure: Failed to Receive. State=Online
2013-05-16T16:08:38.590Z cpu5:2724)iscsi_vmk: iscsivmk_ConnRxNotifyFailure: Sess [ISID: 00023d000007 TARGET: iqn.2001-05.com.equallogic:4-52aed6-56c15d168-d6c0021a85c4f71c-eql-esxi-vol-f TPGT: 1 TSIH: 0]
2013-05-16T16:08:38.590Z cpu5:2724)iscsi_vmk: iscsivmk_ConnRxNotifyFailure: Conn [CID: 0 L: 192.168.90.52:62640 R: 192.168.90.101:3260]
2013-05-16T16:08:38.590Z cpu1:2724)WARNING: iscsi_vmk: iscsivmk_StopConnection: vmhba37:CH:3 T:6 CN:0: iSCSI connection is being marked "OFFLINE" (Event:6)
2013-05-16T16:08:38.590Z cpu1:2724)WARNING: iscsi_vmk: iscsivmk_StopConnection: Sess [ISID: 00023d000007 TARGET: iqn.2001-05.com.equallogic:4-52aed6-56c15d168-d6c0021a85c4f71c-eql-esxi-vol-f TPGT: 1 TSIH: 0]
2013-05-16T16:08:38.590Z cpu1:2724)WARNING: iscsi_vmk: iscsivmk_StopConnection: Conn [CID: 0 L: 192.168.90.52:62640 R: 192.168.90.101:3260]
2013-05-16T16:08:38.590Z cpu4:2724)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic: vmhba37:CH:4 T:6 CN:0: Failed to receive data: Connection closed by peer
2013-05-16T16:08:38.590Z cpu4:2724)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic: Sess [ISID: 00023d010005 TARGET: iqn.2001-05.com.equallogic:4-52aed6-56c15d168-d6c0021a85c4f71c-eql-esxi-vol-f TPGT: 1 TSIH: 0]
2013-05-16T16:08:38.590Z cpu4:2724)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic: Conn [CID: 0 L: 192.168.90.50:53740 R: 192.168.90.102:3260]
2013-05-16T16:08:38.590Z cpu4:2724)iscsi_vmk: iscsivmk_ConnRxNotifyFailure: vmhba37:CH:4 T:6 CN:0: Connection rx notifying failure: Failed to Receive. State=Online
2013-05-16T16:08:38.590Z cpu4:2724)iscsi_vmk: iscsivmk_ConnRxNotifyFailure: Sess [ISID: 00023d010005 TARGET: iqn.2001-05.com.equallogic:4-52aed6-56c15d168-d6c0021a85c4f71c-eql-esxi-vol-f TPGT: 1 TSIH: 0]
2013-05-16T16:08:38.590Z cpu4:2724)iscsi_vmk: iscsivmk_ConnRxNotifyFailure: Conn [CID: 0 L: 192.168.90.50:53740 R: 192.168.90.102:3260]
2013-05-16T16:08:38.590Z cpu1:2724)WARNING: iscsi_vmk: iscsivmk_StopConnection: vmhba37:CH:4 T:6 CN:0: iSCSI connection is being marked "OFFLINE" (Event:6)
2013-05-16T16:08:38.590Z cpu1:2724)WARNING: iscsi_vmk: iscsivmk_StopConnection: Sess [ISID: 00023d010005 TARGET: iqn.2001-05.com.equallogic:4-52aed6-56c15d168-d6c0021a85c4f71c-eql-esxi-vol-f TPGT: 1 TSIH: 0]
2013-05-16T16:08:38.590Z cpu1:2724)WARNING: iscsi_vmk: iscsivmk_StopConnection: Conn [CID: 0 L: 192.168.90.50:53740 R: 192.168.90.102:3260]
2013-05-16T16:08:38.651Z cpu11:2059)ScsiDeviceIO: 2309: Cmd(0x412441cbc700) 0x28, CmdSN 0x800000dd from world 4863 to dev "naa.64ed2a85165dc1561cf7c4851a02c0d6" failed H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2013-05-16T16:08:38.651Z cpu11:2059)NMP: nmp_ThrottleLogForDevice:2318: Cmd 0x28 (0x41244101b280, 4863) to dev "naa.64ed2a85165dc1561cf7c4851a02c0d6" on path "vmhba37:C4:T6:L0" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
2013-05-16T16:08:38.651Z cpu11:2059)ScsiDeviceIO: 2309: Cmd(0x41244101b280) 0x28, CmdSN 0x800000b0 from world 4863 to dev "naa.64ed2a85165dc1561cf7c4851a02c0d6" failed H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2013-05-16T16:08:38.681Z cpu11:2059)NMP: nmp_ThrottleLogForDevice:2318: Cmd 0x28 (0x41244101b280, 4863) to dev "naa.64ed2a85165dc1561cf7c4851a02c0d6" on path "vmhba37:C4:T6:L0" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
2013-05-16T16:08:38.731Z cpu11:2059)NMP: nmp_ThrottleLogForDevice:2318: Cmd 0x28 (0x41244101b280, 4863) to dev "naa.64ed2a85165dc1561cf7c4851a02c0d6" on path "vmhba37:C4:T6:L0" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
2013-05-16T16:08:38.796Z cpu11:4863)NMP: nmp_ThrottleLogForDevice:2318: Cmd 0x28 (0x41244101b280, 4863) to dev "naa.64ed2a85165dc1561cf7c4851a02c0d6" on path "vmhba37:C4:T6:L0" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
2013-05-16T16:08:38.842Z cpu5:2088)WARNING: dell_psp_eql_routed: psp_eqlUpdatePathSelection:1014:Path is reported as inactive (0x50475304)
2013-05-16T16:08:38.842Z cpu5:2088)WARNING: dell_psp_eql_routed: psp_eqlUpdatePathSelection:1014:Path is reported as inactive (0x50475304)
2013-05-16T16:08:38.842Z cpu5:2088)WARNING: dell_psp_eql_routed: psp_eqlUpdatePathSelection:1014:Path is reported as inactive (0x50475304)
2013-05-16T16:08:38.846Z cpu8:2207)ScsiPath: 4963: DeletePath : adapter=vmhba37, channel=2, target=6, lun=0
2013-05-16T16:08:38.846Z cpu8:2207)WARNING: dell_psp_eql_routed: psp_eqlUpdatePathSelection:1014:Path is reported as inactive (0x50475304)
2013-05-16T16:08:38.846Z cpu8:2207)WARNING: dell_psp_eql_routed: psp_eqlUpdatePathSelection:1014:Path is reported as inactive (0x50475304)
2013-05-16T16:08:38.846Z cpu8:2207)WARNING: ScsiPath: 5022: Remove path: vmhba37:C2:T6:L0
2013-05-16T16:08:38.846Z cpu8:2209)ScsiPath: 4963: DeletePath : adapter=vmhba37, channel=3, target=6, lun=0
2013-05-16T16:08:38.846Z cpu8:2209)WARNING: dell_psp_eql_routed: psp_eqlUpdatePathSelection:1014:Path is reported as inactive (0x50475304)
2013-05-16T16:08:38.846Z cpu8:2209)WARNING: ScsiPath: 5022: Remove path: vmhba37:C3:T6:L0
2013-05-16T16:08:38.846Z cpu8:2209)ScsiPath: 4963: DeletePath : adapter=vmhba37, channel=4, target=6, lun=0
2013-05-16T16:08:38.846Z cpu8:2209)WARNING: ScsiPath: 5022: Remove path: vmhba37:C4:T6:L0
2013-05-16T16:08:39.199Z cpu1:2724)iscsi_vmk: iscsivmk_ConnNetRegister: socket 0x410023490210 network resource pool netsched.pools.persist.iscsi associated
2013-05-16T16:08:39.200Z cpu1:2724)iscsi_vmk: iscsivmk_ConnNetRegister: socket 0x410023490210 network tracker id 1 tracker.iSCSI.192.168.90.100 associated
2013-05-16T16:08:39.200Z cpu1:2724)iscsi_vmk: iscsivmk_ConnNetRegister: socket 0x410023461d30 network resource pool netsched.pools.persist.iscsi associated
2013-05-16T16:08:39.200Z cpu1:2724)iscsi_vmk: iscsivmk_ConnNetRegister: socket 0x410023461d30 network tracker id 1 tracker.iSCSI.192.168.90.100 associated
2013-05-16T16:08:39.200Z cpu4:2724)iscsi_vmk: iscsivmk_ConnNetRegister: socket 0x41002cdc23b0 network resource pool netsched.pools.persist.iscsi associated
2013-05-16T16:08:39.200Z cpu4:2724)iscsi_vmk: iscsivmk_ConnNetRegister: socket 0x41002cdc23b0 network tracker id 1 tracker.iSCSI.192.168.90.100 associated
2013-05-16T16:08:39.201Z cpu4:2724)iscsi_vmk: iscsivmk_ConnNetRegister: socket 0x41002cdc2a50 network resource pool netsched.pools.persist.iscsi associated
2013-05-16T16:08:39.201Z cpu4:2724)iscsi_vmk: iscsivmk_ConnNetRegister: socket 0x41002cdc2a50 network tracker id 1 tracker.iSCSI.192.168.90.100 associated
2013-05-16T16:08:39.208Z cpu4:2724)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic: vmhba37:CH:0 T:6 CN:0: Failed to receive data: Connection closed by peer
2013-05-16T16:08:39.208Z cpu4:2724)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic: Sess [ISID: TARGET: (null) TPGT: 0 TSIH: 0]
2013-05-16T16:08:39.208Z cpu4:2724)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic: Conn [CID: 0 L: 192.168.90.50:63278 R: 192.168.90.100:3260]
2013-05-16T16:08:39.208Z cpu4:2724)iscsi_vmk: iscsivmk_ConnRxNotifyFailure: vmhba37:CH:0 T:6 CN:0: Connection rx notifying failure: Failed to Receive. State=Bound
2013-05-16T16:08:39.208Z cpu4:2724)iscsi_vmk: iscsivmk_ConnRxNotifyFailure: Sess [ISID: TARGET: (null) TPGT: 0 TSIH: 0]
2013-05-16T16:08:39.208Z cpu4:2724)iscsi_vmk: iscsivmk_ConnRxNotifyFailure: Conn [CID: 0 L: 192.168.90.50:63278 R: 192.168.90.100:3260]
2013-05-16T16:08:39.217Z cpu4:2724)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic: vmhba37:CH:2 T:6 CN:0: Failed to receive data: Connection closed by peer
2013-05-16T16:08:39.217Z cpu4:2724)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic: Sess [ISID: TARGET: (null) TPGT: 0 TSIH: 0]
2013-05-16T16:08:39.217Z cpu4:2724)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic: Conn [CID: 0 L: 192.168.90.52:51827 R: 192.168.90.100:3260]
2013-05-16T16:08:39.217Z cpu4:2724)iscsi_vmk: iscsivmk_ConnRxNotifyFailure: vmhba37:CH:2 T:6 CN:0: Connection rx notifying failure: Failed to Receive. State=Bound
2013-05-16T16:08:39.217Z cpu4:2724)iscsi_vmk: iscsivmk_ConnRxNotifyFailure: Sess [ISID: TARGET: (null) TPGT: 0 TSIH: 0]
2013-05-16T16:08:39.217Z cpu4:2724)iscsi_vmk: iscsivmk_ConnRxNotifyFailure: Conn [CID: 0 L: 192.168.90.52:51827 R: 192.168.90.100:3260]
2013-05-16T16:08:39.225Z cpu4:2724)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic: vmhba37:CH:3 T:6 CN:0: Failed to receive data: Connection closed by peer
2013-05-16T16:08:39.225Z cpu4:2724)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic: Sess [ISID: TARGET: (null) TPGT: 0 TSIH: 0]
2013-05-16T16:08:39.225Z cpu4:2724)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic: Conn [CID: 0 L: 192.168.90.51:56110 R: 192.168.90.100:3260]
2013-05-16T16:08:39.225Z cpu4:2724)iscsi_vmk: iscsivmk_ConnRxNotifyFailure: vmhba37:CH:3 T:6 CN:0: Connection rx notifying failure: Failed to Receive. State=Bound
2013-05-16T16:08:39.225Z cpu4:2724)iscsi_vmk: iscsivmk_ConnRxNotifyFailure: Sess [ISID: TARGET: (null) TPGT: 0 TSIH: 0]
2013-05-16T16:08:39.225Z cpu4:2724)iscsi_vmk: iscsivmk_ConnRxNotifyFailure: Conn [CID: 0 L: 192.168.90.51:56110 R: 192.168.90.100:3260]
2013-05-16T16:08:39.233Z cpu4:2724)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic: vmhba37:CH:4 T:6 CN:0: Failed to receive data: Connection closed by peer
2013-05-16T16:08:39.233Z cpu4:2724)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic: Sess [ISID: TARGET: (null) TPGT: 0 TSIH: 0]
2013-05-16T16:08:39.233Z cpu4:2724)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic: Conn [CID: 0 L: 192.168.90.53:57285 R: 192.168.90.100:3260]
2013-05-16T16:08:39.233Z cpu4:2724)iscsi_vmk: iscsivmk_ConnRxNotifyFailure: vmhba37:CH:4 T:6 CN:0: Connection rx notifying failure: Failed to Receive. State=Bound
2013-05-16T16:08:39.233Z cpu4:2724)iscsi_vmk: iscsivmk_ConnRxNotifyFailure: Sess [ISID: TARGET: (null) TPGT: 0 TSIH: 0]
2013-05-16T16:08:39.233Z cpu4:2724)iscsi_vmk: iscsivmk_ConnRxNotifyFailure: Conn [CID: 0 L: 192.168.90.53:57285 R: 192.168.90.100:3260]
2013-05-16T16:08:39.494Z cpu0:2724)WARNING: iscsi_vmk: iscsivmk_StopConnection: vmhba37:CH:0 T:6 CN:0: iSCSI connection is being marked "OFFLINE" (Event:4)
2013-05-16T16:08:39.494Z cpu0:2724)WARNING: iscsi_vmk: iscsivmk_StopConnection: Sess [ISID: TARGET: (null) TPGT: 0 TSIH: 0]
2013-05-16T16:08:39.494Z cpu0:2724)WARNING: iscsi_vmk: iscsivmk_StopConnection: Conn [CID: 0 L: 192.168.90.50:63278 R: 192.168.90.100:3260]
2013-05-16T16:08:39.495Z cpu0:2724)iscsi_vmk: iscsivmk_ConnNetRegister: socket 0x41002347f910 network resource pool netsched.pools.persist.iscsi associated
2013-05-16T16:08:39.495Z cpu0:2724)iscsi_vmk: iscsivmk_ConnNetRegister: socket 0x41002347f910 network tracker id 1 tracker.iSCSI.192.168.90.104 associated
2013-05-16T16:08:39.495Z cpu0:2724)WARNING: iscsi_vmk: iscsivmk_StopConnection: vmhba37:CH:2 T:6 CN:0: iSCSI connection is being marked "OFFLINE" (Event:4)
2013-05-16T16:08:39.495Z cpu0:2724)WARNING: iscsi_vmk: iscsivmk_StopConnection: Sess [ISID: TARGET: (null) TPGT: 0 TSIH: 0]
2013-05-16T16:08:39.495Z cpu0:2724)WARNING: iscsi_vmk: iscsivmk_StopConnection: Conn [CID: 0 L: 192.168.90.52:51827 R: 192.168.90.100:3260]
2013-05-16T16:08:39.496Z cpu0:2724)iscsi_vmk: iscsivmk_ConnNetRegister: socket 0x410023461d30 network resource pool netsched.pools.persist.iscsi associated
2013-05-16T16:08:39.496Z cpu0:2724)iscsi_vmk: iscsivmk_ConnNetRegister: socket 0x410023461d30 network tracker id 1 tracker.iSCSI.192.168.90.104 associated
2013-05-16T16:08:39.496Z cpu0:2724)WARNING: iscsi_vmk: iscsivmk_StopConnection: vmhba37:CH:3 T:6 CN:0: iSCSI connection is being marked "OFFLINE" (Event:4)
2013-05-16T16:08:39.496Z cpu0:2724)WARNING: iscsi_vmk: iscsivmk_StopConnection: Sess [ISID: TARGET: (null) TPGT: 0 TSIH: 0]
2013-05-16T16:08:39.496Z cpu0:2724)WARNING: iscsi_vmk: iscsivmk_StopConnection: Conn [CID: 0 L: 192.168.90.51:56110 R: 192.168.90.100:3260]
2013-05-16T16:08:39.497Z cpu0:2724)iscsi_vmk: iscsivmk_ConnNetRegister: socket 0x41002cdc23b0 network resource pool netsched.pools.persist.iscsi associated
2013-05-16T16:08:39.497Z cpu0:2724)iscsi_vmk: iscsivmk_ConnNetRegister: socket 0x41002cdc23b0 network tracker id 1 tracker.iSCSI.192.168.90.101 associated
2013-05-16T16:08:39.497Z cpu0:2724)WARNING: iscsi_vmk: iscsivmk_StopConnection: vmhba37:CH:4 T:6 CN:0: iSCSI connection is being marked "OFFLINE" (Event:4)
2013-05-16T16:08:39.497Z cpu0:2724)WARNING: iscsi_vmk: iscsivmk_StopConnection: Sess [ISID: TARGET: (null) TPGT: 0 TSIH: 0]
2013-05-16T16:08:39.497Z cpu0:2724)WARNING: iscsi_vmk: iscsivmk_StopConnection: Conn [CID: 0 L: 192.168.90.53:57285 R: 192.168.90.100:3260]
2013-05-16T16:08:39.498Z cpu0:2724)iscsi_vmk: iscsivmk_ConnNetRegister: socket 0x41002cdc2a50 network resource pool netsched.pools.persist.iscsi associated
2013-05-16T16:08:39.498Z cpu0:2724)iscsi_vmk: iscsivmk_ConnNetRegister: socket 0x41002cdc2a50 network tracker id 1 tracker.iSCSI.192.168.90.102 associated
2013-05-16T16:08:40.873Z cpu0:2724)WARNING: iscsi_vmk: iscsivmk_StartConnection: vmhba37:CH:2 T:6 CN:0: iSCSI connection is being marked "ONLINE"
2013-05-16T16:08:40.873Z cpu0:2724)WARNING: iscsi_vmk: iscsivmk_StartConnection: Sess [ISID: 00023d000007 TARGET: iqn.2001-05.com.equallogic:4-52aed6-56c15d168-d6c0021a85c4f71c-eql-esxi-vol-f TPGT: 1 TSIH: 0]
2013-05-16T16:08:40.873Z cpu0:2724)WARNING: iscsi_vmk: iscsivmk_StartConnection: Conn [CID: 0 L: 192.168.90.52:62320 R: 192.168.90.104:3260]
2013-05-16T16:08:40.874Z cpu0:2724)WARNING: iscsi_vmk: iscsivmk_StartConnection: vmhba37:CH:0 T:6 CN:0: iSCSI connection is being marked "ONLINE"
2013-05-16T16:08:40.874Z cpu0:2724)WARNING: iscsi_vmk: iscsivmk_StartConnection: Sess [ISID: 00023d000005 TARGET: iqn.2001-05.com.equallogic:4-52aed6-56c15d168-d6c0021a85c4f71c-eql-esxi-vol-f TPGT: 1 TSIH: 0]
2013-05-16T16:08:40.874Z cpu0:2724)WARNING: iscsi_vmk: iscsivmk_StartConnection: Conn [CID: 0 L: 192.168.90.50:56018 R: 192.168.90.104:3260]
2013-05-16T16:08:40.875Z cpu0:2724)WARNING: iscsi_vmk: iscsivmk_StartConnection: vmhba37:CH:3 T:6 CN:0: iSCSI connection is being marked "ONLINE"
2013-05-16T16:08:40.875Z cpu0:2724)WARNING: iscsi_vmk: iscsivmk_StartConnection: Sess [ISID: 00023d000006 TARGET: iqn.2001-05.com.equallogic:4-52aed6-56c15d168-d6c0021a85c4f71c-eql-esxi-vol-f TPGT: 1 TSIH: 0]
2013-05-16T16:08:40.875Z cpu0:2724)WARNING: iscsi_vmk: iscsivmk_StartConnection: Conn [CID: 0 L: 192.168.90.51:55351 R: 192.168.90.101:3260]
2013-05-16T16:08:40.876Z cpu0:2724)WARNING: iscsi_vmk: iscsivmk_StartConnection: vmhba37:CH:4 T:6 CN:0: iSCSI connection is being marked "ONLINE"
2013-05-16T16:08:40.876Z cpu0:2724)WARNING: iscsi_vmk: iscsivmk_StartConnection: Sess [ISID: 00023d010008 TARGET: iqn.2001-05.com.equallogic:4-52aed6-56c15d168-d6c0021a85c4f71c-eql-esxi-vol-f TPGT: 1 TSIH: 0]
2013-05-16T16:08:40.876Z cpu0:2724)WARNING: iscsi_vmk: iscsivmk_StartConnection: Conn [CID: 0 L: 192.168.90.53:60923 R: 192.168.90.102:3260]
2013-05-16T16:08:40.963Z cpu6:3533)ScsiScan: 1098: Path 'vmhba37:C0:T6:L0': Vendor: 'EQLOGIC ' Model: '100E-00 ' Rev: '5.2 '
2013-05-16T16:08:40.963Z cpu6:3533)ScsiScan: 1101: Path 'vmhba37:C0:T6:L0': Type: 0x0, ANSI rev: 5, TPGS: 1 (implicit only)
2013-05-16T16:08:40.964Z cpu6:3533)ScsiScan: 1582: Add path: vmhba37:C0:T6:L0
2013-05-16T16:08:40.964Z cpu6:3533)ScsiScan: 1098: Path 'vmhba37:C2:T6:L0': Vendor: 'EQLOGIC ' Model: '100E-00 ' Rev: '5.2 '
2013-05-16T16:08:40.964Z cpu6:3533)ScsiScan: 1101: Path 'vmhba37:C2:T6:L0': Type: 0x0, ANSI rev: 5, TPGS: 1 (implicit only)
2013-05-16T16:08:40.965Z cpu6:3533)ScsiScan: 1582: Add path: vmhba37:C2:T6:L0
2013-05-16T16:08:40.965Z cpu6:3533)ScsiScan: 1098: Path 'vmhba37:C3:T6:L0': Vendor: 'EQLOGIC ' Model: '100E-00 ' Rev: '5.2 '
2013-05-16T16:08:40.965Z cpu6:3533)ScsiScan: 1101: Path 'vmhba37:C3:T6:L0': Type: 0x0, ANSI rev: 5, TPGS: 1 (implicit only)
2013-05-16T16:08:40.965Z cpu6:3533)ScsiScan: 1582: Add path: vmhba37:C3:T6:L0
2013-05-16T16:08:40.966Z cpu6:3533)ScsiScan: 1098: Path 'vmhba37:C4:T6:L0': Vendor: 'EQLOGIC ' Model: '100E-00 ' Rev: '5.2 '
2013-05-16T16:08:40.966Z cpu6:3533)ScsiScan: 1101: Path 'vmhba37:C4:T6:L0': Type: 0x0, ANSI rev: 5, TPGS: 1 (implicit only)
2013-05-16T16:08:40.967Z cpu6:3533)ScsiScan: 1582: Add path: vmhba37:C4:T6:L0
2013-05-16T16:08:40.967Z cpu1:3533)ScsiPath: 4541: Plugin 'NMP' claimed path 'vmhba37:C4:T6:L0'
2013-05-16T16:08:40.967Z cpu1:3533)ScsiPath: 4541: Plugin 'NMP' claimed path 'vmhba37:C3:T6:L0'
2013-05-16T16:08:40.967Z cpu1:3533)ScsiPath: 4541: Plugin 'NMP' claimed path 'vmhba37:C2:T6:L0'
2013-05-16T16:08:40.967Z cpu1:3533)ScsiPath: 4541: Plugin 'NMP' claimed path 'vmhba37:C0:T6:L0'
2013-05-16T16:08:41.063Z cpu0:2724)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic: vmhba37:CH:1 T:6 CN:0: Failed to receive data: Connection closed by peer
2013-05-16T16:08:41.063Z cpu0:2724)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic: Sess [ISID: 00023d000008 TARGET: iqn.2001-05.com.equallogic:4-52aed6-56c15d168-d6c0021a85c4f71c-eql-esxi-vol-f TPGT: 1 TSIH: 0]
2013-05-16T16:08:41.063Z cpu0:2724)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic: Conn [CID: 0 L: 192.168.90.53:56744 R: 192.168.90.103:3260]
2013-05-16T16:08:41.063Z cpu0:2724)iscsi_vmk: iscsivmk_ConnRxNotifyFailure: vmhba37:CH:1 T:6 CN:0: Connection rx notifying failure: Failed to Receive. State=Online
2013-05-16T16:08:41.063Z cpu0:2724)iscsi_vmk: iscsivmk_ConnRxNotifyFailure: Sess [ISID: 00023d000008 TARGET: iqn.2001-05.com.equallogic:4-52aed6-56c15d168-d6c0021a85c4f71c-eql-esxi-vol-f TPGT: 1 TSIH: 0]
2013-05-16T16:08:41.063Z cpu0:2724)iscsi_vmk: iscsivmk_ConnRxNotifyFailure: Conn [CID: 0 L: 192.168.90.53:56744 R: 192.168.90.103:3260]
2013-05-16T16:08:41.063Z cpu0:2724)WARNING: iscsi_vmk: iscsivmk_StopConnection: vmhba37:CH:1 T:6 CN:0: iSCSI connection is being marked "OFFLINE" (Event:6)
2013-05-16T16:08:41.063Z cpu0:2724)WARNING: iscsi_vmk: iscsivmk_StopConnection: Sess [ISID: 00023d000008 TARGET: iqn.2001-05.com.equallogic:4-52aed6-56c15d168-d6c0021a85c4f71c-eql-esxi-vol-f TPGT: 1 TSIH: 0]
2013-05-16T16:08:41.063Z cpu0:2724)WARNING: iscsi_vmk: iscsivmk_StopConnection: Conn [CID: 0 L: 192.168.90.53:56744 R: 192.168.90.103:3260]
2013-05-16T16:08:41.063Z cpu12:2701)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.64ed2a85165dc1561cf7c4851a02c0d6" state in doubt; requested fast path state update...
2013-05-16T16:08:41.063Z cpu12:2701)ScsiDeviceIO: 2309: Cmd(0x412441465980) 0x28, CmdSN 0x800000fd from world 4863 to dev "naa.64ed2a85165dc1561cf7c4851a02c0d6" failed H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2013-05-16T16:08:41.063Z cpu12:2701)ScsiDeviceIO: 2309: Cmd(0x41244037cbc0) 0x28, CmdSN 0x80000080 from world 4863 to dev "naa.64ed2a85165dc1561cf7c4851a02c0d6" failed H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2013-05-16T16:08:41.092Z cpu12:2060)ScsiDeviceIO: 2309: Cmd(0x412441bb3300) 0x28, CmdSN 0x8000002e from world 4863 to dev "naa.64ed2a85165dc1561cf7c4851a02c0d6" failed H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2013-05-16T16:08:41.092Z cpu12:2060)NMP: nmp_ThrottleLogForDevice:2318: Cmd 0x28 (0x412441baeec0, 4863) to dev "naa.64ed2a85165dc1561cf7c4851a02c0d6" on path "vmhba37:C1:T6:L0" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x5 0x24 0x0. Act:EVAL
2013-05-16T16:08:41.092Z cpu12:2060)ScsiDeviceIO: 2309: Cmd(0x412441baeec0) 0x28, CmdSN 0x80000059 from world 4863 to dev "naa.64ed2a85165dc1561cf7c4851a02c0d6" failed H:0x2 D:0x0 P:0x0 Possible sense data: 0x5 0x24 0x0.
2013-05-16T16:08:41.146Z cpu13:5262)ScsiDeviceIO: 2309: Cmd(0x412441834100) 0x28, CmdSN 0x80000040 from world 4863 to dev "naa.64ed2a85165dc1561cf7c4851a02c0d6" failed H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2013-05-16T16:08:41.170Z cpu12:2060)ScsiDeviceIO: 2309: Cmd(0x4124403a6c80) 0x28, CmdSN 0x8000006a from world 4863 to dev "naa.64ed2a85165dc1561cf7c4851a02c0d6" failed H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2013-05-16T16:08:41.200Z cpu8:2056)ScsiDeviceIO: 2309: Cmd(0x41244037b5c0) 0x28, CmdSN 0x800000c9 from world 4863 to dev "naa.64ed2a85165dc1561cf7c4851a02c0d6" failed H:0x2 D:0x0 P:0x0 Possible sense data: 0x2 0x3a 0x1.
2013-05-16T16:08:41.218Z cpu11:2088)WARNING: dell_psp_eql_routed: psp_eqlUpdatePathSelection:1014:Path is reported as inactive (0x50475304)
2013-05-16T16:08:41.233Z cpu2:2201)WARNING: dell_psp_eql_routed: psp_eqlUpdatePathSelection:1014:Path is reported as inactive (0x50475304)
2013-05-16T16:08:42.072Z cpu7:2209)ScsiPath: 4963: DeletePath : adapter=vmhba37, channel=1, target=6, lun=0
2013-05-16T16:08:42.074Z cpu15:2063)WARNING: NMP: nmp_DeviceStartLoop:721:NMP Device "naa.64ed2a85165dc1561cf7c4851a02c0d6" is blocked. Not starting I/O from device.
2013-05-16T16:08:42.074Z cpu13:2061)WARNING: NMP: nmp_DeviceStartLoop:721:NMP Device "naa.64ed2a85165dc1561cf7c4851a02c0d6" is blocked. Not starting I/O from device.
2013-05-16T16:08:42.074Z cpu7:2209)WARNING: ScsiPath: 5022: Remove path: vmhba37:C1:T6:L0

Thanks for any information you can provide.

Anonymous

5 Practitioner

 • 

274.2K Posts

June 3rd, 2013 15:00

Whenever a connection is moved, either by the array for balancing IO across available array interfaces, page movement when you have more than three members in a pool, or with MEM installed it will periodically move connections, you WILL see redundancy lost messages.  As you noted, by the time you check the paths the connection has already logged back in as it should.   For that briefest of periods, less than a second typically, that session is gone, and a new one is being created at the same time.  The volume is still accessible via the other sessions.   So it's an info message, not an indication of a problem.  When the array balances slices to a new member, in pools more than three, expect to see numerous, simulaneous messages from all the hosts connecting to that datastore.  This flurry is completely normal.  

Anonymous

5 Practitioner

 • 

274.2K Posts

May 16th, 2013 12:00

Please open a support case for this.   ESX will always report these "errors' in their logs, whenever a network balance occurs.  They will need diags from both arrays, and vm support files to triage this.

Regards,

ndjake

5 Posts

May 16th, 2013 12:00

Thanks Don.  I've had a case open for a week or so and they are reviewing the logs now.  I was just wondering if anyone had similar experiences.

Azriphale

6 Posts

May 31st, 2013 08:00

I'm not 100% sure it is the same issue we seem to have but it sounds like it might be related. We get the same events logged in the Eql event log and the vmk logs seem more or less the same - sorry there is way too much text to check it. 

We have:-

  • 3 Eql boxes in a group, all running 6.0.2 (R305616) (H2) firmware.
  • A 6 host esxi HA cluster running 5.0U1 build 623860.
  • Hosts are all PowerEdge R815 and have six nics; two Intel and four (on the motherboard) Broadcom.
  • The iSCSI SW adapter is set to use one of each nics so hopefully eliminate issues with VMware nic drivers and firmware.
  • Both vmnics are on a single switch and pinned to a single vmk port (vmk2 and vmk3).
  • We also have vmk1 port with a lower ip bound to both physical nics.
  • Delayed ACK has not been disabled but testing has shown that this is not the cause. Our bandwidth usage is nowhere near high enough for this to be a concern (I like over engineering :emotion-2:).
  • The Eql boxes and hosts are both connected to a 4 switch Cisco 3750G stack. The stack runs two vlans; one for VM management and monitoring, the other for iSCSI and an nfs host. Client traffic goes through another switch.
  • Flow control is enabled end-to-end.
  • Jumbo frames are in use.

We also get periodic reports of path redundancy being lost by the ESXi hosts. We can actually reproduce these by rebooting an individual ESXi host. Once the host has rebooted, it will lose path redundancy on at least one volume as it runs the load balancing. We used this to eliminate delayed ack as a possible cause. It is a really odd issue as it seems to have no effect on performance. We also tried reverting MEM back to  1.1.0 from 1.1.2 without sucess.  Despite that, I've still got a number of things I need to eliminate before I'm ready open a support call. 

The thing that particularly bothers me is that the disconnect seems to happen before the connect., which is (in my view) logically the wrong way round for a load balancing process. 

Did you get anywhere with your support call?

Anonymous

5 Practitioner

 • 

274.2K Posts

May 31st, 2013 08:00

Hello Azriphale,

When either the array or if MEM is installed, moves a connection, ESX will report that as an error.  We have seen some cases where MEM is moving them around more often than needed.  

Re: DelayedACK won't impact the load balancing, it only effects performance.  So too does Large Recieve Offload (LRO).

Our standard suggestion is to turn both of them off for best results.

The ESX build it quite old though.  Current build if well over one million.

You might want to keep an eye on the 3750G stacking bus.  The "G" model sends all packets over the stack, even when the packet destination is on the same switch.  The "E" and "X" models have a faster bus that doesn't do this.  

Regards,

gerrybarnett

129 Posts

June 3rd, 2013 14:00

Hi,

I am seeing what seems the exact same issue. I have very similar setup with 5 node cluster (5.0U2) and 6 member EQL array (6.02), Have implemented all recommended ESX/iscsi settings from this forum. Fairly frequently I see path redundancy lost messages from Vcenter but when I check all paths are active/OK. I can trigger these alerts by creating new EQL volumes and presenting to the ESX hosts so looks like rebalancing and settles immediately. I have not opened a case as yet as site is under development. Would be interested to know if anyone gets to the bottom of this as it is annoying if nothing else.

ndjake

5 Posts

June 3rd, 2013 14:00

Hi Azriphale,

This ticket is still open.  I continue to see the path redundancy lost messages across my hosts.  Initially, I didn't have the delayed ack and iscsi timeout settings set correctly on each node, but I've corrected that issue and I've also disabled LRO on each host and the issue persists.  Again, I'm not seeing any type of performance issue and I don't see any type of I/O loss during these events which last usually less than 20 seconds.  I suspect they are MEM or NLB events but I don't see that clearly stated in the EQL Group logs or vmkernel log.  I'll update when we get it sorted out.

Jake

gerrybarnett

129 Posts

June 4th, 2013 02:00

Do you know if there is a way to reduce the vcenter spam in this scenario, especially when the "problem" is just a legitimate transitioning of resources.

ndjake

5 Posts

June 4th, 2013 05:00

Does the EQL group note the event in its log as a rebalance?  The first entry I see in the group event log is usually something like this:

iSCSI session to target '192.168.90.102:3260, iqn.2001-05.com.equallogic:4-52aed6-dc615d168-d000021a7ff4f6cb-eql-esx-vol-c' from initiator '192.168.90.42:56606, iqn.1998-01.com.vmware:VHOST04-1123e289' was closed.   Logout request was received from the initiator.

I thought I saw someone post an entry online (can't find it now) that showed the array actually requesting the rebalance which was then followed by the logouts/logins.

Anonymous

5 Practitioner

 • 

274.2K Posts

June 4th, 2013 05:00

Not that I am aware of.   Only VMware would know for sure.   Ideally, they would only issue the alarm if the error persisted for a period of time.  Say 10 seconds or more.

Anonymous

5 Practitioner

 • 

274.2K Posts

June 4th, 2013 06:00

In either case, an event in the logs will be logged.  If the array Network Load Balancer (NLB) determines if there's an imbalance of sessions over the available interfaces.   The verbage varies a little based on FW version.  If MEM or any of the HIT kits are in use, especially with 6.x EQL FW, you will see the logout request from the initiator.

Origin3k

4 Operator

 • 

1.9K Posts

June 4th, 2013 09:00

I have to manage around a dozen of ESXi Cluster + (Single) EQL installation and as soon as we enable the vCenter Alarms we got the email notification about "lost path blabla"  for random datastores on random hosts. The host reconnectet without a problem in the same second. We see this  per once day... once per week. For those customer who are to nervous we disable the notification :(

Only in one case we have seen that EQL ALB went crazy. The 4th. array (Same RAID and capacity) was added to a pool and we see the LUN migration to the new member as expected. 2 weeks later we deploy a volume and than the hell opens and the group manager starts to play "ping pong" and move volume between the members.

With the "soft limit" of spreading a volume over up to 3 members and when a volume "leave" a member all 10 ESXi hosts produce email notification about losing the connection. Since we have only 1 and 2TB volumes the process for moving around takes some time. We see IOPS/Traffic in SANHQ as a flat line in this time for a Volume. As soon as the migration was finished and a short periode a nother volume starts moving.

This goes for about 6-8 weeks and Dell support means that this is an expecting behaviour :/ - The moving stops 3 weeks later.

Regards,

Joerg

View More

View All

No Events found!

Top