Windows 2012 cluster iSCSI Errors 9, 39, 129, 7, 20, 1

Question

Hi,

We are using Windows 2012 iSCSI cluster which runs on two w2012 hyper-v vms. All storage used is EqualLogic LUNS and HIT (the latest) is installed on VMs. Sometimes during some heavy load (backup, restore) we receive plenty of Windows errors like:

EventID 9: Target did not respond in time for a SCSI request. The CDB is given in the dump data.
EventID 39: Initiator sent a task management command to reset the target. The target name is given in the dump data.
EventID 129: The description for Event ID 129 from source iScsiPrt cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.
EventID 7: Initiator failed to connect to the target. Target IP address and TCP Port number are given in dump data.
EventID 20: Connection to the target was lost. The initiator will attempt to retry the connection.
EventID 1: Initiator failed to connect to the target. Target IP address and TCP Port number are given in dump data.

On EqualLogic we have erors:

Level: ERROR
Time: 01/07/13 15:46:42
Member: EqualLogic
Subsystem: MgmtExec
Event ID: 7.4.3
iSCSI login to target '192.168.1.44:3260, iqn.2001-05.com.equallogic:4-52aed6-d60bdf198-6970019b0db5194d-volume1'
from initiator '192.168.1.178:61060, iqn.1991-05.com.microsoft:university.domain.local' failed for the following
reason:
Management did not respond to request.

I have tried unsuccessfully:
netsh int tcp set global RSS=disabled
netsh interface tcp set global autotuninglevel=disabled
netsh int tcp set global chimney=disabled

Noticed that performance with iSCSI volumes slows down then the errors occur. How could we investigate and solve this?

Thanks

awer1 · Answer

It is 5.2, not sure about the switches as network team works with them, is anything else we could do maybe something from windows side. Could this be helpful, just fix is not for Windows 2012:

support.microsoft.com/.../2522766

Thanks

awer1 · Answer

If it is network related problem any idea whre to look at? as going to network team we will need some suggestions, Thanks

awer1 · Answer

Switches are Force 10 S60, all configuration I have for switches: Flow Control - Enabled or On; MTU Size - 9000. On servers we have default 1500 MTU. Not sure regarding PORTFAST but as this solution was designed for iSCSI might be PORTFAST was enabled.

HIT configuration:

Maximum sessions per volume slice = 2

Maximum sessions per entire volume = 6

EgualLogic firmware is 5.2.5 as I see on controllers. But it should work even with this version.

So we came that either PORTFAST is not enabled or it is Windows 2012 or EqualLogic storage issue?

Thanks

awer1 · Answer

Thank you for explanation about switch buffer. one more question how to open the case? thanks

Latif_ee7dc4 · Answer

Awer1, was there a conclusion from Dell for this? We have a very similar configuration (Same OS and firmware. We recently experienced similar issues.

rdf6 · Answer

I have experienced this in a Server 2008 R2 Hyper-V cluster, getting these kinds of errors at both the host and guest level.  No pattern to which host or guest would throw the errors, was just so random.  Did extensive troubleshooting with Dell and tried several HIT Kit versions.  Dell storage engineers really believe this is an issue with the built-in Microsoft iSCSI initiator as the EqualLogic arrays are basically reporting that the other end of the connection (the host) disappeared unexpectedly.  I even opened a case with Microsoft, but, that was a complete waste of time because the MS engineer considered the issue resolved because the issue never occurred any time while we were actively troubleshooting, nor could I replicate the problem on-demand.  Didn't matter that the log was full of these errors from just hours before engaging on the call.  Anyway, I never found a solution and eventually gave up and moved on with my life.

dpitdept · Answer

For those still investigating this issue. I accidentally stumbled upon what caused this issue in our environment while troubleshooting a cluster freezing issue.

The moment these events showed up was right after I installed the Broadcom NetXtreme Gigabit Ethernet 214.0.0.0 driver version. Prior to that we were using the default 17.2.1.0.

Currently we are running Windows Server 2019, but we've experienced this issue on 2008 R2, 2012 R2, 2016, and now on 2019, so it's most likely not OS related, but something with drivers for the NICs. I hope this can help others out.

dwilliam62 · Answer

Hello,

In regards to Server 2019. Some recent KBs have resulted in connections issues with PS Series SANs.

See this thread.

https://www.dell.com/community/EqualLogic/EQL-iSCSI-login-failures-after-MS-patches-KB4497934-KB4499177/m-p/7314811#M13841

At login or a connection re-balance, Errors 39, 129 are normal. The login_redirect process in iSCSI that PS uses generates a specific error the tells the iSCSI initiator not to login into discovery address but an IP address the array sends to the server during login.

Regards,

Don

EqualLogic

Windows 2012 cluster iSCSI Errors 9, 39, 129, 7, 20, 1

Was this post helpful?