RHEL 6.4 + Hit 1.3 + FW 6.0.11: iscsi connections sometimes not restored

What model array do you have?   Possibly a 6110/4110?

I would suggest opening a support case so the logs from array and host can be reviewed.

How long do you wait before restarting the service?   Every two minutes the service will check the current connections.   I would expect the open-iscsi initiator to see that a connection was lost and needs to be restored.

Regards,

Waiting about 30 minutes didn't help :( , the only way was a restart of ehcmd.

PS-M4110XS

Single interface arrays have different rules with HIT but I think a review of the logs is needed.  With single interface arrays it will create / remove as needed based on load.   Since there's just the one interaces with vertical failover there's no requirement for two connections.

Donald, first of all many thanks for your answer.

I'm going to get the logs of my hosts and then I'll post the relevant info.

Thanks again.

You are very welcome.  I hope you do submit a support request for this issue.  If there is a problem, the sooner the dev folks get it the better.  

Hi, some updates and more detailed info: we have opened a SR (# 900766879) and we have sent the logs requested, but so far we were only told to enable STP on our switches (...) and that maybe the culprit is some kind of misconfiguration in the hosts. Now the SR should escalate, but maybe you could get more info than me about this SR :D .

Reproducing the issue with a colleague, we noticed that:

1) We restart the "A" switch. When the switch is up and running again, the initiator reestablishes two connections.

2) A few minutes later then we restart the "B" switch too: again, the initiator reestablishes two connections as soon as the second switch backs online.

3) Now (and that is the issue) if we restart the "B" switch" again, when the switch backs online the host reestablishes immediately TWO iSCSI connections out of two, but a few seconds later the initiators disconnects the second connection and uses only one path. I'd like to point out that seems to be a "clean" logout of the initiator, we see the logouts on EQL event log. At this point, only restarting ehcmd daemon on the hosts restores TWO iSCSI connections.

I understand that is not a "real world" scenario (a triple switch failure in a few minutes, I've never seen before such a thing), but, due to stiff business requirements, I have to configure a 99.99999999999...% available system. Actually I don't know if this could be considered an issue or this is only the normal behavior of EQL HIT using a one-port SAN like the M4110XS.

I think my hosts are correctly configured, performances are really good and the storage survives a "normal" failover scenario with no problems; I followed EQL's best practices configuring  switches, hosts and storage, as I usually do.

Hello,

This  sounds more and more ike expected behavior with single port array.  One way to verify this is, remove the HIT kit from the equation, and just use the standard Linux MPIO instead and repeat the test.

If the connection is logged out with "logout request from initiator" or similar message, that is HIT/LE logging out the connection on purpose.

With vertical failover support, even with one connection to the array, one a switch failure the existing connection is maintained by using the physical port of the other controller, that's connected to the surviving switch.

Re: 99.9999   I'm always curious to know how companies are configuring their environment to even 7x 9's.  %99.9999999 is 3.15 seconds of unavailability per year total.  In the case of EQL, FW upgrade or CM failover will exceed that number.   What do you do on your hosts to achieve this?     At another company I moved the majority of servers from Windows to Linux.  Ended up with greater uptime but I could never get even seven nines without clustering everything and completely mirroring all storage.  Just rebooting for updates defeated it.  

Even the MPIO in Linux/Windows takes longer than that to determine path has failed and resume on the remaining paths.

Regards,