I have several PS6000 / PS6100 running FW 5.2.11 and 6.0.x configured for async replication. I have the following issues:
Roughly every 30 minutes my backup EQL reports the following error in its main log:
ERROR 21.10.15 17:06:33 node1 Partner cluster3-slow2: iSCSI: login timed out. Make sure the partner IP address is correct and reachable.
At that time no replication job is scheduled. All replication jobs run fine. This only happens to one of my many replication partners. I.e. backup never complains about cluster3-slow or cluster3-slow4 which it is also replicating.
Nothing else is logged, I can ping all interfaces from every EQL in question both internally (from the respective partner) and externally.
Now I'm out of ides what might be happening here...
upgarding FW is my last resort since we have a zero-downtime target and have to migrate all volumes prior to upgrades. Also the error happens both between EQL of the same and of different versions, so this can be ruled out as main cause.
yeah I know about the age of the firmware, but I believe an error of the sort I am seeing right now would have shown up if it's just for a bug in the firmware. Besides that I see that old versions get a long time on their last iteration before they are retired. Seems that no major bugs where found in this period.
In short: I don't believe in updates "just in case" on utility devices like storage arrays.
And while I recognize the need for support providers to keep their clients on the narrowest possible base I have almost never experienced a problem this obvious beeing fixed by just upgrading software.
That said: I have just upgraded my backup EQL to the latest 7.1 and the error is still there - only on cluster3-slow2.
And just a short note on your points regarding timeout settings: Back in the good old days of firmware 4.2 a controller restart took around 20 seconds on a PS6000 till the initiators were online again. Today a restart needs at least 60 seconds and while my initiators survive the timeout (I'm using Xen/Debian btw) a 60-100 second "hang" does not match our zero-downtime target. I am aware that we have to live with that in case of a failover. But that's a different story...
Thanks for your time I will report back any findings for documentation.
I'm back with an update: I brought the outgoing member to 6.0.11 and a new PS6000 unit - the error persists. So I'm checking for issues outside the Equallogics now. I've gathered traffic dumps via a mirror port but cannot see anything suspicious atm.
BTW: Upgrade from 5.2.11 to 6.0.11 needs to be done through the CLI which restarts the whole array without doing a failover. That leads to around 5 minutes downtime. Good I did not try this on a live array...
I think in the 6.0.7 Release Notes my issue is documented:
Under certain conditions, when updating an existing array to firmware v6.0.7 using the Group Manager GUI, you might be prompted to complete the update process by issuing a restart using the CLI instead of the GUI. This is due to RAID check enhancements that were added to firmware v6.0.7. Once you are running v6.0.7 or later, updates to firmware will not require a restart using the CLI.
The corresponding message during the Update is:
Array firmware update from version V5.2.9 to V6.0.11 failed. Reason: The array cannot be restarted using the GUI because of a RAID issue. Use the CLI 'restart' command to continue.
timevers
7 Posts
1171
0
Posted October 21st, 2015 13:00
Hi,
upgarding FW is my last resort since we have a zero-downtime target and have to migrate all volumes prior to upgrades. Also the error happens both between EQL of the same and of different versions, so this can be ruled out as main cause.
Regards
Tim