Any issues with XtremIO upgrades from 4.0.15-20 to 4.0.15-24?

4.0.15-20 to 4.0.15-24 is an NDU (non disruptive upgrade) process. Dell EMC Support will run pre-upgrade checks in your environment to make sure we are aware of any potential issues before the NDU process starts.

Hi,

We have got this upgrade done (same versions) as there was an advisory recommending upgrade. We didn't faced any issues. Maybe EMC can suggest more on this.

Thank you, the above is great news, glad to hear your upgrade went smoothly.

Avi, thank you for your comment, and yes my earlier updates were NDU, but the 4.0.15-20 to 4.0.15-24 was NOT a NDU.

EMC did all the pre-checking and everything passed, but a production database crashed during the update.

I will update this post when I get the RCA results, because many things can cause issues during an update.

In my case the previous update was done less than 90 days earlier without issues.

I believe my hosts were all configured correctly, so that is why I was asking the community if anyone else has experienced issue with the specified version jump from 4.0.15-20 to 4.0.15-24.

I was very surprised the upgrade had issues because the previous upgrade had no issues at all.

4.0.15-20 to 4.0.15-24 is about as simple an upgrade as they come.  There are no "firmware" changes in this version, so there's no need to reboot any of the storage controllers - just a quick blip as we reload the new XIOS code, and that's it.

This is one additional step that the person carrying out the upgrade will do due to the fact that there wasn't an reboot, but that's completely transparent.

There's certainly no expectation of any problems for this (or any other) upgrades. Most of the times we see issues during upgrades it's down to things like multipathing or timeouts not being set correctly on the host, but I'm sure support will be working with you to try and work out exactly what went wrong in this case and get it fixed.  We're actually working on a set of scripts that will validate the host-sided configuration before an upgraded (or at any other time) to help avoid such issues - the one for VMware is in final testing, and (physical) Windows and Linux will follow shortly.

Hi Mdeitrick  and Scotthoward,  thank you for your input. 

EMC Support suggested I open a support case with the switch vender.  Which I have done and that investigation is taking place.  No root cause identified yet, but we are still digging. 

Just an update, the switch vender did not find any FC switch issues shortly before, during, and after the upgrade.

Did the Fabric analysis mention anything about credit starvation or slow draining devices? Are they any ISLs present and were all switches review (not just those with links to the XtremIO)? I had one recently where a reset of unrelated switch ports on a switch that was connected via ISL resulted in excessive noise. If you would like a second pair eyes to review the case, please provide the service request number and I'll have a look.

mdeitrick,

EMC support just requested I have the vender check for flapping on the port. No flapping was detected.

Regarding SR #

Service Request Number

07017682

Former Service Request Number

85818360

I am still waiting for official RCA.  Preliminary info that was provided.

Referenced two possible reasons for the Outage.

The AIX fix which public knowledge, I just did not know about it.

Make sure your  AIX systems have IBM IV87492: IMPROVE HANDLING OF ABORTED COMMANDS APPLIES TO AIX 7100-04 - United States  installed.

The other possibility that EMC referenced, was an XtremIO bug, but since the preliminary info has EMC confidential all over it, I will let someone from EMC disclose EMC bug issue and details.

The bug you speak of is related to a reset of the Platform Manager (PM) due to a bug in the qlogic FC firmware. To understand the problem better you need to understand a little about how the NDU process works. During the NDU, the PM enables the HBA FW Heartbeat mechanism (typically used to monitor the progress of controller reboots). In "noisy" SAN environments the HBA FW Heartbeat may unexpectedly timeout, causing a reset of the controllers FC drivers. If this occurs on more than one controller at the same time, this could result in the System Manager (SYM) temporarily closing I/O gates. It takes a lot to hit these conditions and the environment is only at risk if the HBA FW Heartbeat remains active during moments where the SAN experiences excessive "noise" - this could be related to many RSCN events, CRC errors, etc. I'll review the service request and consult with the owner in the hope of driving a quick a response.

EMC Final RCA indicates the issue was the missing AIX patch.  I have a XtremIO upgrade scheduled this week.  The upgrade scheduled this week is a bigger jump in code  4.0.2-80 to 4.0.15-24.  I have the AIX patch applied to the AIX servers.

Thanks for the follow up. Good luck with your future upgrade. I hope the experience is positive and goes a little smoother. Feel free to reach back out to us if you have any further questions or concerns. Thanks again!