Dell PS Series Equallogic firmware v9.1.5 and v8.1.14 have been released

Hello, I notice that 9.1.5 has been pulled from the support site as of this morning.  I would like to know if this is an error or if there is a problem with this version.  I have already installed 9.1.5 on one of my groups and was planning to update two other groups soon.  Please advise. 

Hello,

 An issue was recently identified in Dell EMC PS Series firmware versions 9.1.5 and 8.1.14 which could, in rare circumstances, negatively impact array operations. If you have previously downloaded either of these firmware versions and have not yet installed it, Dell EMC recommends that you not install it at this time. An updated version of the firmware will be made available shortly to replace the affected versions. If you have installed either of these firmware versions, please contact your Dell EMC technical support representative. 

Regards,

Don

Since you don't yet have a fix, can you be a little less coy about what the issue is, who may be affected by it, and what mitigations there are currently?

Seeing as the upgrade to 9.1.5 was the only thing that fixed a different issue we were having where we were seeing latency around 15 *seconds* (15000 ms) I am keen to hear a tiny bit more about why I need to roll back.

I called in.  The support engineers were not given details of exactly what the problem is yet or how it gets triggered.

If you are on 8.1.14 the easy option is upgrade to 9.0.9

If you are on 9.1.5 they can give you the secret commands to allow downgrading without reset I guess.  (You have good backups already right)  

Or you can live with the unknown peril until a patch arrives.

Hello,

 You can downgrade from 9.1.5 to 9.1.4 without doing a reset.  Same goes for 8.1.x.  You use the same process as upgrading using the CLI.  Transfer the 9.1.4 kit to the array and at the CLI type 'update'.   The prompts will guide you through the process.   Then restart array to complete the downgrade.

Trying to downgrade a major rev, like 9.1.x to 8.1.x would require a reset of the array.

 Regards,

Don

We need some more information here.  Given the nature of these devices many of us can not just get an outage window to roll back firmware without knowing exactly what is at risk and under what circumstances.

I called into support and spoke to an engineer who tried to get more information for me and all he could come back with is the same message we received in the email, that in some rare circumstances the array could be negatively impacted.

I don't think it is too much to ask that we be told, directly, what the circumstances are, and exactly how impacted we will be.

Dell is putting a lot of stress on a lot of system admins by withholding this information.  I get to sweat it out for at least the next several days until I can get a window scheduled to roll this firmware back.  For all I know both of my SANs that I just upgraded to 9.1.5 this past Saturday are about to fall out from underneath me and I don't even know if that just means a momentary hiccup in connectivity or total loss of all data.

Hello,

 For the moment that is the messaging we have. No one thus far affected has lost any data.  The array goes offline, making the data unavailable until the issue is resolved.  Then the firmware downgraded to prevent the issue from reoccurring. 

Regards,

Don

I am NOT currently having the previous issue, BECAUSE of 9.1.5. Asking me to roll back would also bring back my previous issue, which (if I wasn't clear enough before) makes the Array functionally useless. I am currently running normally, except for this mysterious new firmware issue that I have no idea how to deal with or mitigate. I'm trying not to drop this issue on my Juniors, but I'm leaving for my vacation next week.

Can you at least confirm that when a solution is ready, that it will just involve a new firmware update? If that is the case, I know they can handle that no problem, but if it's going to involve a rebuild or some other kind of hot shenanigans it'll have to wait until I get back (which leaves us in this unspecified critical state for much longer than I'd prefer).

Can we get more information on the potential cause of the issue?  Is it happening after a certain period of uptime, during a snapshot, during a specific proccess?  Anything you can provide will help us know how much we are risking.  If Dell doesn't know what is causing it, just say that.

I do not have the luxury of doing this downgrade without scheduling downtime.  These units are the shared storage for a Hyper-V cluster, which from previous experience does not like controller failovers.  It doesn't cause a failure of the cluster but it does wreak havoc as the cluster nodes think there is a storage issue and begins failing roles around.

So I have no choice to wait until this weekend at the earliest to perform this work.  All the while I get to wonder if at any moment I lose my entire production environment.

Hello,

 Yes, the information I have is that a new firmware release is being created.  When posted it will only require a normal firmware upgrade process to implement.

 Regards,

Don

So, three weeks later - is there ANY new information about the firmware update? I am becoming more and more nervous about my situation and as time moves on and NO new information is released its becoming harder and harder to justify a wait-and-see posture on this.

Likely a bug so embarrassing and stupid they refuse to say as it would endanger future sales if the truth were told.

Are there any updates on this?  Knowing what might cause it would be helpful also any ETA on a fixed version?

Thanks!
Eric