This post is more than 5 years old
6 Posts
0
2835
Upgrading PS6500 from 7.0.9 to latest and VMware 6.5U1
We have a nest of two PS6500 (10k and 7k2) which are in support and running our VMware clusters (2 clusters each of three R720s). The PS6500 firmware is currently 7.0.9 on both and as we are upgrading VMware to 6.5U1 I am assuming that to keep MEM compatibility we have to upgrade the firmware to v9.1.xx in steps as described on the support site, and install MEM 1.5.
I have several questions:
- I can use the EQL Update Manager to update the offline controller and then fail over to upgrade the second. During the failover time the iSCSI connections will "fail" and have to login again. iSCSI connections are about 250 to each and our IOPS are low (300 peaking to 1k) according to SanHQ.
Q. I'm assuming that we can do this production system upgrade without having to shutdown any VMs? I can shutdown roughly half during our quiet period but other VMs are somewhat essential and I plan to keep them up. (Yes, we have backups!)
2. As we upgrade each VMware host, MEM v1.5 needs to get installed and configured on the host.
Q. Other hosts will continue to work with 1.2 (or 1.3, I forget which) despite the firmware upgrade? If not then I assume the performance will be degraded during this VMware update period?
3. What can go wrong!
Q. I'm assuming that a failover could fail if there was a Controller issue but is there any way to test this in advance? Can I failure over the controllers via the Group Manager to "pretest" the controllers?
Finally I must say that after 5-years of use these PS6500s have just worked (knock on wood).
...Andy
Anonymous
274.2K Posts
1
November 27th, 2017 09:00
Hello Andy,
re: 1. Regardless of how you upgrade firmware, it will copy the firmware to both controllers. Then restart the passive to make the upgrade effective. Failover to bring that member up to new firmware level, and restart the now passive older revision controller, to bring it up as well.
Re:Q You can do this to a production system. Best to do it when IO is lower. This reduces the time it takes for the controllers to complete the failover process.
Re:2. Yes MEM v1.2 or 1.3 will continue to work. So performance should not degrade.
Re: 3. I would say the most common issue is write failure on the SD card. Doesn't happen often.
Best piece of advice I can offer is configure and test serial port connections to your controllers. If something should go wrong you will need that in order to triage it. My personal preference has always been to upgrade firmware via the serial port vs. Group Manager or DSUM. You get the most information possible via serial and if something does happen you are right there to see it, and capture any failure information.
BTW, you will need to upgrade to 8.1.x before you can upgrade to 9.1.x. Updated version of 8.1.x and 9.1.x are due out very soon, worth waiting for the before upgrading
Before you upgrade or test failover, have you setup the VMware servers to EQLs best practices?
This Tech Report covers how to do that.
http://en.community.dell.com/techcenter/extras/m/white_papers/20434601/download
There are several timeout values that need to be extended to better handle restarts/failovers. Also within the Guest VMs, there is also a timeout in the OS. Windows can be especially sensitive.
With the firmware downloads on the EQL support site, there is a document on changing that timeout as well.
Regards,
Don
andysibley
6 Posts
0
November 27th, 2017 09:00
Don,
First thanks for a very comprehensive answer. We don't often do PS6500 firmware upgrades as you can tell!
I'm in the process of checking the EQL best practices (have technote) and we're scheduling the firmware upgrade either this week or to after Christmas (more likely following your comments on firmware releases). I'll also get familiar with the serial port just in case.
Many thanks...Andy
Anonymous
274.2K Posts
0
November 27th, 2017 10:00
Hi Andy,
You are most welcome!
One thing I did fail to mention, is also check for drive firmware updates There's a single kit for all drives and models with the firmware downloads. The Website lists the drives or you can just run it and it will check. Also the Dell Storage Update Manager will check as well.
Firmware update is do before holidays
Re: Serial I would not do one w/o it. In fact in our lab all the arrays serial ports are hooked up via a serial mux.
Regards,
Don
Anonymous
274.2K Posts
0
November 30th, 2017 12:00
Hello,
The latest firmware has been released to the website by the way.
Don
andysibley
6 Posts
0
December 28th, 2017 11:00
Don,
Just to let you (and everyone else) know that the update from 7.0.9 to 9.1.4 was painless using the Update Manager. As you suggested I monitored the controllers via the serial port and there were no issues at all. The serial port does help especially when the Update Manager reports losing connection but you see the activities going on via RS232.
BTW this was on a production system (1st maintenance in >3 years) and although I shutdown the majority of VMs I didn't see any issues with the ones I left up and running.
Many thanks...Andy
Anonymous
274.2K Posts
0
December 29th, 2017 03:00
Hello Andy,
That's great news, thank you for the update. Glad it all went well. In my lap we use serial muxs to make monitoring the serial ports super easy. I highly recommend them for arrays, switches, etc.. They come in handy.
If you haven't looked at the OS considerations PDF that's located with the firmware download I suggest you do so. Make sure all your servers and VMs are configured with the proper timeouts. Not just for firmware upgrades, but in case there's a controller failover during production this will allow servers to ride out the failover.
Regards,
Don