Isilon upgrade to 7.2.0.2 and real time issues with this version

Thank You Ken for the information, before we start the upgrade do we need to stop any services, NFS and SMB client access to the cluster which should not interupt our upgrade?

So I'm guessing you've never done an upgrade on an Isilon cluster, in this case.

I believe that the 7.1 to 7.2 upgrade requires a full simultaneous reboot of the cluster, so you will need to account for that. However, no services or jobs need to be stopped during the install process. You just need to warn your users it's giong to go down for like 15 minutes or so.

If it is a rolling upgrade from 7.1 to 7.2 (using the command isi update --rolling), then no services will go down per se, but NFSv4 and SMB will be interrupted as IP addresses move from node to node (NFSv3 will be fine). Whether this is impactful depends on whether there is activity to that particular node which is rebooting and giving up its IPs at that moment.

Also, the amount of time it will take depends on the size of your cluster. For me, a rolling upgrade on my 33 node cluster takes on the order of 6-8 hours. On my 60 node cluster, I imagine it would probably take around 12-15 hours, just because it will wait for one node to reboot and rejoin the cluster before starting another.

A down/up upgrade will take around an hour or two to install (while all services are up and running), and then however long it takes for the nodes to simultaneously reboot. In my case, it's typically around 10-15 minutes.

In either case, I would strongly recommend doing the upgrade from the command line either from a serial console or from within a screen session. You do not want to lose connection to the cluster during the upgrade process, because if if that ssh session you're running the upgrade from closes, guess what, the upgrade process stops. And then you have to re-run it.

Hi Bhuvan,

I'm not going to attempt to write you an upgrade procedure (it's pretty straightforward, just follow the docs), but I recently did the 7.1.1.2 to 7.2.0.2 upgrade, and it was all smooth. No notable problems with 7.2.0.2 yet, aside from what seem to be extraneous or at least over-enthusiastic multiscan job scheduling. NFS seems much improved over earlier versions, but remember that if you have zones, it's going to totally screw with you. Sync jobs are just fine, including to cluster with earlier versions.

--Ken

Hello Bhuvan,

You shouldn't need top stop any services prior to your upgrade. The best information on Isilon OneFS upgrades is included in our Upgrades Info Hub here on the ECN:

OneFS Upgrades - Isilon Info Hub

Please check it out as it also includes a process and planning guide that should be consulted prior to your upgrade. I hope this helps! Let us know if you run in to any other questions.

Hello Ken and Bhuvan,

In my experience, there is no need to stop all connections to the cluster when upgrading. Clients may be disconnected when a node reboots during a rolling upgrade, especially if they are using stateful protocols to connect (i.e. SMB, NFSv4, etc.) From OneFS 7.1.1.x to 7.2.x a rolling upgrade is supported. Please see the release notes for the version you intend to upgrade to in order to determine upgrade compatibility. We do have a link to the release notes in support.emc.com (requires a login to EMC Online Support) within our Upgrade Info Hub:

OneFS Upgrades - Isilon Info Hub

As for which node to connect to, it is recommended that you connect to an IP address of your lowest numbered node in your cluster. I would recommend using one of these:

10gige-1, Node 01  (10.11.13.67, 10.11.13.72, 10.11.13.73, 10.11.13.75)

Also, please do utilize the screen utility should it be available in order to perform the upgrade as it will be persistent should you for any reason become disconnected from your SSH session.

I suspect that refers to the time during reboot, but an Isilon engineer will probably be able to answer that better than I. In my experience however, we do always get a maintenance window, but I have not seen connections drop during an upgrade unless a node is in process of rebooting.

It doesn't really matter which IP you go through; the system is generally smart enough to reboot the node running the upgrade last (if you are doing a rolling). HOWEVER, after you ssh in (to whichever node), DO start a screen session and do the update inside that.

Bhuvan,

You also have the option of engaging the EMC Remote Proactive support team to assist with your upgrade with a valid support contract. If you'd like to know more, you can check out the following KB: http://support.emc.com/kb/193448

(this does require a login to EMC Online Support).

If you would prefer to perform the upgrade yourself, you can always engage EMC Isilon Support by creating an SR if there is an issue.

To create a service request, you have a couple options:

1. Log in to your online account on support.emc.com and go to this page: https://support.emc.com/servicecenter/createSR

2. Call in to EMC Isilon Support at 1-800-782-4362 (For a complete local country dial list, please see this document: http://www.emc.com/collateral/contact-us/h4165-csc-phonelist-ho.pdf)

bhuvankumar,

From a support standpoint we typically do the following:

Bare in mind this is simplified, and the link Katherine Johnson mentioned should be used for a more in-depth breakdown of things to do/look out for.

1) SSH to an IP on the first node in the cluster

2) run pre-upgrade check

3) If everything checks out and nothing needs to be changed run upgrade (from the first node in the cluster)

As carlilek said good idea to do it from a screen in case you loose your ssh connection for some reason.

Node 2 will start to upgrade and reboot, then node 3, 4, and so one. With node 1 being the last node to reboot.

So when the node your on starts to reboot you know you are finishing up, you can simply SSH to another node and wait until Node 1 comes back up.

Hopefully that helps clarify the functionality of the rolling reboot and procedure.

Thank you for the information

as we don't have upgrade support with EMC we are performing the upgrade and yes we will involve EMC for any issue during the upgrade. We are performing this upgrade during maintenance window.

Since we are performing the upgrade remotely and we have GUI and CLI (ssh) option only earlier upgrade we performed through GUI option--> Help--> About This Cluster-->OneFS Upgrade

Now we are planning with CLI ssh option if you have any suggestion pls suggest or i need to follow the ssh process said by Shane

Hello Bhuvan,

I would highly recommend following the process Shane Dekart mentioned above during your maintenance window.

Prior to your maintenance window, the process and planning guide will be your best and most complete source of information for making sure your upgrade goes smoothly. All pre-upgrade tasks should be completed before your maintenance window in an ideal situation. If you have any questions about any of the pre-upgrade tasks, please let us know!

Sure Katie

To my understanding there are two upgrades and which is the best one to perform? Rolling?

1. Simultaneously upgrade where all the nodes reboot at a time

2. Rolling upgrade where one by one node reboots with our confirmation

If we go with CLI upgrade then i think Simultaneously upgrade is best one where we can have our ssh session in contact?

Also would like to know any security patches need to apply after successful version upgrade to 7.2.0.2?

bhuvankumar,

I recommend the simultaneous, as it's one 15-25 minute outage, pending reboot times.

Rather then 15-25 minutes a node effecting state-full connections on that node when it is rebooted IE (SMB NFSv4)

I understand full cluster down time is not always a viable option for that the rolling is what's needed.

A rolling upgrade by default is automated, it reboots each node as it goes.

You can specify the --manual option if you would like which requires you to log into each node and reboot it manually.

I have never personally done this with the --rolling and --manual combined, I don't see any reason as to why it's not a viable option though.

When the node you are ssh'd into reboots you will loose the ssh connection. So either route rolling or simultaneous will at some point disconnect you from the ssh session, and you will need to re-establish connection.

A list of currently available patches can be seen here.

https://support.emc.com/docu50781_Current_Isilon_OneFS_Patches.pdf?language=en_US

Just as an FYI the 7.2.0.2 code contains a memory leak in LWIO.  Our LWIO counts were climbing around 10000 a day, around 1000000 the node will stop responding, the fix is included in 7.2.0.3 and i believe 7.2.1.0 is already out.....no notable issues on 7.2.0.3 after running on it 5 days now. 

One interesting thing I noticed with the Isilon OneFS upgrade from 7.2.0.2 to 7.2.0.3 is 3 level warnings at 18 mins, 36 mins and 54 mins. That means when you initiate a rolling upgrade, it warns you whether you want to proceed with the upgrade at 18, 36 and 54th minute and then it continues with installing the code image and rebooting the nodes in an order.

The drawback is until unless you press or type YES - the warning level will not move to its next level. I felt 18th minute was OK but 36th and 54th minute of confirmation is un-necessary.

Is there a way we can avoid this time issue ? so that we can just hit "yes" and go for a coffee break or sleep.