Isilon: Upgrade to 8.1.0.4 - Has anyone had an issues?

Hi,

We have upgraded from 8.x to 8.1.0.4, no issue reported till now. It’s just

that I feel the job engine is running a bit slow.

FSA job still does not completes

Regards

On Thu, 11 Oct 2018 at 2:21 PM, TonyHoover21

Just to add, I'm using SMB not NFS, so perhaps that's why I haven't been impacted by what Ed posted.

Isilon OneFS 8.0.0.7 and 8.1.0.4: NFS becomes unresponsive to specific client requests

Same thing here. We have 400 NFS clients connected to the Isilon and several times a day  NFS mounts stalls when a few clients do heavy reading and writing. It seems like the Isilon runs out of resources and lowers the TCP window to 0.

Isilon OneFS v8.1.0.4 B_MR_8_1_0_4_057(RELEASE)

Client OS Centos7

Clients are connected via   Mellanox IB to ethernet PROXY gateways

Because there was no patch, we wrote a script that runs on the Isilon cluster and drops the wedged sessions.  You'll find sessions stuck in FIN_WAIT_2 state.  You can run /usr/sbin/tcpdrop to kill them.

Earlier releases also have the problem - we have clearly documented it on 8.0.0.6 and 8.1.0.3.

We need to get the patch rolled out but now is not a convenient time for the massive undertaking (over 100 nodes) .  Some days I really dislike Isilon QA 

I am not sure we are seeing this problem.

We use static NFS mounts and the problem so far more looks like the use of memory mapped files as one side of the problem. And simultaneous reads and write from the same node as another side of the problem. It also seems like the problem does't spread to other nodes even if they are serviced by the same Isilon node.

Do you BTW have a link to document describing the FIN_WAIT_2 issue?

I just discovered today, that the problem we hit is covered by this patch:

Isilon OneFS 8.0.0.7 and 8.1.0.4: NFS becomes unresponsive to specific client requests

But it seems to unavailable to customers without support. Given the serious nature of this problem I simply don't understand this policy.

Has anybody succeeded to download OneFS 8.1.0.4 - patch-228133?

UPDATE:

According to the latest patch list the correct patch is patch-239649 which I was able to download.

We discovered our own symptoms and workarounds which is how we came up with the FIN_WAIT_2 state symptom.  We've been fighting this for months - long before EMC announced a patch - and long before even our TAM mentioned that there was an issue (and he never mentioned the issue existed on 8.0.0.6).

> it seems to unavailable to customers without support. Given the serious nature of this problem I simply don't understand this policy.

If your environment is in production, you should have a support contract that entitles you to updates.  EMC does have to be paid by somebody to develop the patches and that's where support contracts come into play.  Ideally they would have caught in this QA but their product stability pales in comparison to another big NFS player.

We installed patch-239649 8 days ago and have not seen any NFS hangs since. Before we had numerous hangs every day.

Off Late, we have started getting complains of nfs getting hung on few of the app servers that do heavy IO operations, I am planning to Install patch-239649 this weekend. hope this fixes the issue.

Regards

Vivek

Can anyone who has installed patch 239649 tell me an approximate installation time per node?  I need to request a change window for this and want to make sure I request sufficient time, but long change windows are highly scrutinized here.

Thanks!

It takes about 15 minutes to boot a node.

If you chose sequential reboot multiply that by number of nodes.

We used sequential and with some 50 node it took 12-13 hours.

Just to inform you that we have had so many problems with 8.1.0.4 that EMC now recommends us to upgrade to 8.1.2.0.