Looking for feedback on what we experienced yesterday which caused a disruption on certain hosts in our production and development VM environment.  I currently have an SR open but haven't been able to get anyone from the Isilon support team to start working on it (24 hours and counting so far, 3 follow-up calls... sounds like they are slammed with some severity 1 issues right now and in fairness, I stabilized our environment by suspending the node we had issue with).  Ultimately I need assistance with a root-cause analysis and determining if I can place the troubled node back in SmartConnect duty.

Our current x410 4-node cluster is running OneFS 8.0.0.4 (target code) with highly recommended patch-188239.  Because we are exclusively NFSv3 right now and have experienced some of the issues described as fixed by the patch, I opted to install patch-191603 (a roll-up to address multiple NFS and SmartConnect issues).  Over the last year we've done rolling reboots to the cluster countless times (patches, OneFS upgrades, etc.) and never experienced a disruption... until now it has been completely transparent in all cases.

We have a very light workload (relative to the power of the cluster) so as luck would have it, almost all our NFS clients were on the 4th node.  The patch itself appears to have installed correctly and did the expected rolling reboot.  When it got to the final 4th node, SmartConnect did what it was supposed to do and distributed the clients to the other three nodes, and when the 4th node came back up it took some of the clients back.  In fact, it was perfectly balanced by the same number of IP addresses per node when all was said and done.  Very symmetrical.

The problem was that clients assigned to node 2 immediately lost all connectivity to storage, it just disappeared.  No indication whatsoever of any problems on the Isilon side, my first hint of trouble was the VM guys going into a panic about VMs suddenly going dark from certain hosts.  A quick check found all those hosts on node 2 and I saw that the throughput for node 2 was exactly zero, nothing in or out.  I "fixed" the issue by suspending node 2 from SmartConnect duty and rebooting the node.  This caused the SmartConnect to immediately send those clients to the other 3 nodes and connectivity was restored instantly to those hosts (they were able to see the storage again).

So I'm looking for feedback from anyone else who might have experienced a similar phenomenon either with this patch or in general.  I will still expect the Isilon support team to assist with root-cause analysis but if anyone has ideas or advice I'd love to hear it.  I still currently have node 2 suspended from SmartConnect duty waiting to discuss with support on whether I can enable it again.  Thanks.