tsap1
1 Nickel

RHEL VMs lose connection to Isilon NFS mounts randomly

Has anyone experienced hosts randomly losing connectivity to the isilon?  here is a scrubbed output from the server that lost it's mount.  you will see that a df-hT shows it lost connectivity.  I verified the settings are correct in /etc/fstab.  I unmounted and mounted and it came back lickity-split.  I have around 200 connected clients and it randomly occurs.  Any assistance would be appreciated.  I've also opened a case, so if know one knows I'll post the resolution once found.  thanks in advance

[user@rhel-host ~]$ df -hT

Filesystem    Type Size  Used Avail Use% Mounted on

df: `/apps/rhelFS/nfs': Input/output error

[user@rhel-host ~]$ more /etc/fstab

isilon-nfs01:/ifs/data/nfs/customer/rhelFS-nfs01 /apps/rhelFS/nfs      nfs rw,vers=3,hard,bg,nolock,dirsync,acdirmax=1,acdirmin=1,lookupcache=none 0 0

[user@rhel-host ~]$ sudo su

[root@rhel-host user1]# umount /apps/rhelFS/nfs

[root@rhel-host user1]# mount -a

[root@rhel-host user1]# df -hT

Filesystem    Type Size  Used Avail Use% Mounted on

isilon-nfs01:/ifs/data/nfs/customer/rhelFS-nfs01

nfs     95T  9.8T   83T  11% /apps/rhelFS/nfs

0 Kudos
2 Replies
ed_wilts
2 Iron

Re: RHEL VMs lose connection to Isilon NFS mounts randomly

Yes, we a LOT of issues when the RHEL guests were hosted on KVM on Isilon.  When we migrated those same guests to VMware on NetApp filers, the problems all went away.

We saw lots of timeouts in dmesg and /var/log/messages, similar to this:

<datestamp> <guest> kernel: BUG: soft lockup - CPU#0 stuck for 69s!

I did have an open case with EMC about this and I repeatedly heard that the next release of OneFS was going to make it better, but the problems didn't go away until we migrated off of Isilon completely for our virtualized workload.  We're still a significant Isilon customer for our NFS workloads.  I love Isilon but believe in using the best tool for the job and Isilon + KVM just wasn't a good combination for us.

0 Kudos
Peter_Sero
3 Zinc

Re: RHEL VMs lose connection to Isilon NFS mounts randomly

If you changed both, vm environment and filer, you can't tell much about the individual aspects...

In contrast (and not actually comparable to the original question), using VMware Workstation on RHEL 5, we run virtual Isilon clusters with all the OneFS drives (vmdks)  placed on a single NFS mount from a physical Isilon (X200).

Never worked with 6.5.x, works great with 7.0.2.9 and 7.1.1.2 (on the physical cluster).

To the original poster: Which vm environment? How is your network set up? Log messages from Isilon, vm environment and guest RHEL?

Cheers

-- Peter

0 Kudos