Please consider moving this question as-is (no need to recreate) to the proper forum for maximum visibility.  Questions written to the users' own "Discussions" space don't get the same amount of attention and can go unanswered for a long time.

You can do so by selecting "Move" under ACTIONS along the upper-right.  Then search for and select: "Isilon Support Forum".

Isilon Support Forum

The first thing to verify is the IP allocation policy.  You didn't mention what it is set to, but what I'm implying from your comments is that it is Round-Robin.  The reason I ask first is if it were something such as connection count, the cluster may respond with the same IP (at least until that node is balanced with the others).  However, again, I am certain it is Round-Robin, but simply want to mention this as it is a very quick check.

The fact that continuous nslookup on the Linux host does return different IP's; however, the ping doesn't confirms for me (imo) that you are dealing with client-site caching.  Remember, nslookup ignores local DNS cache and as a network troubleshooting tool, queries the DNS server directly; ping does (may) not.

For example only, not an issue for you, but in regards to Windows clients (such as Windows XP or 2003) that might exhibit the same, there is a KB article to disable client side caching.  I would look for a similar option in RHEL.

How to Disable Client-Side DNS Caching in Windows XP and Windows Server 2003

Let's see what others suggest first.  In the meantime though, can you verify that there is an interval/delay (5 seconds, 30 seconds, 1 minute, etc) between pings where it does return a different IP?  This would also suggest a client-side cache.  For instance, if between pings you wait, say 30 seconds, and it returns a different IP (per the defined IP allocation policy), it suggests a local client-side TTL of up to 30 seconds.

Now, just to mention it, there is a known 1 sec DNS cache in Windows DNS Server where the server itself caches (we discussed client-side caches above) its response and for each query within a second would return the same IP; however, this would be more of an issue with many different clients that connect within a second of each other and connect to the same node.  This may be something to consider, but would be difficult to reproduce manually (but with nslookup).

mkbkbtekbektekkfvdsvkdamkvmdkvmdavbdvds:MKsdvm;lkwill try the ping suggestion of every 5 sec/30 sec and 1 minute later once I am onsite. Will also do research on any KB for How to Disable Client-Side DNS Caching in Linux.

Hi Vimal,

This is definitely not normal linux behavior.  I agree with Chris above, it looks like some sort of client side issue.  Do you have another *nix host to test with?  Maybe try a couple of other clients to see if the behavior is unique to that machine.

Cheers,

Matt

i Matt,

I have tried this out on 3 different Unix servers. All are with the same results. What I found out was, if I were to run server nscd reload before the ping then I'll get a reply from a new node each time.

Hi All,

Good news, after I stopped the nscd services of the Linux machine, the ping works now. Looks like nscd caches DNS entries.

Thanks for the update.  Peter, nice find.

Peter you linked to a Mozilla URL talking about browser-based DNS caching?

I researched this issue also and found a RedHat bug 656014 (RHEL6) describes this particular issue with nscd and the fix is likely in Errata RHBA-2011-0584 now superseded by RHSA-2013-1605.

https://bugzilla.redhat.com/show_bug.cgi?id=656014

http://rhn.redhat.com/errata/RHBA-2011-0584.html

http://rhn.redhat.com/errata/RHSA-2013-1605.html

Thanks mfoster, that was the link I had intended; sorry I had copied the URL from the wrong browser tab -- P.