Glad to hear you found the issue.
Just to clarify you guys are using the Isilon SmartConnect name to mount the NFS share?
Would you be willing to share your client NFS mount options?
This is what we are using (or at least this is what the DBA sent me from the mount command):
<smartconnect.dns.name>:/<sharename> on /<mountpoint> type nfs (rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp,actimeo=0,vers=3,timeo=600,addr=<isilon-node-ip>)
Additional feedback from tests run by the DBA still prove this is a very strange issue:
A metadata export with no data gives error and data export of one or two tables don't give error and certain databases always give the error and certain don't. But everything works fine when when we use ipaddress to mount the NFS share from Isilon.
This is what they were told are the correct mount options from Oracle.
smartconnect.dns.name:/exported-directory /mountpoint rw,bg,hard,nointr,rsize=32768, wsize=32768,tcp,noac,vers=3,timeo=600,actimeo=0
The only difference I see is calling out the type of mount for the OS on yours but who knows. There was some discussion around the nointr parameter since most modern linux distros ignore that (as well as the rsize an wsize.) NFS mount options can be very frustrating to standardize on since what works for some can sometimes not work for others. They wanted to use this since this is what Oracle had told them and until this little issue this had worked for years. One of the DBA's also put this out there in our hipchat room regarding these settings.
For RMAN backup sets, image copies, and Data Pump dump files, the "NOAC" mount option should not be specified - that is because RMAN and Data Pump do not check this option and specifying this can adversely affect performance.
We had a breakthrough, which I am embarrassed to admit I did not clue in on earlier...oh well. Anyway...
All the tests we have been running so far have been from one datacenter to another. Meaning Oracle Exadata client servers have been located in DC1 and our Prod Isilon located in DC2. We ran another test late last week this time using our DR Isilon located in DC1 (same DC as the Oracle Exadata client servers). We mounted the NFS share using the smartconnect name and the datapump export worked perfectly.
So it seems that the issue is somewhere in the network path between datacenters. Again, when using IP to mount, works fine every time. Using smartconnect name to mount, it will start to work, then fail.
I am not a networking guy so no idea what is causing the problem but hopefully our network team can figure it out.
Found the issue.
ASA Firewall needed to allow Isilon network inbound on port 2049 on the Oracle Exadata network.
ASA SYSLOG messages
%ASA-6-106015: Deny TCP (no connection) from x.x.x.x/x to x.x.x.x/x flags RST on interface someinterface
The security appliance discarded a TCP packet that has no associated connection in the security appliance connection table.
The security appliance looks for a SYN flag in the packet, which indicates a request to establish a new connection.
If the SYN flag is not set, and there is not an existing connection, the security appliance discards the packet.
Recommended Action: None required unless the security appliance receives a large volume of these invalid TCP packets.
If this is the case, trace the packets to the source and determine the reason these packets were sent.
OMG! We have the EXACT same situation we've ben dealing with for about 6 months since we upgraded to 188.8.131.52 and Oracle 12.1 (and we have an ASA firewall). I can't believe I missed this post. We're running tests now to an IP address vs. FQDN. I will let you know if this is the issue and if the fix to the ASA firewall fixes it. Thank you so very much.