PowerScale: 'ls -l' and reads on NFS file systems are slow on the first attempt only

On NFS clients mounting NFS exports on PowerScale OneFS, 'ls -l' or reading of files within the NFS-mounted File System are slow on the first attempt, or after being inactive for some time, and then all subsequent attempts are fast.

For example, when running a 'dd' read test on a 30GB file:

First attempt: 

[administrator@docker1 ~]$ time dd if=/mnt/nfs-share/test8.file of=/dev/null bs=1024k
30720+0 records in
30720+0 records out
32212254720 bytes (32 GB) copied, 59.8951 s, 538 MB/s

real    0m59.900s
user    0m0.024s
sys     0m16.669s

However, all subsequent tests: 

[administrator@docker1 ~]$ time dd if=/mnt/nfs-share/test8.file of=/dev/null bs=1024k
30720+0 records in
30720+0 records out
32212254720 bytes (32 GB) copied, 6.07811 s, 5.3 GB/s

real    0m6.082s
user    0m0.019s
sys     0m5.950s

This is normal to see on most Linux distros. Linux clients have an integrated caching mechanism for reads. It is client-side caching. Some minimal caching is also done on the Isilon end as well. However, this does not explain why that first "true" read, without any caching, is slow.  It is preferable to disable the read cache in order to accurately troubleshoot the slowness, for example, when generating pcaps.

To disable caching for pcaps, or testing, you can run the following command on your NFS client machine: 

echo 3 > /proc/sys/vm/drop_caches

Or:

Add the following mount option to your fstab file:
actimeo=0 --> performs real time read.