I have a customer who is seeing very poor NFS read and write performance when SCAN ON OPEN and SCAN ON CLOSE are turned ON. What is the implication if both of these settings are turned OFF? What does DELL EMC recommend? Also, the customer has 1 AV Server per node which are Virtual, and are running McAfee AV Software. Should the AV servers be Physical instead of Virtual?
You asked a number of questions here:
1. Physical or Virtual shouldn't matter, as long as the virtual resources aren't drastically oversubscribed, then virtual is fine for ICAP. The number you have matches the best practices of 1 ICAP server per node.
2. Terrible performance with scan on read or scan on write. That's a given. With ICAP you're sending the full file to the ICAP server (not just a signature of that file), letting it scan that file, and then if it's all OK, tell the cluster it's OK to send it to the user for read, or send it to disk on write. So this should be expected.
3. What's recommended? Well treat the Isilon cluster like any other storage device. Would you scan LUNs on a VMAX or XtremIO with every read, or every write? No, never. Why? Because it creates a huge performance bottleneck. Now the hosts mounting those LUNs probably have AV software installed on them, but that is different. What you could and should do instead is setup scheduled scans, perhaps weekly, and like in a block world, setup the AV software on your clients to scan these files. The OneFS filesystem is mounted with the no exec option, so malicious code on it couldn't run even if it wanted to.
isilon-1# mount | grep /ifs
OneFS on /ifs (efs, local, noatime, noexec)
This is why you can write a shell script, put it in /ifs/scriptname.sh, chmod +x it, and still not execute it.
Agree with what Chris says here. On-access scanning is best done on the clients as they will always have more collective horsepower than the server. If you must do on-access scanning, consider only doing on-close as it hurts performance less than on-open. But, ideally, on-access scanning should be handled by clients and occasional full scan can be handled by the cluster.