PowerScale: NFS Performance Issues due to ICAP server overload
Summary: NFS Performance is impacted while Antivirus Scan settings "Scan on open" or "Scan on close" is enabled, causing ICAP servers to overload.
Symptoms
The performance impact on NFS described in this KB apply to a cluster configured with ICAP servers. First, check the ICAP Antivirus server settings:
# isi antivirus icap settings view Fail Open: Yes Glob Filters: - Glob Filters Enabled: No Glob Filters Include: No Path Prefixes: /ifs/nasprod2024/smb Repair: No Report Expiry: 1W Scan On Close: Yes <<<-- Enabled Scan On Open: Yes <<<-- Enabled Scan Cloudpool Files: No Scan Size Maximum: 100.00M Service: Yes Quarantine: Yes Truncate: No
Confirm nfs threads waiting on "av" or "av_wait" in the general performance script.
https://confluence.cec.lab.emc.com/display/ISILON/General+Performance+Troubleshooting
>>> Wait Channel Values<<< 11 av_wait 20 kqread 20 av 4 sbwait 1 opwres 4 sigwait 107 ucond
Confirm in log messages on the affected node in the time frame impacted.
- ICAP Servers overload messages in /var/log/messages
2025-05-06T17:42:20.639078-07:00 <0.5> nasprod2024-4(id4) /boot/kernel.amd64/kernel: [bam_av.c:1176](pid 14848="nfs")(tid=100543) Scan 1:155d:543d::HEAD failed with timeout (probably due to ICAP server overload); doing opdata cleanup with state: dequeued 2025-05-06T17:42:20.639138-07:00 <0.5> nasprod2024-4(id4) /boot/kernel.amd64/kernel: [bam_av.c:1176](pid 14848="nfs")(tid=100524) Scan 1:0c2e:9531::HEAD failed with timeout (probably due to ICAP server overload); doing opdata cleanup with state: dequeued
- Avscan coring and restarting n /var/log/messages
2025-05-06T17:00:03.571382-07:00 <0.5> nasprod2024-1(id1) /boot/kernel.amd64/kernel: /lib/libthr.so.3:_pthread_create+0x906 2025-05-06T17:00:03.571392-07:00 <0.5> nasprod2024-1(id1) /boot/kernel.amd64/kernel: -------------------------------------------------- 2025-05-06T17:00:03.571403-07:00 <0.6> nasprod2024-1(id1) /boot/kernel.amd64/kernel: pid 68624 (isi_avscan_d), jid 0, uid 0: exited on signal 11 from pid 0 (unknown) (core dumped) 2025-05-06T17:00:03.905443-07:00 <3.3> nasprod2024-1(id1) isi_avscan_d[63725]: [0x4d07b71b3000] Setting process flag P_NO_SEMLOCK for avscan 2025-05-06T17:00:03.905857-07:00 <3.4> nasprod2024-1(id1) isi_avscan_d[63725]: [0x4d07b71b3000] isi_avscan_d starting up 2025-05-06T17:00:03.949657-07:00 <3.4> nasprod2024-1(id1) isi_avscan_d[63725]: [0x4d07b71b3000] isi_avscan_d completed start up, ready...
- Avscan logs show signs of an overloaded ICAP server
2025-05-06T17:38:58.867768-07:00 <3.6> nasprod2024-1(id1) isi_avscan_d[63725]: [0x4d07b7998200] Scan request timeout exceeded and dropped for 1:17c5:0b6e::HEAD, policy id: SCAN_ON_OPEN, report id: SCAN_ON_OPEN. 2025-05-06T17:38:58.868241-07:00 <3.6> nasprod2024-1(id1) isi_avscan_d[63725]: [0x4d07b71b4900] Scan request timeout exceeded and dropped for 1:17b7:6f7d::HEAD, policy id: SCAN_ON_OPEN, report id: SCAN_ON_OPEN. 2025-05-06T17:38:58.868439-07:00 <3.6> nasprod2024-1(id1) isi_avscan_d[63725]: [0x4d07b7997300] Scan request timeout exceeded and dropped for 1:0e78:89b3::HEAD, policy id: SCAN_ON_OPEN, report id: SCAN_ON_OPEN.
Confirm the path scanned using isi_get. For example, the above LIN 1:0c2e:9531 from messages log should match an NFS export path.
A valid path for LIN 1:0c2e:9531::HEAD is /ifs/nasprod2024/nfs/libarc-p1/mysql/#ib_16384_0.dblwr
Cause
Customer's ICAP servers are overloaded from scanning files on open and close. The ICAP guide recommends a 1:1 ratio at least.
PowerScale OneFS CLI Administration Guide-ICAP Servers
Resolution
For immediate relief, disable "scan on open" and then test performance.
# isi antivirus icap settings modify --scan-on-open=false
If the issue persists, disable "scan on close" and then test performance.
# isi antivirus icap settings modify --scan-on-close=false
For permanent resolution, and if the desire is to enable both settings, then increase the number of ICAP servers.
Scan on close is the recommended 1:1 ICAP servers to nodes ratio.
Scan on open is the recommended 2:1 ICAP servers to nodes ratio.