Many enterprises have strict security policies in place to detect, clean (remove), or quarantine viruses. This is often performed at the individual user level with per-system anti-virus (AV) solutions from third-party security vendors. Many of these same enterprises use large, centralized storage platforms to contain user home directories or group-project repositories. Because these are the same file types that reside on end-user workstations, viruses must not reside on the storage systems. Because end-user solutions do not work well for centralized storage depots, a different type of solution is required.

Third-party software is often used to scan the storage array through end-user access or manually scheduled policies from a central anti-virus scan server. Common AntiVirus Agent (CAVA) provides an anti-virus solution for Dell PowerScale storage. CAVA uses third-party anti-virus software to identify and eliminate known viruses before they infect files on the system.

This white paper covers the general considerations, configuration, performance, and sizing of the CAVA solution for PowerScale.

Dell Technologies and the authors of this document welcome your feedback on this document. Contact the Dell Technologies team by email.

Authors: Andrew Chung, David Hartley

Note: For links to other documentation for this topic, see the PowerScale Info Hub.

Figure 1 illustrates a high-level architecture of the CAVA anti-virus solution for PowerScale. Dell Common Event Enabler (CEE) sits between the PowerScale cluster and the anti-virus applications. When clients trigger the scanning workflow, Dell PowerScale OneFS generates a request to the CEE or CAVA agent through the HTTP protocol. Then, the anti-virus application fetches the scanning files from PowerScale through the hidden SMB share called CHECK$. These files are scanned by the anti-virus applications.

Figure 1.   The overall architecture of the CAVA solution for PowerScale

The detailed architecture is explained separately by three scanning workflows from the OneFS perspective:

• On-demand scan (also known as on-access scan): This scan is triggered by the proper SMB operation, such as a read or close operation, and depends on your scan profile. There are two scan profiles in OneFS:

• Standard profile: Captures a close and rename operation from an SMB perspective, and triggers the scan operation on the corresponding file.
• Strict profile: Captures a read, close, and rename operation from an SMB perspective, and triggers the scan operation on the corresponding file.

• Scheduled scan: This scan is triggered by the job engine either manually or by schedule.
• Manual scan: This scan is triggered by the CLI command or the corresponding PAPI call to scan a single file.

Generally, the CAVA solution provides better performance, lower total cost of ownership (TCO), and much less CPU and memory usage from the PowerScale perspective, compared with an anti-virus solution based on ICAP.

Note: ICAP is still supported, which remains unchanged from the previous versions of OneFS.

On-demand scan

Figure 2.   The workflow of the on-demand scan

Figure 2 shows the workflow of an on-demand scan. Depending on which scan profile is selected, the SMB requests with a read or close operation code are captured by the I/O manager in OneFS. It extracts the file path and name to provide to the Avscan filter, which can be configured on a multi-tenant access-zone basis. The Avscan filter will then filter according to the following aspects:

• File extension to include
• File extension to exclude
• File path to exclude

If the file matches all the criteria, an internal process lwavscand sends the HTTP scanning request to the CEE/CAVA agent. Simultaneously, OneFS sets the locks on this file. Then, the anti-virus application tries to fetch this file through the hidden SMB share CHECK$ from the file system on OneFS. CAVA supports only downloading a part of a file for scanning, which helps the performance. After the corresponding content is downloaded to the CAVA server, it runs the scan with the anti-virus engine, and CEE sends the scan results and response back to the process lwavscand. At this stage, some scanning attributes are written to this file and the lock is released. The scanning attributes are:

• Scan time
• Scan Result
• Anti-virus signature timestamp
• Scan current

The previous SMB workflow can then continue if the file is not infected. Otherwise, the file is denied access. If there are errors during the scan and the scan profile is strict, the setting Open on fail determines the next action.

Scheduled scan

Figure 3 shows the workflow of the scheduled scan in OneFS. The scheduled scan is triggered by the job engine. Similar to other jobs in OneFS, the impact and schedule configuration can be set accordingly. It also implements the filter, which is slightly different from the filter in the on-demand scan. The detailed filtering criteria are:

• File extension to include
• File extension to exclude
• File path to exclude
• File path to include (only in the scheduled scan)

Figure 3.   The workflow of the scheduled scan

If the file matches all the criteria, an internal process lwavscand sends the HTTP scanning request to the CEE/CAVA agent, and simultaneously, OneFS sets the locks on this file. The anti-virus application then attempts to fetch this file through the hidden SMB share CHECK$ from the file system on OneFS. CAVA supports only downloading a part of a file for scanning, which helps the performance. After the corresponding content is downloaded to the CAVA server, it runs the scan with the anti-virus engine, and CEE sends the scan results and response back to the process lwavscand. All the scanning attributes are recorded under /ifs/.ifsvar/modules/avscan/isi_avscan.db.

Manual scan

Figure 4 shows the workflow of a manual scan. The manual scan is triggered by the CLI, PAPI, or web interface. Unlike an on-demand scan or scheduled scan, it has no filters. The scanning files are directly assigned by the parameters. The daemon process lwavscand sends the HTTP scanning request to the CEE or CAVA agent, and simultaneously, OneFS sets the locks on this file. The anti-virus application then attempts to fetch this file through the hidden SMB share CHECK$ from the file system on OneFS. CAVA supports only downloading a part of the file for scanning, which helps the performance. After the corresponding content has been downloaded to the CAVA server, it runs the scan with the anti-virus engine, and CEE sends the scan results and responds to the process lwavscand. All the scanning attributes are recorded under /ifs/.ifsvar/modules/avscan/isi_avscan.db.

Figure 4.   The workflow of manual scan

There are two types of connections between CAVA servers and PowerScale nodes:

• SMB connection to fetch files or contents for scanning through a hidden share (CHECK$).
• CEE connections for scan requests, scan responses, and other functions. These connections are HTTP.

For each type of connection, OneFS has a different implementation for load balancing. The following sections explain these two methodologies in detail.

SMB connection load balancing

OneFS uses a dedicated IP pool and multi-tenant access zone to isolate all CAVA SMB connections from other workloads. Within this IP pool, SmartConnect is enabled to ensure that all SMB connections are evenly spread across all the nodes.

CEE connection load balancing

The rules and conditions of how CEE connections are balanced are:

• Maximum connections per CAVA servers = 20
• Number of different CAVA servers to which a cluster node can connect = 4
• The nth cluster node starts from the nth CAVA server

The following example uses an environment with the following:

• 7 PowerScale cluster nodes
• 7 CAVA servers

The calculation is as follows:

The layout for the connections is listed in Table 1.

Table 1.      Connection layouts

A more complicated example has an environment with the following:

• 7 PowerScale cluster nodes
• 5 CAVA servers

The calculation is as follows:

The layout for the connections is listed in Table 2.

Table 2.      Connection layouts

OneFS supports all the anti-virus vendors supported by CEE or CAVA. CEE regularly updates the table of the supported vendors and their versions. For the most up-to-date list, see the Common Event Enabler CAVA Support Matrix on ELab Navigator.

See Table 3 for the supportability of this feature.

Table 3.      Supportability

Figure 5 shows the overall workflow of how CAVA works in OneFS. The anti-virus applications use the SMB protocol to fetch the file or a portion of a file for scanning in a PowerScale cluster. From the anti-virus perspective, the hidden SMB share CHECK$ must be present on the NAS in order to read the data to be scanned. This share allows access to all files on a PowerScale cluster under /ifs. SmartConnect and a dedicated multi-tenant access zone are introduced in this process to ensure that all the connections from the anti-virus application are fully distributed and load-balanced among all the configured nodes in the IP pool. A hidden role, AVVendor, is created by the CAVA anti-virus service to map the EMC CAVA service account into OneFS.

Figure 5 also shows the overall steps to configure CAVA in OneFS:

1. Create CAVA servers in OneFS.
2. Create an IP pool.
3. Create a dedicated multi-tenant access zone for CAVA (AvVendor).
4. Create Active Directory authentication provided in the multi-tenant access zone (AvVendor).
5. Update role (AVVendor).

The following sections introduce these steps in detail.

Figure 5.   Workflow showing how CAVA works in OneFS

The prerequisites to configure CAVA service in OneFS are listed in Table 5.

Table 4.      Prerequisites to configure CAVA service in OneFS

Installing the anti-virus application and CEE is a prerequisite (as shown in Table 5). During the configuration of CEE, a domain user is created. This domain user is the service account for Windows service EMC CAVA and is used to access the hidden SMB share CHECK$ to get the files and content for scanning. In this example, the user is LORG\cavausr.

After the anti-virus servers are installed and configured, create the corresponding entries in the OneFS so that PowerScale is aware of them. To perform this action, use the following command:

# isi antivirus cava servers create --server-name=avsh01 --server-uri=10.7.a.b --enabled=1

Alternately, use the WebUI as shown in Figure 6.

Figure 6.   Add CAVA server

You can add multiple CAVA servers into a OneFS configuration for the proper number of the CAVA servers for a given PowerScale cluster. (For details, see the Sizing section.)

Note: Before performing the following steps, ensure that the Service Enabled is set to No in the CAVA settings.

The purpose of creating an IP pool is to facilitate the connections from anti-virus applications. A dedicated IP pool must be used for CAVA antivirus. This pool of IPs needs to be accessible by the antivirus servers. Using high performance front end interfaces is recommended.

Note: Do not mix the IP range in this dedicated IP pool with others for a regular SMB client connection.

The load balancing for the anti-virus workload is achieved by the SmartConnect zone in this IP pool. Because this is a dedicated IP pool for CAVA servers, all the anti-virus scanning workloads should be evenly distributed within the pool. To perform this action, use the following CLI command:

# isi network pools create groupnet0.subnet0.pool1 --ranges=10.7.u.v-10.7.x.y --sc-dns-zone "cavacluster1.west.isilon.com" --ifaces=1:ext-1

In this example:

• The IP pool is groupnet0.subnet0.pool1
• The IP range is 10.7.u.v to 10.7.x.y
• The SmartConnect Zone name is cavacluster1.west.isilon.com. Make sure that the DNS delegation is created for it.
• The network interface is the ext-1 from node 1

After the IP pool is created, associate it with the CAVA by using the following command. After this command, this IP pool is only available to the CAVA servers, and simultaneously, the corresponding multi-tenant access zone of this IP pool is changed to AvVendor.

# isi antivirus cava settings modify --ip-pool="groupnet0.subnet0.pool1"

This action will make the IP Pool unavailable to all other users except antivirus servers. Do you want to continue? (yes/[no]): yes

"

IP Pool groupnet0.subnet0.pool1 added to CAVA antivirus.

Note: The access zone of IP Pool groupnet0.subnet0.pool1 has been changed to AvVendor.

This function is also available through the WebUI, as shown in Figure 7.

Figure 7.   Associate IP pool with CAVA settings

Note: Be sure to create the DNS delegation for the zone name associated with the IP pool created in this step.

When the antivirus service type is changed to CAVA, a multi-tenant access zone is automatically created. This multi-tenant access zone will be automatically configured to use the IP address pool created in the section Create an IP Pool. To enable the CAVA service, use the following command:

# isi antivirus cava settings modify --service-enabled=1

View the CAVA settings and ensure that the Server Enable setting is Yes.

# isi antivirus cava settings view

       Service Enabled: Yes

     Scan Access Zones: System

               IP Pool: groupnet0.subnet0.pool1

         Report Expiry: 8 weeks, 4 days

          Scan Timeout: 1 minute

Cloudpool Scan Timeout: 1 minute

     Maximum Scan Size: 0.00kB

At the current stage, AvVendor is created in the multi-tenant access zone list:

# isi zone zones list

Name     Path

--------------

System   /ifs

AvVendor /ifs

--------------

Total: 2

Note: The ID of the multi-tenant access zone AvVendor is -2 but can display in the CLI as 4294967294 (binary 2’s complement).

All the anti-virus application servers and the PowerScale cluster should be in the same domain. Use the following CLI command to join the PowerScale cluster into the domain. In this example, the domain name is lorg.west.isilon.com.

# isi auth ads create lorg.west.isilon.com --user administrator

Then, add the authentication provider to the multi-tenant access zone AvVendor:

# isi zone zones modify AvVendor --add-auth-providers=lsa-activedirectory-provider:lorg.west.isilon.com

The CHECK$ share is a hidden share that allows access to all files on the cluster. It is used exclusively by the anti-virus software running on a Microsoft Windows server. Because the CHECK$ share allows access to all files on the cluster, any user accessing the share must have a unique privilege. The hidden ISI_PRIV_AV_VENDOR privilege in the AVVendor role is added to give the user account that is used by the anti-virus software access to the CHECK$ share.

Use the following CLI command to assign the user LORG\cavausr to the role AVVendor in the multi-tenant access zone AvVendor:

# isi auth roles modify AVVendor --zone=AvVendor --add-user lorg\\cavausr

At the current stage, the configuration for the CAVA service on the PowerScale is finished. Use the following command to ensure that the System Status is RUNNING:

vshen-p21evix-1# isi antivirus cava status

        System Status: RUNNING

        Fault Message: -

          CEE Version: 8.7.7.0

          DTD Version: 2.3.0

            AV Vendor: Symantec

In general, CAVA has better performance than ICAP. The following sections compare the performance between CAVA and ICAP in a PowerScale cluster.

The performance results may vary depending on the PowerScale model, total number of nodes, anti-virus application vendors, size of the CAVA or ICAP servers, workload, file size distribution, and other factors. The following example only gives an overall idea of what the CAVA performance looks like and how it compares with ICAP.

The following sections cover the following:

•  Test environment
•  Scenario 1: Performance of scan on close
•  Scenario 2: Performance of scan on read
•  Scenario 3: Performance impact to metadata read/write

The following table shows the components in the test environment for Scenario 1 and Scenario 2:

Table 5.      Test environment for Scenarios 1 and 2

The following table shows the components in the test environment for Scenario 3:

Table 6.      Test Environment for Scenario 3

Test methodology

In this scenario, the client servers generate different sizes of files which are 10 KB, 100 KB, 1 MB, 10 MB, 100 MB, and 200 MB. From a PowerScale perspective, a standard scan profile is applied to ensure that the scan occurs when the file is closed. Regarding the detailed CAVA configuration on PowerScale, see the following profile:

vshen-p21evix-1# isi antivirus cava filters view System

                 Zone: System

              Enabled: Yes

         Open-on-fail: Yes

      File Extensions: *

File Extension Action: include

 Scan If No Extension: No

        Exclude Paths: -

         Scan-profile: standard

         Scan-on-read: No

        Scan-on-close: Yes

       Scan-on-rename: Yes

 Scan Cloudpool Files: No

Test results

Figure 8 shows how the number of files scanned per second changes along with the increase of the average file size. With the increased size of the files, the overall trend of the files scanned per second is decreased.

Figure 8.   File scanned per second (scan-on-close)

Figure 9 shows the relationship between average file size and the scanning throughput. Theoretically, the results equal the files scanned per second times the average file size in each category. With the increase of file size, the scanning throughput increases. In this case, after it reaches 2 GB/sec, it goes flat.

Figure 9.   Scan throughput (scan-on-close)

The average CPU utilization on the PowerScale cluster is stable under 8% no matter the overall scan throughput or various average file sizes. For more details, see Figure 10.

Figure 10.    Average CPU utilization on PowerScale (scan-on-close)

Test methodology

In this scenario, the client servers generate read requests against the files which have been generated in Scenario 1. The CAVA strict profile is applied to ensure that scan-on-read is enabled. Regarding the detailed CAVA configuration on PowerScale, see the following profile:

vshen-p21evix-1# isi antivirus cava filters view System

                 Zone: System

              Enabled: Yes

         Open-on-fail: Yes

      File Extensions: *

File Extension Action: include

 Scan If No Extension: No

        Exclude Paths: -

         Scan-profile: strict

         Scan-on-read: Yes

        Scan-on-close: Yes

       Scan-on-rename: Yes

 Scan Cloudpool Files: No

Test results

Figure 11 illustrates how the number of files scanned per second changes along with the increase of the average file size. With the increased size of the files, the overall trend of the files scanned per second is decreased.

Figure 11.    Files scanned per second (scan-on-read)

Figure 12 shows the relationship between the average file size and scanning throughput. Theoretically, the results equal the files scanned per second times the average file size in each category. With the increase of file size, the scanning throughput increases. In this case, after it reaches 300 MB/sec, it goes flat. Compared with the results in Scenario 1, it is obvious that the scan efficiency is much lower with scan on read. Scan-on-read can provide better protection against viruses, however, the scanning efficiency is lower than scan-on-close. We recommend carefully selecting the scan profile to match your business requirements and performance level.

Figure 12.    Scan throughput (scan-on-read)

The average CPU utilization on a PowerScale cluster is stable around 6% no matter the overall scan throughput or various average file size. For more details, see Figure 13.

Figure 13.    Average CPU utilization on PowerScale (scan-on-read)

Test methodology

In this scenario, we included the following subscenarios to better understand the performance impact from the CAVA workload:

• Baseline: There is no CAVA workload. The only workload running on the cluster is metadata intensive read and write.
• Path excluded: The path for the workload has been excluded from the CAVA configuration.
• Standard profile: Scan-on-close, scan-on-rename.
• Strict profile: Scan-on-close, scan-on-rename, scan-on-open.

Note: For comparison, all four subscenarios use the same requested operation rate that is configured in the workload benchmark tool.

Test results

Figure 14 shows the results of the achieved operations per second in the four subscenarios defined in the previous sections. The standard scanning profile, or if we exclude the workload path from the CAVA configuration, has almost no performance impact compared with the baseline result. The standard profile implements scan-on-close, which means that any file newly written or modified is submitted for virus scanning in the background after it is closed by the client. Scan-on-close is not synchronous in the host I/O path, and it slows down the incoming workload. In this case, the incoming workload proceeds unabated.

For the strict profile, the performance drops by 34%. This is because the strict profile adds scan-on-read. Any file not previously scanned, when opened for read by a client, is submitted for scanning at that time and it is synchronously in the host I/O path.

Figure 14.    Achieved operations per second

Figure 15 shows the average latency in the four subscenarios defined in the previous sections. The standard scanning profile, or if we exclude the workload path from the CAVA configuration, has almost the same latency compared with the baseline. In the strict profile, the latency increases by 52%.

Figure 15.    Average latency

In this section, two sizing methodologies are introduced for sizing the number of CAVA servers with a given PowerScale cluster. Both methodologies can only result in an estimated number of the CAVA servers. For more granular sizing that is based on various workloads, see the CEE CAVA calculator described in Using the Common Event Enabler on Windows Platforms.

No matter which methodology to use, we recommend using the following general best practices for sizing:

• Use Windows machines with at least 16 GB memory and two-core processors.
• Use a minimum of two CEE or CAVA machines for redundancy.

Methodology 1 only requires having the cluster node number as the known condition. The number of CAVA servers follows the following algorithm.

For example, in a 22 node PowerScale cluster, The number of CAVA servers is:

In this case, we need six CAVA servers.

Methodology 1 only provides an estimated calculation that is based on the total number of nodes as the known condition. This methodology suits both the on-demand scan and scheduled scan. For some scenarios where customers do not have details for the I/O pattern of their workflow, this methodology can be used for sizing.

Methodology 2 is much more granular than Methodology 1 because it accounts for the various workloads. This methodology only applies to the on-demand scan due to the sizing and because expected performance can vary based on different job performance impact levels.

The assumption for this methodology is as follows:

Typical antivirus scan roundtrip time (RTT) for each CAVA connection = 40ms

The known condition for this methodology is as follows:

Define the total number of the requests per second which will trigger the on demand scan as R

Maximum connections per CAVA servers=20

The algorithm is as follows:

In the following example, the scan profile is standard, the SMB close requests are 1,000 operations per second, and the SMB rename requests are 200 operations per second. The number of CAVA servers required is as follows:

Minimum number of CAVA servers required=(1000+200)/500=1200/500=2.4≈3

Not all nodes on network (NANON) is supported in CAVA configurations. More details are provided as follows:

• Scheduled scan: This works by default. In OneFS 9.1.0.0, an improved job engine has been implemented to detect a list of nodes not having connection to CAVA servers, and then inform the job engine not to distribute scan jobs to these nodes. This means that only the PowerScale nodes with the front-end connection to the CAVA servers will be triggered to run scheduled scan.
• On-demand scan or manual scan: If the PowerScale node that triggers the on-demand scan or manual scan has the front-end connection to the CAVA servers, it works. Otherwise, a CELOG event - SW_AVSCAN_CAVA_SERVER_OFFLINE is raised for notification.

The following is an example in which two IP pools are isolated.

Table 7.      IP pools configuration

In this example, the following behavior is expected:

• If the on-demand scan or manual scan is triggered from Node 1 to Node 3, it works.
• If the on-demand scan or manual scan is triggered from Node 4, it fails with CELOG event SW_AVSCAN_CAVA_SERVER_OFFLINE.
• The scheduled scan works properly.

The CAVA configuration in OneFS supports scanning files in CloudPools. By default, scanning of CloudPools files is disabled to prevent the unexpected cost of file callback. To enable this setting, go to the scan zones settings in the CAVA configuration tab. Figure 16 shows the details.

Figure 16.    Enable scanning CloudPools files

OneFS has a separate, configurable scan timeout for CloudPools files, because scanning a CloudPools stub file may take more time than a regular on cluster file. To configure the timeout value for scanning CloudPools files, go to the settings section under the CAVA configuration tab. See Figure 17 for details.

Figure 17.    Configure timeout value for scanning CloudPools files

When the stub file is read in the kernel level, CloudPools fetches the file content and stores the data in the BCM cache. Because the anti-virus application is the first SMB client to read from the file, it triggers the file-block fetches.

For the scheduled scan, OneFS bypasses the BCM cache because it would continuously overflow the BCM cache, making it worthless. For the on-demand scan or manual scan, the BCM cache would not overflow as often, and the subsequent SMB client read would result in a cache hit.

Transferring anti-virus file attributes using SyncIQ is not supported. Files without those attributes are scanned again. This means that a file that has been scanned on a source cluster and is then replicated to a secondary cluster will have the same file scanned again even with no changes in the anti-virus definition.

OneFS supports all anti-virus vendors supported by CEE or CAVA, but some anti-virus applications may behave differently than others.

The details of the infected files are maintained at the anti-virus-application level. We recommend checking log files when the infection is detected. For an infected file, OneFS denies all access to it, but the detailed action is taken by the anti-virus application. Consult your anti-virus vendors or documentation to understand how they deal with the infected files.

The following sections explain several common error scenarios and general troubleshooting methods.

CEE or DTD Version shows as 0.0.0.0

This situation often means that the cluster cannot contact any CEE server at all. This could be due to a firewall blocking the TCP connection on port 12228 or it could be that the CAVA service is not running.

Figure 18.    All CAVA servers offline

CAVA listens on TCP port 12228 for incoming requests from clients. Use the following methods to check the connectivity.

Use Telnet

From a cluster node that has an IP address in the network pool assigned previously, run the following telnet command to try to connect to the CAVA server:

telnet <CAVA_IP> 12228

Figure 19.    Use telnet to test connectivity

Enter any character such as the ‘.’ (period) and press enter. If there is no response, the port may be blocked by a firewall, or the CAVA service may not be running.

A listening CAVA server should respond in a manner similar to the following:

Figure 20.    Normal response from CAVA server

Use Python

If telnet is not available and Python is available, the following short script can also test for an open port:

python -c "x='<CAVA_IP>'; from socket import *; import sys; s=socket(AF_INET,SOCK_STREAM); s.settimeout(2); s.connect((x,12228)); sys.stdout.write('CAVA port is open\n')"

An open port will print “CAVA port is open”:

Figure 21.    Output for an open port

A closed port will return a socket.timeout error:

Figure 22.    Output for a closed port

If port 12228 is unreachable, check firewall settings in the network between the cluster and the CAVA server. Use the traceroute command to see all the hops in the network path to the CAVA server.

CEE and DTD Version shows a correct value

In this case, the cluster can contact a CEE server and can receive the CEE and DTD versions from the CAVA server.

Figure 23.    DTD and CEE versions show the correct versions

This validates that PowerScale can send heartbeat messages to CAVA and get responses back. This means that there are no firewall ports in the path and that the CAVA service is at least installed and responding to port 12228.

There are several issues that could be potential issues in this situation.

1. CAVA service is using the Local System account.

2. CAVA service user account does not have local administrator rights to the scan server.
3. Service user account does not have the proper permissions on the PowerScale cluster.

CAVA service is using the Local System account

Double check that the service is using a user account and that the password is correct.

CAVA service user account does not have local administrator rights to the scan server

On the Windows server, verify that the user is in the local administrator group. On a Domain Controller, the user normally needs to be in the Domain Administrators group.

You can detect this case by looking into the Event viewer and seeing an entry similar to the following:

Figure 24.    Missing local administrator rights

CAVA service user account does not have the proper permissions on the PowerScale cluster

Use the following CLI command to verify that the Service user account is in the AvVendor role:

# isi auth roles members list AvVendor –zone=AvVendor

If your test to scan a file does not trigger any actual scan, the first area to investigate is whether the anti-virus software is configured properly.

For example, see if the Real-time protection setting for Windows Defender is turned off. There may be some group policies that may disable the on access scan even after enabling the feature. Double check to make sure that Real-time protection is actually enabled.

To debug CAVA directly:

1. Install DebugView from Sysinternals.

2. CAVA needs to be configured to output debug messages. This requires adding some keys into the registry. Start by opening up the Registry Editor and navigating to: HKEY_LOCAL_MACHINE\SOFTWARE\EMC\CEE\Configuration. Modify the Debug and Verbose keys and set them to 0x3F.

Figure 25.    Enable debug and verbose log for the CAVA service

3. Use the services.msc to restart both the CAVA service and the CEE Monitor service.
4. DebugView will allow you to view debug messages coming from CAVA itself. DebugView does not need to be installed, it runs as a standalone executable. The program should be run with Administrator rights.
5. Run the executable for DebugView that matches your OS. For most cases, you will want to use dbgview64.exe.
6. Under the Capture menu item, make sure that the following items are checked:
   1. Capture Win32
   2. Capture Global Win32
   3. Capture Events

Figure 26.    DebugView

7. You should now see the events showing up in the capture window.

Figure 27.    Captures in DebugView

A detailed explanation of how to read CEE debug logs is out of scope for this document. However, just retrying an operation that is failing, such as a scan request, and looking through the messages for any obvious errors can be very helpful in pinpointing where the AV scan process is failing.

Here are some tips for looking through the CAVA logs.

1. Save the log to a text file to be able to look at the full log messages using File -> Save As
2. You should see CheckHeartBeat return back NORMAL results
3. Actual scan requests have a tag that starts with <Args action= "7"
4. You can look at the CheckFileRequest message and decode the name fields. The fields are Base64 encoded. Simply copy the text string and use a Base64 decoder (https://www.base64decode.org/) and interpret the output as UTF-16LE. For example:

XABcAGMAYQB2AGEALgBwAHMAMQAuAGQAZQBtAG8ALgBsAG8AYwBhAGwAXABDAEgARQBDAEsAJABcAFQAaABpAHMASQBzAEEAUwBhAG0AcABsAGUARgBpAGwAZQAuAGUAeABlAA==

Decodes to:

\\cava.ps1.demo.local\CHECK$\ThisIsASampleFile.exe

To turn off debug, change the Debug and Verbose values in the registry back to 0 and restart the services.

To install CEE:

1. In the Drivers & Downloads page of Common Event Enabler, select the most recent CEE version for Windows and click the download button:

Figure 28.    Download the Windows version of CEE

2. After downloading, mount the DVD image by double-clicking the file – EMC_CEE_Pack_8_7_1.iso and it will automount as a DVD.
3. Install the 64 bit or 32 bit version of CEE as required.
4. Restart the server to complete the installation process.
5. After restarting the server, open Event Viewer and navigate to Windows Logs -> Application. Verify that CEE is running by looking for the following four events:

Table 8.      Events

When CAVA performs a scan, it does so under the credentials of the service account of the CAVA service. By default, the service account is the Local System account. This account does not exist on the PowerScale cluster. Use the following steps to change the service account to either a user in the Domain or a local user on the PowerScale cluster being scanned.

1. In the Windows machine on which CEE has been installed, run the following command:

services.msc

2. Find the service - EMC CAVA, open the properties dialog, select the Log On tab at the top, and modify the service to use a specific account. This account that is specified is required to be a member of the Local Administrators group. This is required to allow CAVA to scan the system process list to find the AV engine process.

Figure 29.    Change the service account

Note: It is not recommended to use a domain controller as a CAVA scan server. To gain the ability to scan the process list, you need to add a user to the Domain Admins group which gives them much higher privileges than required to perform an AV scan.

In order for CAVA to perform an on demand scan, the Real-time protection option for Windows Defender must be enabled. You can reach this setting by selecting Windows Settings and then the Windows Defender configuration page:

Figure 30.    Enable real-time protection for Windows Defender

To configure CAVA in OneFS, see the section Deployment and configuration.

Testing to see if AV scanning is working can be done very simply by using the industry approved EICAR test file. This file is a simple text string that is a valid Windows executable and that all AV vendors will recognize as a virus for test purposes.

1. Download the EICAR file and put it under /ifs.

2. Initiate a manual scan of the file:

Figure 31.    Manual scan

Open Event Viewer in Windows, then navigate to Applications and Services Logs, Microsoft, Windows, Windows Defender, and Operational. The following warning message indicates that the CAVA anti-virus software has detected this EICAR testing virus.

Figure 32.    Malware is detected

Note: The path in the Event Viewer should always start with file:\\<fqdn_of_cluster>\CHECK$\.