Celerra CAVA errors

Karl,

I have upgrade Mcafee to 8.5 patch8. Now, the server_viruschk ALL output shows the AV servers going offline sometimes and coming back online.

  I changed the settings enabled the when reading from disk option also in the Mcafee properties. What else you think? Getting lot of high water errors too!

Really need your help

Thanks,

Bannu.

The high water marks means that lots of files are waiting to be scanned - one or more of the CAVA servers are going offline and the queue of files to scan is getting large.  This is why I recommended you upgrade only one CAVA server to the latest patch and see how it behaves.

Log into the CAVA servers via Remote Desktop and open the On-Access Scan Stastistics in McAfee.  Are the CAVA servers actually scanning all time?  Do they pause?  Do you see the McAfee On-Access scanning service pausing and restarting (you will see the McAfee shield icon get a red circle for a time, then go away).  If you see these things, most likely McAfee is starting and restarting all the time.  This was why we upgraded from 8.5i patch 3 to 8.7i, as I indicated earlier.

One of the behaviors of McAfee is that when a file scan exceeds the time limit you set for Maximum Scan Time (45 seconds by default), McAfee aborts the scan by stopping/restarting the service.  This is long enough to convince CAVA that the AV client is not running.  At this point, you should probably open a ticket with EMC support and engage McAfee - they may be able to assist further.

I can't remember if 8.5i had a setting for it, but you could try and uncheck "Prevent McAfee services from being stopped".  This shortens the cycle for McAfee to restart, but it doesn't remove the problem entirely.

Good luck!

Karl

Have you tried using one physical server and see if problem goes away?

Claude

Here's the paragraph from the guide "Using CAVA for Celerra"

You can enable the CAVA scan-on-first-read functionality using the server_viruschk command. The command sets the reference time on the virus-checker configuration file. The Data Mover uses the access time of a file during an open to see if the file must be scanned. This time is compared with the time reference that is in the virus checker configuration on the Data Mover. If the access time of the file is less than this reference, the file is scanned before it is opened by the CIFS client. The time reference is updated with a field of the response of the virus checker only if the time given in this field is greater than the time reference. CAVA sets the access time when it detects a virus definition file update. The now = sets reference time to the current time. The none = option disables the time scan (scan-on-first-read) functionality. The reference time is stored in memory and in the viruschecker.dat file located in the /.etc directory. The time is persistent after a stop or start of the virus-checker service or after restarting the Data Mover.

This means that any file newer than the Scan on read time (Fri Mar 19 15:55:46 2010) will be scanned when it is  read by any CIFS user.  So, any file with a timestamp of Fri Mar 19 15:56 or later will be scanned.  If your users are creating lots of new files, then yes - each one of those new files is being scanned on read.

If you want to turn off this feature, you can use server_viruschk server_2 -set accesstime=none.  This should clear that flag and stop the CAVA service from checking files on reads.

Another observation - I see that your viruschecker.conf is set to scan every filetype except PAGEFILE.SYS and *.TMP.  I won't argue for or against that approach, but I will say that is probably means you will lots of CAVA servers to keep up with the demand, if you have lots of users and lots of open files.

We exclude very few file types as well, but we also run many, many CAVA servers to offset that.  Also, we spent time getting CAVA to work on VMs, so that we could quickly spin up more CAVA servers, if we had too much load.  Of course, this can be very costly, with licensing fees for VMs, more antivirus client licenses, etc. etc.

Thanks!

Karl

Hi Karl,

EMC support recommended me to disable the scan on first read on all the DMs, because it will impact the CAVA environment and which is not recommended setting, which I disabled on all DMs. No CAVA errors now.

I tried testing the EICAR file on putting one of those DMs (created a CIFS share). When I chose the delete files automatically on the AV server, I can see that EICAR file was deleting automatically. But, when I chose the option deny access to files from the drop down, it should rename the EICAR file to .vir extension and which is not happening(file was sitting there). Any ideas or comments.

Also found this output from one of the DM.

[nasadmin@ccs1 ~]$ server_log server_2 |grep -i VC
2010-03-22 10:39:04: VC: 5: last message repeated 1 times
2010-03-22 10:39:08: VC: 5: last message repeated 1 times
2010-03-22 10:40:10: VC: 5: last message repeated 1 times
2010-03-22 10:40:21: VC: 5: last message repeated 1 times
2010-03-22 10:41:07: VC: 5: last message repeated 1 times
2010-03-22 10:41:21: VC: 5: last message repeated 1 times
2010-03-22 10:41:39: VC: 5: last message repeated 1 times
2010-03-22 10:41:49: VC: 5: last message repeated 1 times
2010-03-22 10:42:24: VC: 5: last message repeated 1 times
2010-03-22 10:42:32: VC: 5: last message repeated 1 times
2010-03-22 10:42:34: VC: 5: last message repeated 1 times
2010-03-22 10:42:39: VC: 5: last message repeated 1 times
2010-03-22 10:42:53: VC: 5: last message repeated 1 times
2010-03-22 10:42:56: VC: 5: last message repeated 1 times
[nasadmin@ccs1 ~]$

Thanks,

Bannu.

Hi Bannu -

I'm completely guessing, but I think your error message is the inability of McAfee to rename the file to .vir and make the file unavailable.  Since CAVA seems to be able to perform the delete operation sucecssfully, I don't know why the rename/deny doesn't work.  I'd suggest you open a ticket with EMC about that, if you haven't done so.  It's possible the file is still locked/open by the scan and can't be closed to be renamed.  Sketchy, I admit, but I have no other good guesses.

Thanks!

Karl

Hello Karl

I have the same issue.

Actually I have 5 CAVA-Server and following exclusions in the viruschecker.conf (excl=*.pst:*.ost:*.edb:*.zip:*.TAR:*.ARJ:*.ARC:*.Z:????????).

command: server_stats server_2 -summary  cifs -i 5

shows following amount of connections/open files

server_2    CIFS     CIFS     CIFS     CIFS Avg    CIFS     CIFS     CIFS Avg      CIFS       CIFS
Timestamp   Total    Read     Read    Read Size    Write    Write   Write Size  connections   open
            Ops/s    Ops/s    KiB/s     Bytes      Ops/s    KiB/s     Bytes                   files
11:44:43     21602      652     3988        6262      172     3445       20455        11470    16229
11:44:48     21243      561     3050        5565      119     2184       18746        11466    16244
11:44:53     22420     1282     4423        3534      219     3138       14644        11467    16316

How many CAVA-Server do you have implemented?

Do you mean the same issue with McAfee (if that's your AV client)?  With the messages in the server_log?  Performance issues??

In my environment, I have roughly 8000 home-dir users accessing an NS80.  I'm using seven CAVA servers, running McAfee 8.7.i patch 2.  We scan most filetypes, but can still keep our average scan times around 30ms.

1 File Mask(s):
*.*
6 Excluded File(s):
*.TMP ???????? *JOURNAL *.DOTM *.LNK *SQLITE

[nasadmin@NS80-CS0 ~]$ server_viruschk server_2  -audit
server_2 :
Total Requests: 2604242494.
Requests in progress: 0.

NO ANSWER from the Virus Checker Servers: 0.
ERROR_SETUP: 0.
FAIL: 0.
TIMEOUT: 0.
Total Infected Files: 17.
Deleted Infected Files: 17.
Renamed Infected Files: 0.
Modified Infected Files: 0.
Detected Infected Files: 0.
min=518 uS, max=300061690 uS, average=28700 uS

0 files in the collector queue.
0 files processed by the AV threads.

If you haven't done so, I suggest trying the CAVA calculator, included with the CAVA installation, to guestimate the number of CAVA servers you need.  Also, you might want to enable the CAVA sizing tool and watching your CAVA servers in realtime.  I used to run four CAVA servers for this load, but I determined that whenever one CAVA server was down for updating its DAT files, three servers would bog down almost immediately.  Four of the CAVA servers are physical, and three are VMs.  I could easily add four more VMs in minutes, if it seemed like the load was too much.

Are VMs an option for you?

Hello Karl.

Thank you for the fast reply.

We are using Sophos 7.6.16 as AV-Client and 2 of 5 CAVA-Servers are VM.

The problem with the CAVA-Tool is to estimate what kind of user type you should take (normal, power, casual) cause this is incapable of measurement.

So I will take the amount of open connections +30%.

Thank you

Michael

Michael,

there are actually two tools - the first one does a "guesstimate" based on the number and type of users.

The second one actually monitors your running CAVA servers and gives you indications if adding more servers would be benefitial

Rainer

Hi Karl/Rainer,

I am also getting same errors on my Celerra NS80 array.

We are Using two AV servers(VMs) whih Mecfee 8.8 version and CAVA 4.9.3 Version. This setup has been configured recently.Erlier we had two Physical Machins with SAV with CAVA 4.2.2.

After changed the setup we are facing lot of perfomance issues.

Today i have opend  the viruschk.conf file which my team has created, the INTERESTING thing is i did not find our new AV servers in the .conf file. still i could able to see old physical servers names.

and one more intersting thing is when i use the server_viruschk server_2 command i can able to see new AV servers in the command output.

Can you please give me some suggestions

And also can you please look into the below error. when i tried to open my celerra managment toll i am getting below error.

Today i have opend  the viruschk.conf file which my team has created, the INTERESTING thing is i did not find our new AV servers in the .conf file. still i could able to see old physical servers names. and one more intersting thing is when i use the server_viruschk server_2 command i can able to see new AV servers in the command output. Can you please give me some suggestions

Sounds like you are look at an old viruschk.conf file.

Please download a actual viruschk.conf file from the datamover (server_file server_X -get ...) and look at this one, this one must reflect the config you are seeing doins a server_viruschk server_X

Hi Peter,

Yes, you are correct. Thanks!!

We are getting perfomance issues in our NAS array, Can you please let us know the best practice to use the AV scanning for NAS array's

Current Environment:

2*Windows 2008 Serevr (Virtual Machine) per each site

Mecfee 8.8

CAVA 4.9.3

NAS array's

VNX 7500 array, which is in V-Block

NS80G, Backend storage V-MAX

NS80G, Backend Storage Clariion CX3-80