UNSOLVED

clbarnett

updated

16 years ago

C

clbarnett

4 Posts

0

1980

October 1st, 2010 13:00

CAVA Heartbeat

I'm trying to track down the root cause of a problem we've experienced. For some reason, even though the Windows CAVA servers are running and the AV is running on them, our Celerra has been reporting that the AV servers are down. This has happened 3 times in the past week or so, at different times but always between 10PM and midnight. We see this message: Sep 27 22:04:38 2010 VC:3:30 Slot 2: 1285643078: Error on CAVA server xxx.xxx.xxx.xxx: OFFLINE, ntStatus: CONNECTION_DISCONNECTED

What exactly is the heartbeat check the Celerra does to see if the CAVA servers are alive and well? Does this monitoring consist of a TCP Keep-Alive packet? And the expected response is another TCP Keep-Alive packet? Or is there something more sophisticated going on where the vc client actually checks if the AV is running?  I gues,s I'm trying to understand what would cause that above message to appear on the vc client side when from the CAVA side there's nothing going on (no files are being checked, no backups are going on, no updates are going on, no error messages are being logged).

  • clbarnett

    4 Posts

    564

    0

    Posted October 1st, 2010 14:00

    Thanks for the quick response.  I thought I read somewhere it's a 10-second heartbeat.  I'll have to get someone to give me the output on the Celerra.

    We've engaged our network/Datacom folks, and they say there's nothing unusual going on at that time.  The AV servers do not update at that time (I'm manually updating them while we're working on this).  There are no scheduled reboots. One of the AV servers doesn't even have backups running, so I know it's not backups interfering. I can tell by the Windows event logs that there are no service starts or stops at the time of the errors, and I can look at the AV logs to see that the AV does not start or stop.  We have no users on the system yet, so I know there is no AV scanning activity going on during this timeframe.  I don't know what you mean by a process to set the time on read, but we don't scan on read on the Celerra side at all.   Also, the Celerra reports both CAVA servers are down at the same time, even though they are physically located on different parts of the network and neither show any sort of errors.

  • bergec

    275 Posts

    564

    0

    Posted October 1st, 2010 14:00

    I think there is a 60 seconds hearthbit ower RPC (ONC or MS RPC), check the output of "server_viruscheck server_2" which will show the configuration (params that you have set such as nthreads and default values for other params).

    Now the question is "why are the seen as Disconnected?"

    Anything going on on the network at that time, are all AV servers doing an update of the Virus definition at the same time (or rebooting for some reason), or is there a process somewhere to set thetime on read (for scan on read enable)?

    Claude

  • sebbyr

    99 Posts

    564

    0

    Posted October 3rd, 2010 14:00

    Hello.


    Could you please check the Event Viewer on the CAVA server and determine what's going on, if anything, with the AV software service?  It could be that this is going down, which would cause the CAVA service to go down, and therefore the viruschecker service losing connection.

    Thanks.

    - Sebby Robles

  • clbarnett

    4 Posts

    564

    0

    Posted October 4th, 2010 06:00

    Hi Sebby,

    As stated above, I can tell by the Windows event logs that there are no service starts or stops at the time of the errors, and I can look at the AV logs to see that the AV does not start or stop.

    Is there any documentation on what the 'heartbeat' packets look like? My datacom guy is convinced that they are simple 'Keep-Alive' packets, but if they are not these simple 'Keep-Alive' packets, then the data he has collected is irrelevant.

    Thanks!

    Cheryl

  • Burt3

    10 Posts

    564

    0

    Posted December 10th, 2010 10:00

    I am having the same issues.  Sometimes the error happens multiple times a day.  Ous happen most any time of the day or night.

    Most of the time I connect to the NAS and find that "server_viruschk ALL"  will show all the CAVA Servers to be ONLINE.  As in the other post, there are no services interruptions, no windows updates, nothing going on on the CAVA servers.  I am looking for the following:

    1) I would like to find out the root cause for this issue so it can be corrected.

    2) until the root cause can be identified and resolved, is there anything that can be done to "double check" the status of before reporting the error?

    3) are there settings that can be updated to reduce the liklihood of reporting non-errors?

  • bergec

    275 Posts

    564

    0

    Posted December 16th, 2010 23:00

    Are you running AV engines on physical servers or VMware VMs?

    Claude

  • clbarnett

    4 Posts

    564

    0

    Posted December 17th, 2010 06:00

    I have 3 physical servers, one is a blade and 2 are physical servers.

    To the other user above asking if there can be some sort of a 'double-check', I was looking at the publication "Configuring EMC Celerra Events and Notifications", version 5.6.  On p. 62 of 138, it summarizes the Celerra Network Server notifications (actions) you can take when an event occurs.  One of the actions is 'exec', which will 'Execute the procedure specified by with the arguments specified by .  I guess it would be possible to write some sort of script which would double-check?  I'm not a Celerra admin, just the AV person, so I can only theorize that it looks like there is a framework that would allow that.  Can't help with the details.