Could you please post detailed error logs here, that would be helpful to investigate the issue.

Regards,

Pawan

Hello,

Avamar does not allow to run multiple backup of any client at a same time. If we start second backup of a client for which already one backup is running, second backup should be in queued state. Are you trying to run multiple backup at same time ?

Regards,

Pawan

Hi Pawan,

No.  I have one schedule for this server and it runs once.  That is what is puzzling.  Could there be a plugin conflict since separate backups might run for different plugins?

I am trying to past the log messages or attach a small Notepad document.  Can that be done here?

Yes, you can copy or attach logs here. If any log contains internal information (Client IP or server IP) please remove that.

Regards,

Pawan

Hi Pawan,

Below are the messages:

2014-06-19 19:49:45 avtar Info <5581>: Logging in on connection 0 with Session Ticket
2014-06-19 19:49:45 avtar Info <5582>: Avamar Server login successful
2014-06-19 19:49:45 avtar Info <5018>: - Session ID: 0
2014-06-19 19:49:45 avtar Info <5550>: Successfully logged into Avamar Server (Compression enabled)
2014-06-19 19:49:45 avtar Info <7563>: Back up of "/" on server "xxx.xxxxxx.xxxxx.xxxx" for /Proxys/xxxx.xxxxx.xxxx
2014-06-19 19:52:21 avtar Info <5586>: Loading cache files from /usr/local/avamar/var
2014-06-19 19:52:21 avtar Info <5573>: - Loaded cache file /usr/local/avamar/var/f_cache.dat (2,884,128 bytes)
2014-06-19 19:52:21 avtar Error <5064>: Cannot open file "/usr/local/avamar/var/p_cache.dat"
2014-06-19 19:52:21 avtar Info <5065>: Creating new cache file /usr/local/avamar/var/p_cache.dat (1,573,408 bytes)
2014-06-19 19:52:21 avtar Error <5803>: Error writing 32-byte header to cache file /usr/local/avamar/var/p_cache.dat. Possibly out of disk space
2014-06-19 19:52:21 avtar Info <5573>: - Loaded cache file /usr/local/avamar/var/p_cache.dat (1,573,408 bytes)
2014-06-19 19:52:21 avtar FATAL <5224>: Cannot run two backups at the same time
2014-06-19 19:52:21 avtar Info <6149>: Error summary: 3 errors: 5803, 5224, 5064
2014-06-19 19:52:21 avtar Info <8468>: Sending wrapup message to parent
2014-06-19 19:52:21 avtar Info <5314>: Command failed (3 errors, exit code 10003: Lock error: Process already running or error creating local lock file)

Hello,

Please try this (though it is for windows client but you can try same steps for linux client) -

Cause 1:

Check for failed or hung Avtar job on the client workstationby display all the running jobs in the task manager.

On the client workstation:

1. Bring up the  Windows Task Manager
2. Click on the "  Process" tab.
3. Select "  Show processes from all users" to display all jobs.
4. Click on "  Image Name" to sort the jobs alphabetically in descending order.
5. Look for an avtar job that is still running. 
    
   If an old avtar job is found then you have three ways to kill it. 
   1. Kill the job through the Windows Task Manager. 
   2. Kill the old job with the "kill.exe" command from the Windows Resource Kit. 
   3. Reboot the client.

Method 1:

If you logged in to the workstation as a localadministrator

1. Record the Process ID displayed with the avtar job.
2. Cick on the avtar job to highlight it.
3. Rght click on the job and select "  End Process".

If the job is terminated then try rerunning the job.

If the error message "Cannot kill job from taskmanager" or"The operation could not be completed. Access isdenied" error message is displayed then you will have to usemethod 2.

The reason the job cannot be killed in this stage is becausethe ID you are logged in with does not have the administrativerights to kill the job. This issue is documented in MS KB 155075.Forr more information see, http://support.microsoft.com/kb/155075.


Method 2:

While logged in with an ID that has administrative rights onthe client workstation:

1. Open a command shell. 
2. Verify you have the "kill.exe" command installed on the workstation. 
3. If the "  kill.exe" command is not installed on the workstation then install it from the Windows Resource Kit. 
4. Issue the "kill " command to kill the job from the command line. 
   - - - 
     kill -f 1828 
   process avtar.exe (1828) - '' killed 
   - - - 
   The kill command will reply that the job has been killed. 
5. Verify the job has been killed by checking for the process in the Windows Task Manager. 
   - - - 
     pslist " find "av" 
   avscc 2640 8 6 117 6388 0:00:09.390 0:33:34.201 
   avagent 2428 8 13 148 6100 0:00:10.734 0:13:22.286 
   - - - 
6. Rerun the job. 

Method 3:
If all the above fail then reboot the client workstation.

1. Verify there are no avtar jobs running. 
2. Rerun the Job. 
   - - - 
   Sample output from the msinfo32 command showing a "  hung" avtar job. 
   Note there are no resources allocated to the job. 
   - - - 
     System Information report written at: 12/06/2007 12:27:35 PM 
   [Running Tasks] 
    
   Name Path Process ID Priority Min Working Set Max Working Set Start Time Version Size File Date 
   avagent.exe c:\program files\avamar\bin\avagent.exe 2352 8 204800 1413120 11/20/2007 1:59:40 PM 3, 7, 1, 100 2.95 MB (3,092,539 bytes) 8/30/2005 3:40:18 PM 
   avscc.exe c:\program files\avamar\bin\avscc.exe 2640 8 204800 1413120 12/6/2007 12:16:17 PM 3, 7, 1, 100 3.25 MB (3,407,935 bytes) 8/30/2005 3:40:32 PM 
   avtar.exe \\?\c:\program files\avamar\bin\avtar.exe 1828 6 204800 1413120 11/16/2007 9:00:00 PM Unknown Unknown Unknown 
   - - -

  Cause 2:

Verify there is sufficient space on the drive where theclient is installed.

This issue is often resolved by reinstalling the client on alocal drive with sufficient space for the cache files to go grow totheir maximum size (RAM x 5/16).

Regards,

Pawan

Thanks Pawan. I will try these and let you know what happens.

Dave

Hello Dave,

Were you able to fix this issue ?

Regards,

Pawan

Hi Pawan,

Thanks for the response and information.

I will know if this is fixed after I change the excludes and check if the files were excluded from the next backup.

I’ll let you know what happens if you want.

Dave

Yes, Please let us know. It will be helpful for others

Regards,

Pawan

Hi Pawan,

Facing similar kind of issue but here its not even loading the cache.dat files, we have only scheduler for this client and this is Solaris OS . here are the logs. Both client and avamar version is 7.1.100-370. Any idea?

2015-04-30 17:00:03 avtar Info <8650>: Opening filename cache file '/opt/AVMRclnt/var/f_cache.dat'

2015-04-30 17:00:03 avtar Error <5064>: Cannot open file '/opt/AVMRclnt/var/f_cache.dat'
2015-04-30 17:00:03 avtar Error <19061>: Unable to load filename cache file '/opt/AVMRclnt/var/f_cache.dat'
2015-04-30 17:00:03 avtar Info <8650>: Opening hash cache file '/opt/AVMRclnt/var/p_cache.dat'
2015-04-30 17:00:03 avtar Error <5064>: Cannot open file '/opt/AVMRclnt/var/p_cache.dat'
2015-04-30 17:00:03 avtar Error <18907>: Unable to load hash cache file '/opt/AVMRclnt/var/p_cache.dat'
2015-04-30 17:00:03 avtar FATAL <5224>: Cannot run multiple backups with the same hash cache name at the same time. Consider changing VARDIR or using --cache-prefix option.

Regards,

Dileep

Hi Dileep,

It seems different issue. Were you able to run successful backup earlier ? I would suggest to rename current var directory to something else and create a new var directory and try to run a small test backup.

Regards,

Pawan

I would suggest to rename current var directory to something else and create a new var directory and try to run a small test backup.

I would not recommend this as it will wipe out the cache files and those are very important to backup performance.

This is probably just a hung avtar. Please use the ps utility to check if there is an avtar process hung on this system. If you find one, use kill to terminate it and the next backup should run fine.

Is it possible to run manually 2 backups on cli , by copying avtar to 2 folders and renaming binaries to avtar1 and avtar2, each binary will write to its own cache files