I thought Avamar only allowed a single job per client to run, so if you run more than one or a scheduled job starts when another is already running, the new jobs just queue up and wait. That's been my normal experience.
However last night one of our SQL backups appears to have run at the same time as the filesystem backup. Both are scheduled for 8pm and normally the filesystem backup completes and the SQL job runs immediately afterwards.
Today I looked at the logs and they show the filesystem backup starting at 8:00 and the SQL backup starting at 8:01, with all but one database completing the backup and the last one having a VDI timeout error.
Has anyone seen this behavior before? Is that normal/expected? If not, any idea why it occurred?
2016-01-25 20:00:01 avtar Info <7977>: Starting at 2016-01-25 20:00:01 Central Standard Time [avtar May 24 2014 00:00:10 7.0.102-43 Windows Server 2003 Standard Server Edition Service Pack 2-x86]
2016-01-25 20:00:01 avtar Info <10684>: Setting ctl message version to 3 (from 1)
2016-01-25 20:00:01 avtar Info <16136>: Setting ctl max message size to 268435456
2016-01-25 20:00:01 avtar Info <6648>: Successfully connected to 127.0.0.1:1026 with proprietary encryption
SQL backup log:
Log #1: avtar log 2016-01-25 20:01:28 Central Standard Time [7.0.102-43 Windows Server 2003 Standard Server Edition Service Pack 2-x86]
2016-01-25 20:01:28 avtar Info <5008>: Logging to C:\Program Files\avs\var\Every_2_hours_10am-8pm-SQL_Full_-_Every_2_Hours-1453773600957#0-3006-SQL.avtar.log
2016-01-25 20:01:28 avtar Info <7977>: Starting at 2016-01-25 20:01:28 Central Standard Time [avtar May 24 2014 00:00:10 7.0.102-43 Windows Server 2003 Standard Server Edition Service Pack 2-x86]
2016-01-25 20:01:28 avtar Info <10684>: Setting ctl message version to 3 (from 1)
2016-01-25 20:01:28 avtar Info <16136>: Setting ctl max message size to 268435456
2016-01-25 20:01:28 avtar Info <6648>: Successfully connected to 127.0.0.1:2136 with proprietary encryption
One of my colleagues was able to track down the escalation. This is bug number 206145 which is fixed in 7.2.0. As a workaround, you can separate the datasets so that the SQL and filesystem backups run in two different groups. This should force one of the backups to queue.
Thanks Ian. I think I found the documentation on the bug (https://emc--c.na5.visual.force.com/apex/KB_BreakFix_1?id=kA1700000001DjK) but that's not the same behavior as what I'm seeing. Neither of my backups failed. I did have a VDI timeout on one of the SQL databases, but I don't think that's related as the other databases backed up fine.
My backups are already running in different groups, one for the SQL backup (run every two hours) and the other for the filesystem backup (start at 8pm).
Do you know how the issue occurs? Every time I've ever run two backup jobs at the same time, or have tried to start a job when another is already running, it always queues. In this case the filesystem job had been under way for nearly 90 seconds before MCS attempted (and succeeded) to run the 8pm SQL job. Plus this doesn't occur every day, only occasionally.
I've seen this before for plug-ins that use multiple avtars to implement multi-streaming. I believe it depended on the order in which the jobs started (if the filesystem job started first, the SQL job would queue but not vice-versa). Let me see if I can dig up the details.
pchassang
33 Posts
1405
0
Posted January 26th, 2016 07:00
To help you, we need log output of jobs.
regards.
Pascal