Avamar 7 & vCenter 5.1 - VM fails with 'snapshot creation failed'

I use to have this a LOT,  now very rarely.

One of the things I was told was to not start more than 100 vm backups at a time.

meaning in a group.

so I now have 3 groups

7 pm

8 pm

9 pm

I keep each one to 100 or less vm's

you don't have 100 VMs being backing up similanteniously, do you ? So many are running and the rest are sitting queued up ?  I have one group with 600 VMs, by default only 50 backups are running at the same time the rest are waiting in line.

My VMs are going to DD as well, what kind of DD do you have ..90 streams only ?

I would love to hear from EMC engineers about the desctinction between starting X number of VMs versus queing X number of VMs.

I did not say running I said start.....

they told me there was a difference

no group was to have more than 100 in it.

so for you it would be 6 groups.

and they would not have to be one hour apart, I did that to make it easy to see for me.

The EMC tech said not to start/queue more than 100 at a time.

I was just following instructions  ( and as I only have 3 proxies x 8 I only have 24 running at once.)

and the reason I only have 24 is I have my vm's landing on DataDomain

and I have to consider the 90 streams that DD can handle

so that has to include my vm backups, my vm replications, and any database backups that are going on via CIFS or NFS

I have to stay below that 90 count on the DD to make sure jobs don't die.

that dd is an 860 - and I was told it can only do 90 streams

I don't know if it was an Avamar issue or a vCenter issue.... like don't kill the vcenter by trying to queue up a bunch of servers at once.

Think maybe that when you queue the job it has to talk to vcenter to verify it is there?  That could be a load on vcenter maybe?

I just know I had issues, and that was 2 of the things they stressed to me.

Thanks for the feedback, everyone.

Ahmad Arif - to answer your questions:

1. This vCenter is fairly small - there are only 24 VM's total, which all kickoff at the same time.
2. There is only one single proxy deployed for this vCenter.
3. No known issues that I'm aware of, but obviously something is causing some problems!

Before I open an SR with EMC for further investigation, can you let me know how many VM's can be handled by a single proxy?  At what threshold are additional proxies suggested?  I'm not opposed to deploying an additional proxy or splitting the VM backups into different groups, if the issue might be caused by timeouts/queued VM's.

Thanks

1- that small I would think that is not an issue

and the number of proxies - really goes with your window -

as you are on 7.0.2-47  your proxies with one proxy in vCenter should show up as

1-1

1-2

1-3

....

1-8

One proxy in vcenter can do 8 vm's at a time and can show up as 8 proxies in avamar

if you make your window no need for more

but as you only have 24 proxies I would not think you would need to make more than one job.

you might need to do a ticket so they can look at logging to find out what is your cause.

Different aspects could lead to different root cause:

1. If you are runnning Avamar 7.0, Maximum number of proxy sessions back to the Avamar MCS is 48… if you have a single proxy with 8 streams, that will be 8 MCS sessions if all streams are being used at once.  In Avamar 7.1 (I think, if someone can confirm here), that changes to reduce the load on the MCS… each proxy will only establish a single connection to the MCS so a single proxy with 8 streams will only create a single MCS connection instead of 8..

So you have a limitation here that could cause it

2. Data domain model supporting the maximum number of streams, like discussed above.

3. If your vCenter version is 5.1 update1, there is a known issue where the connection between the third party backup and vcenter drops which leaves the snapshot unremoved from the VMs, the kb article is here: VMware KB: VMware VirtualCenter Server service fails randomly when using 3rd party backup solution

Hope this helps.

with your scenario, where you have hot add disks on your proxy, I would check the VPXD logs on vCenter for any signs of drop in connections.