Highlighted
Karl_Binger
1 Nickel

Avamar Backup Rates

Jump to solution

Hi Avamar Community -

We are currently installing Avamar 7.3.1 with Data Domain 4500 (DDOS 5.7.2).  We were expecting to get fairly fast backups since we are on a multi 10G link (Cisco UCS 6248's uplink to Cisco 5K's (which traverse to the 7K for routing)) with the storage being EMC VMAX and Xtreme IO.  However, we are noticing VM image level backups (i.e. first full backups) taking an extensive window to backup. For example, we have a VM that has 4.75TB provisioned space but that is using 2.9TB of that space.  We had to cancel the backup job after 9 hours.  We did another VM level backup for a machine that has 4.4TB that is only using 1.3TB.  This VM level backup took 8.75 hours.  We did other tests outside of VMware (and the UCS chassis) and the initial backups are consistently slow. 

My ultimate questions is: what should we be expecting for backups?  What kind of throughput are yall getting (or have got) on your first initial backup?

Any information is appreciated.

Karl

0 Kudos
1 Solution

Accepted Solutions
8 Krypton

Re: Avamar Backup Rates

Jump to solution

The bottleneck for Avamar or Avamar / Data Domain backups is almost always the client filesystem. For VM Image backups, this is generally limited by the performance of the underlying datastore. However, if you want to be certain which component is the bottleneck, further analysis would be required.

When we say the typical backup performance is 100GB per hour or 1M files per hour, these are "real life" numbers based on analysis of performance statistics gathered from large numbers of customer systems. Your system may back up faster (or slower) but these are typical rates.

0 Kudos
9 Replies
jerome_fillon
1 Nickel

Re: Avamar Backup Rates

Jump to solution

Hi Karl

Initially, you have to consider that image level backup will backup the VM disk (the VMDK file) therefore for the size of the file to backup this will be the provisioned space that you need to take into account.

Now there are a lot of backup optimisation in Avamar for image level backups such as change block tracking and other tweaks.

But they only start to take effect from the second backup onward.

The initial backup remains a full backup that will scan and backup the whole VMDK, it is expected to take a significantly longer than any subsequent backup.

0 Kudos
Karl_Binger
1 Nickel

Re: Avamar Backup Rates

Jump to solution

Thank you for the response, but I'm really looking for some numbers on what people are getting.  As my initial thread noted, this is not just for VM backups; rather it is across our environment.  Also, how can Avamar backup empty space?  If a VM has 4TB provisioned but only using 1TB, how can it "read" empty space and why would it take so long to read a bunch of 0's?

0 Kudos
jerome_fillon
1 Nickel

Re: Avamar Backup Rates

Jump to solution

In VM backup we read the VMDK like a physical device so we do not backup the files in it, we are reading the blocks like you would read sectors in a hard drive. So even if they are at "0" they are still present (except if the VMDK is thin provisioned) and still backed up.

now if you look at the best practice guide: https://support.emc.com/docu69642_Avamar-7.3-Operational-Best-Practices.pdf?language=en_US ( here Avamar 7.3 but this is the same on all the Avamar) on page 53

A typical backup should take about one hour for every million files in a file server or about one hour for every 100 GB of data in a database server.

Consider a VM to be similar to a database are we are backing up data block and not files

To be noted those are typical conservative performances on backup after the initial backup is taken, as mentioned the initial backup will be slower.

I advise you to onboard clients by wave and observe subsequent backup performance ( the initial backup will be slow as you have to send all the data, even if the data are block of zeros) and for VM please make sure that the change block tracking is activated as it will have a dramatic effect on performance

https://support.emc.com/docu69641_Avamar-7.3-for-VMware-User-Guide.pdf?language=en_US

Jerome

8 Krypton

Re: Avamar Backup Rates

Jump to solution

Just to clarify a few things:

  • We do not read blocks of zeros beyond the allocated extents of a VMDK. We still need to include them in the backup but they are synthesized by the plug-in which is significantly faster than reading them from disk.
  • Large contiguous sections of zeros will be processed into identical chunks / segments. Generally, only the first copy of the "all zero" chunk needs to be sent across the wire since we will get cache hits or successful hash / fingerprint lookups for subsequent copies. The main exception to this is in Data Domain environments where the software sometimes intentionally sends duplicate segments to improve the spatial locality of the data on disk but this is generally a very small percentage of data.
Karl_Binger
1 Nickel

Re: Avamar Backup Rates

Jump to solution

Thank you jerome_fillon - that is good information; however, I'm struggling still with understanding the slowness.  What's the purpose of having multiple 10G NICs and SAS drives on the Data Domain if we are getting 250Mbps (that's megabits) or 31MBps?  100GB per hour cannot be the expected performance in this day and age can it?  That means to backup one of our servers it is going to take over 24 hours (2.4TB VM @ 100GB per hour).   

0 Kudos
8 Krypton

Re: Avamar Backup Rates

Jump to solution

The bottleneck for Avamar or Avamar / Data Domain backups is almost always the client filesystem. For VM Image backups, this is generally limited by the performance of the underlying datastore. However, if you want to be certain which component is the bottleneck, further analysis would be required.

When we say the typical backup performance is 100GB per hour or 1M files per hour, these are "real life" numbers based on analysis of performance statistics gathered from large numbers of customer systems. Your system may back up faster (or slower) but these are typical rates.

0 Kudos
Karl_Binger
1 Nickel

Re: Avamar Backup Rates

Jump to solution

We have tested with our all flash array (XIO) and our main production array (VMAX).  The datastores are enterprise class and certainly capable of far greater throughput.  Additionally, we have only backed up 1 VM at a time and getting this throughput.  We have TSM currently backing up our data and in some cases it has been 10x faster. 

Is there a community forum where I could get some real life numbers from Avamar users?  I really would like to see what people are getting in the present day. 

0 Kudos
8 Krypton

Re: Avamar Backup Rates

Jump to solution

If you need an analysis of the bottleneck in your environment, I would strongly urge you to work with support. Without making a deep dive into your environment and configuration, we can only speak in generalities, we can't give you any specific, useful advice on this forum.

This is certainly the right place to ask if you're looking for input from other Avamar users.

0 Kudos
Karl_Binger
1 Nickel

Re: Avamar Backup Rates

Jump to solution

Understand - we do have tickets open with every vendor looking into this (Cisco NXOS, Cisco UCS, and EMC) - I really just wanted to get some empirical data from the community on what they are seeing for throughput.  Thank you all for your responses.

- Karl

0 Kudos