Start a Conversation

This post is more than 5 years old

Solved!

Go to Solution

5214

March 21st, 2017 08:00

Avamar Backup Rates

Hi Avamar Community -

We are currently installing Avamar 7.3.1 with Data Domain 4500 (DDOS 5.7.2).  We were expecting to get fairly fast backups since we are on a multi 10G link (Cisco UCS 6248's uplink to Cisco 5K's (which traverse to the 7K for routing)) with the storage being EMC VMAX and Xtreme IO.  However, we are noticing VM image level backups (i.e. first full backups) taking an extensive window to backup. For example, we have a VM that has 4.75TB provisioned space but that is using 2.9TB of that space.  We had to cancel the backup job after 9 hours.  We did another VM level backup for a machine that has 4.4TB that is only using 1.3TB.  This VM level backup took 8.75 hours.  We did other tests outside of VMware (and the UCS chassis) and the initial backups are consistently slow. 

My ultimate questions is: what should we be expecting for backups?  What kind of throughput are yall getting (or have got) on your first initial backup?

Any information is appreciated.

Karl

2K Posts

March 28th, 2017 10:00

The bottleneck for Avamar or Avamar / Data Domain backups is almost always the client filesystem. For VM Image backups, this is generally limited by the performance of the underlying datastore. However, if you want to be certain which component is the bottleneck, further analysis would be required.

When we say the typical backup performance is 100GB per hour or 1M files per hour, these are "real life" numbers based on analysis of performance statistics gathered from large numbers of customer systems. Your system may back up faster (or slower) but these are typical rates.

5 Practitioner

 • 

274.2K Posts

March 22nd, 2017 05:00

Hi Karl

Initially, you have to consider that image level backup will backup the VM disk (the VMDK file) therefore for the size of the file to backup this will be the provisioned space that you need to take into account.

Now there are a lot of backup optimisation in Avamar for image level backups such as change block tracking and other tweaks.

But they only start to take effect from the second backup onward.

The initial backup remains a full backup that will scan and backup the whole VMDK, it is expected to take a significantly longer than any subsequent backup.

March 22nd, 2017 08:00

Thank you for the response, but I'm really looking for some numbers on what people are getting.  As my initial thread noted, this is not just for VM backups; rather it is across our environment.  Also, how can Avamar backup empty space?  If a VM has 4TB provisioned but only using 1TB, how can it "read" empty space and why would it take so long to read a bunch of 0's?

2K Posts

March 23rd, 2017 07:00

Just to clarify a few things:

  • We do not read blocks of zeros beyond the allocated extents of a VMDK. We still need to include them in the backup but they are synthesized by the plug-in which is significantly faster than reading them from disk.
  • Large contiguous sections of zeros will be processed into identical chunks / segments. Generally, only the first copy of the "all zero" chunk needs to be sent across the wire since we will get cache hits or successful hash / fingerprint lookups for subsequent copies. The main exception to this is in Data Domain environments where the software sometimes intentionally sends duplicate segments to improve the spatial locality of the data on disk but this is generally a very small percentage of data.

5 Practitioner

 • 

274.2K Posts

March 23rd, 2017 07:00

In VM backup we read the VMDK like a physical device so we do not backup the files in it, we are reading the blocks like you would read sectors in a hard drive. So even if they are at "0" they are still present (except if the VMDK is thin provisioned) and still backed up.

now if you look at the best practice guide: https://support.emc.com/docu69642_Avamar-7.3-Operational-Best-Practices.pdf?language=en_US ( here Avamar 7.3 but this is the same on all the Avamar) on page 53

A typical backup should take about one hour for every million files in a file server or about one hour for every 100 GB of data in a database server.

Consider a VM to be similar to a database are we are backing up data block and not files

To be noted those are typical conservative performances on backup after the initial backup is taken, as mentioned the initial backup will be slower.

I advise you to onboard clients by wave and observe subsequent backup performance ( the initial backup will be slow as you have to send all the data, even if the data are block of zeros) and for VM please make sure that the change block tracking is activated as it will have a dramatic effect on performance

https://support.emc.com/docu69641_Avamar-7.3-for-VMware-User-Guide.pdf?language=en_US

Jerome

March 28th, 2017 06:00

Thank you jerome_fillon - that is good information; however, I'm struggling still with understanding the slowness.  What's the purpose of having multiple 10G NICs and SAS drives on the Data Domain if we are getting 250Mbps (that's megabits) or 31MBps?  100GB per hour cannot be the expected performance in this day and age can it?  That means to backup one of our servers it is going to take over 24 hours (2.4TB VM @ 100GB per hour).   

March 28th, 2017 11:00

We have tested with our all flash array (XIO) and our main production array (VMAX).  The datastores are enterprise class and certainly capable of far greater throughput.  Additionally, we have only backed up 1 VM at a time and getting this throughput.  We have TSM currently backing up our data and in some cases it has been 10x faster. 

Is there a community forum where I could get some real life numbers from Avamar users?  I really would like to see what people are getting in the present day. 

2K Posts

March 29th, 2017 13:00

If you need an analysis of the bottleneck in your environment, I would strongly urge you to work with support. Without making a deep dive into your environment and configuration, we can only speak in generalities, we can't give you any specific, useful advice on this forum.

This is certainly the right place to ask if you're looking for input from other Avamar users.

March 29th, 2017 13:00

Understand - we do have tickets open with every vendor looking into this (Cisco NXOS, Cisco UCS, and EMC) - I really just wanted to get some empirical data from the community on what they are seeing for throughput.  Thank you all for your responses.

- Karl

No Events found!

Top