Avamar Backup Rates

Question

Hi Avamar Community -

We are currently installing Avamar 7.3.1 with Data Domain 4500 (DDOS 5.7.2). We were expecting to get fairly fast backups since we are on a multi 10G link (Cisco UCS 6248's uplink to Cisco 5K's (which traverse to the 7K for routing)) with the storage being EMC VMAX and Xtreme IO. However, we are noticing VM image level backups (i.e. first full backups) taking an extensive window to backup. For example, we have a VM that has 4.75TB provisioned space but that is using 2.9TB of that space. We had to cancel the backup job after 9 hours. We did another VM level backup for a machine that has 4.4TB that is only using 1.3TB. This VM level backup took 8.75 hours. We did other tests outside of VMware (and the UCS chassis) and the initial backups are consistently slow.

My ultimate questions is: what should we be expecting for backups? What kind of throughput are yall getting (or have got) on your first initial backup?

Any information is appreciated.

Karl

ionthegeek · Accepted Answer

The bottleneck for Avamar or Avamar / Data Domain backups is almost always the client filesystem. For VM Image backups, this is generally limited by the performance of the underlying datastore. However, if you want to be certain which component is the bottleneck, further analysis would be required.

When we say the typical backup performance is 100GB per hour or 1M files per hour, these are "real life" numbers based on analysis of performance statistics gathered from large numbers of customer systems. Your system may back up faster (or slower) but these are typical rates.

Karl_Binger · Answer

Thank you for the response, but I'm really looking for some numbers on what people are getting.  As my initial thread noted, this is not just for VM backups; rather it is across our environment.  Also, how can Avamar backup empty space?  If a VM has 4TB provisioned but only using 1TB, how can it 'read' empty space and why would it take so long to read a bunch of 0's?

ionthegeek · Answer

Just to clarify a few things: We do not read blocks of zeros beyond the allocated extents of a VMDK. We still need to include them in the backup but they are synthesized by the plug-in which is significantly faster than reading them from disk. Large contiguous sections of zeros will be processed into identical chunks / segments. Generally, only the first copy of the 'all zero' chunk needs to be sent across the wire since we will get cache hits or successful hash / fingerprint lookups for subsequent copies. The main exception to this is in Data Domain environments where the software sometimes intentionally sends duplicate segments to improve the spatial locality of the data on disk but this is generally a very small percentage of data.

Karl_Binger · Answer

Thank you jerome_fillon - that is good information; however, I'm struggling still with understanding the slowness.  What's the purpose of having multiple 10G NICs and SAS drives on the Data Domain if we are getting 250Mbps (that's megabits) or 31MBps?  100GB per hour cannot be the expected performance in this day and age can it?  That means to backup one of our servers it is going to take over 24 hours (2.4TB VM @ 100GB per hour).

Karl_Binger · Answer

We have tested with our all flash array (XIO) and our main production array (VMAX). The datastores are enterprise class and certainly capable of far greater throughput. Additionally, we have only backed up 1 VM at a time and getting this throughput. We have TSM currently backing up our data and in some cases it has been 10x faster.

Is there a community forum where I could get some real life numbers from Avamar users? I really would like to see what people are getting in the present day.

ionthegeek · Answer

If you need an analysis of the bottleneck in your environment, I would strongly urge you to work with support. Without making a deep dive into your environment and configuration, we can only speak in generalities, we can't give you any specific, useful advice on this forum.

This is certainly the right place to ask if you're looking for input from other Avamar users.

Karl_Binger · Answer

Understand - we do have tickets open with every vendor looking into this (Cisco NXOS, Cisco UCS, and EMC) - I really just wanted to get some empirical data from the community on what they are seeing for throughput. Thank you all for your responses.

- Karl

Avamar

Avamar Backup Rates

Was this post helpful?