Ask the Expert: Avamar Server management best practices

This Discussion is now open for questions to Ian. We look forward to this discussion.

Regards,

Mark

Hello,is there any way to get a kind of dedup ratio for the whole avamar grid? For example like it is solved with Datadomain systems, there you have a dedup ratio on the start page. This is what many customers ask for. They dont like to calculate the ratio from the dpnsummaru output.

Unfortunately there's no quick and easy way to get the dedup ratio for a whole grid. I've filed a Request For Enhancement (RFE) to request that a future version of the Enterprise Manager (EM) report the global dedup ratio for each grid.

In the meantime, it's possible to set up ODBC on a Windows system so that it can query the SQL Views in the Avamar Administrator Server database. The DPN Summary report is based on information found in the v_dpnsummary view, so with an ODBC connection configured you could use database software, a spreadsheet, a reporting package, etc. to run an automatic calculation or report based on the information in v_dpnsummary. It's not the most straightforward option but it would work.

Ian, a two-part question for you:

1:

when we activate a client by right-clicking on the avamar client icon in the toolbar and saying “Manage, Activate”, the Avamar server adds a suffix like ".our-org.com".

So the client name will be something like "websrv01.our-org.com".

We have many avamar servers, and they are not set up identically, so today, we have many different suffixes: "our-org.com", "ent.our-org.com", no-suffix, etc.

On the other hand, when we activate a node using the MCCLI command line, we can give it any name we want, including no suffix at all.  No-suffix seems to works fine, backups work fine, Avamar has the full DNS name in its database somewhere - all is good.

But we'd like to get our client names consistent, if we can.

Is there any reason NOT to just use (for example) "websrv01" instead of "websrv01.our-org.com"?

I would guess that if there WERE a problem, the client would fail to activate.

Right?

2:

We can rename a client using, for example, "mccli client edit --name=webssrv01.our-org.com --new-name=websrv01".

And if we do, do its old backups get orphaned? So would  we have to rename them, too?

Thanks for your advice here.

For the first question, we recommend using the fully qualified name instead of the short name because by default, Avamar will not allow two clients with the same name to be activated to the same grid. This could be an issue if you have, for example, Windows Domain Controllers called "dc1.west.our-org.com" and "dc1.east.our-org.com". One of these clients would fail to activate because of the name conflict.

The answer to the second question requires a bit of background first.

When explaining how client accounts work, I always like to use a bucket as a metaphor for a client account. Inside the bucket, you will find all the backups for that client. Engraved into the handle of the bucket is a unique identifier called the Client ID or CID. There is also a label on the bucket (the hostname) but that's only used for two things:

1. We use the hostname as a human-friendly identifier because humans aren't very good at remembering 20-byte hexadecimal strings (funny, that).
2. We compare the hostname and CID whenever an activated client checks in. If the hostname and CID do not match what we have recorded, we will not issue backup or restore workorders to that client. This is to prevent a rogue client from stealing the hostname of an activated client and impersonating it.

So using the bucket analogy, changing the label on the bucket (the hostname) will not affect the contents of that bucket (the backups).

I ran some quick testing of this and as long as the client name you use as "new-name" is the short name or any of the fully qualified names associated with the client, you shouldn't have any issues. You do have to be careful with this, however. If you make a typo in the new-name, the Avamar Backup Agent will not be able to process workorders because the mapping between the hostname and CID on the client will not match the mapping between the hostname and CID on the Avamar server.

I hope this helps!

Validating a backup is essentially doing a restore and discarding the results. It will prove that the backup is consistent (in other words, the backup is on the server and all the bits that were backed up can be restored) but it doesn't tell us anything about the content of that backup. For example, on Windows clients, if VSS is not functioning correctly, any open files on the client will not be backed up. This backup will be valid (since it is consistent and restorable) but if you were trying to restore one of those open files, you will not be able to do so.

When reviewing the activities, keep in mind that each plugin has a specific purpose. If the filesystem backup succeeds but the VSS backup fails, it will not be possible to perform a bare metal recovery of this system but any data backed up as part of the file system backup will be available for restore. Similarly, if your DB2 backups succeed but your filesystem backups fail, you will be able to recover the databases but not the file system. Naturally, this applies to any plug-in backups.

There are some reports built into the Avamar Administrator that you may find useful for determining if a client is fully protected or not. In particular, take a look at the "Activities - Exceptions", "Activities - Failed" and "Client - No Activities" reports. The two "Activities" reports are pretty self-explanatory (they report on any clients that complete with exceptions or fail). The "Client - No Activities" report will give you information on any client that is not running backups.

It is also possible to create your own reports using the GUI, though the options for customization are somewhat limited.

You can configure the system to send built-in or custom reports to you by e-mail on a daily basis. If you find that these reports are not sufficient for your needs, you could use the ODBC connectivity I mentioned above to "roll your own" or you could use a reporting package such as Data Protection Advisor to generate more in-depth reports "out of the box".

How do we ensure that we have a good backup with Avamar?

We can try using "mccli backup validate..."; but that runs a long time, and I don't see where the results go.

We can look at the "Last Successful Backup Date" from "mccli client show...", but that apparently reports success even if the backup had exceptions.

We can look at activities, and ee if they Completed, Completed With Exceptions, etc.  But do we have to look for a success on each plugin, to be sure?  For example, Windows filesystem and Winodws VSS have to both succeed, right?

Just looking for a straightforward way to day "Yes, we're OK on server xxx".

Thanks.

Ian, thanks for your knowledgeable and well-stated responses.

Here's a new question:

I’m trying to back up my laptop. I get thousands of errors like this:

      2012-03-30 08:37:20 avtar Error <5137>: Unable to open "C:\temp.txt" (code 5: Access is denied).

I can open these files myself.

The Backup Agent service is running as “Local System Account”.

Why can’t it open these files (thousands of them - possibly all files on the machine)?  Any ideas?

Also, the GUI reports Success even though thousands, possibly all, files failed.  That doesn't seem right.  Is there a flag I can set to prevent bogus "Success" results?

My pleasure!

I would recommend reviewing the NTFS permissions for these files. It's possible somebody has deleted the SYSTEM account from the ACL or removed the account's read permissions for these files. By default, the SYSTEM account is granted Full Control for all files on a system and Microsoft recommends that this not be changed.

I've also seen this message if the VSS snapshot fails for some reason (for example if the snapshot is removed out from under the running backup). See if there are any SnapVol errors in the Windows event logs.

Are the backups being marked "Completed" or "Completed w/exceptions"?

For C:/Temp.txt, SYSTEM has Full control.

I see no Snapvol events in Application, Security, or System event logs.

Avamar Client,  "Backup...", History says "Completed Successfully", no indication of exceptions.

Avamar Client, Manage, View console  says "Completed (45055 errors)";

Avamar administrator, Log on to Server, Activities, says "Completed w exceptions".

So I still don't see why these files are not readable. 

Is a windows service involoved? Can I bypass it?  I tried cutting and pasting the "avtar.exe ..." command line from the ..avs/var/clientlogs/... into a command shell to try and run the backup as myself, not through the service.  But I got:

C:\Program Files\avs\bin>avtar --sysdir="C:\Program Files\avs\etc" --bindir="C:\Program Files\avs\bin" --vardir="C:\Program Files\avs\var" --ctlcallport=1706 --

ctlinterface="3001-Windows-Windows-Test -1333110692426" --logfile="C:\Program Files\avs\var\clientlogs\lm-cmdlinetest.log" --sessionattr=dtlt=true

avtar Info <5241>: Logging to C:\Program Files\avs\var\clientlogs\lm-cmdlinetest.log

avtar Info <5551>: Command Line: avtar --sysdir="C:\Program Files\avs\etc" --bindir="C:\Program Files\avs\bin" --vardir="C:\Program Files\avs\var" --ctlcallport

=1706 --ctlinterface="3001-Windows-Windows-Test -1333110692426" --logfile="C:\Program Files\avs\var\clientlogs\lm-cmdlinetest.log" --sessionattr=dtlt=true

avtar FATAL <10790>: Unable to connect to 127.0.0.1:1706 with proprietary encryption

avtar Info <9901>: Cancel Request being processed (setting code from 0 to 536870920)

Appreciate any advice.  Guess I could open a ticket with EMC...

It is possible to run avtar manually but you can't copy and paste the command line from the log because avtar commands that are run as part of scheduled backups use the "CTL" interface to communicate with their caller to retrieve the information required to run the backup (targets, options, etc.) and return status messages. For file system backups the caller will be the "Avamar Backup Agent" or "Backup Agent" service (internally we call it "avagent" which is the name of the binary). Since avagent didn't start this avtar process, it won't be listening for replies and you will receive the FATAL you've pasted above.

I think it would be best to open a service request for this issue. Support can do an in-depth analysis of the logs or work with you live via WebEx and you're likely to get a faster resolution this way.

If you speak with L2 support, they can show you how to run a "degenerate" test that will process the filesystem but discard the results instead of sending them to the server. Such a test is normally used to isolate performance bottlenecks (it measures how fast avtar can read the filesystem since it doesn't have to wait for replies to come back from the server) but it would also be useful for this type of troubleshooting since it would allow you to keep your tests local to the client.

We have one server with 25 million files, scattered through directories six levels deep.

We'd like to throw it at our test Avamar grid; any tuning I should look at on the client 9or  server) side before we set it up for its first backup?

We plan to migrate a few thousand servers to several Avamar grids.  But we don't want to throw too many first-time on-demand backups at an Avamar grid all at once.

How many is too many?  Or is there any reason not to queue up on-demand backups for 1,000 different servers and wait for them to complete?

(Obviously we don't want to throw more data at the grid than it can hold after being de-duped.   We think we have that part figured out.)

Another way of saying it:

We can monitor various things:

  total client GB that will be ingested,

  number of sessions in use on the Avamar server,

  disk IO,

  CPU utilization,

etc.  Is there a metric on one of these that we should be careful not to exceed? 

lmorris99 wrote: We have one server with 25 million files, scattered through directories six levels deep. We'd like to throw it at our test Avamar grid; any tuning I should look at on the client (or server) side before we set it up for its first backup?

The most important thing to do on a client with so many files is to make sure that the file cache is sized appropriately. The file cache is responsible for the vast majority (>90%) of the performance of the Avamar client. If there's a file cache miss, the client has to go and thrash your disk for a while chunking up a file that may already be on the server.

So how to tune the file cache size?

The file cache starts at 22MB in size and doubles in size each time it grows. Each file on a client will use 44 bytes of space in the file cache (two SHA-1 hashes consuming 20 bytes each and 4 bytes of metadata). For 25 million files, the client will generate just over 1GB of cache data.

Doubling from 22MB, we get a minimum required cache size of:

22MB => 44MB => 88MB => 176MB => 352MB => 704MB => 1408MB

The naive approach would be to set the filecachemax in the dataset to 1500. However, unless you have an awful lot of memory, you probably don't want to do that since the file cache must stay loaded in memory for the entire run of the backup.

Fortunately there is a feature called "cache prefixing" that can be used to set up a unique pair of cache files for a specific dataset. Since there are so many files, you will likely want to work with support to set up cache prefixing for this client and break the dataset up into more manageable pieces.

One quick word of warning -- as the saying goes, if you have a hammer, everything starts to look like a nail. Cache prefixing is the right tool for this job because of the large dataset but it shouldn't be the first thing you reach for whenever there is client performance tuning to be done.

On to the initial backup.

If you plan to have this client run overtime during its initial, you will have to make sure that there is enough free capacity on the server to allow garbage collection to be skipped for a few days while the initial backup completes.

If there is not enough free space on the server, the client will have to be allowed to time out each day and create partials. Make sure the backup schedule associated with the client is configured to end no later than the start of the blackout window. If a running backup is killed by garbage collection, no partial will be created.

You will probably want to start with a small dataset (one that will complete within a few days) and gradually increase the size of the dataset (or add more datasets if using cache prefixing) to get more new data written to the server each day. The reason for this is that partial backups that will only be retained on the server for 7 days. Unless a backup completes successfully within 7 days of the first partial, any progress made by the backup will be lost when the first partial expires.

After the initial backup completes, typical filesystem backup performance for an Avamar client is about 1 million files per hour. You will likely have to do some tuning to get this client to complete on a regular basis, even doing incrementals. The speed of an incremental Avamar backup is generally limited by the disk performance of the client itself but it's important to run some performance testing to isolate the bottleneck before taking corrective action. If we're being limited by the network performance, obviously we don't want to try to tweak disk performance first.

The support team L2s from the client teams have a good deal of experience with performance tuning and can work with you to run some testing. The tests that are normally run are:

• An iperf test to measure raw network throughput between client and server
• A "randchunk" test, which generates a set of random chunks and sends them to the grid in order to test network backup performance
• A "degenerate" test which, as I mentioned previously, processes the filesystem and discards the results in order to measure disk I/O performance
• OS performance monitoring to ensure we are not being bottlenecked by system resource availability (CPU cycles, memory, etc.)

Edit -- 2013-08-06: The behaviour for partial backups changed in Avamar 6.1. More information in the following forums thread:

Re: Garbage collection does not reclaim expected amount of space