Avamar: How to control Avtar Memory Utilization by adjusting the Hash and File caches

Information that addresses avtar memory utilization:

There are three major contributors to the overall memory use on the client when avtar runs:

1. The avtar binary itself requires memory when performing a backup (aka snapup). The amount of memory consumed by the avtar process is generally in the range of 20 MB to 30 MB. This amount depends on which OS the client is running, and also fluctuates during the backup depending on the structure of the files that are being snapped up by avtar.

2. The avtar process loads two cache files into memory. These two cache files exist in the Avamar var directory. The first of these is the file cache (f_cache.dat). The file cache stores the hashes of the file attributes, and is used to quickly identify which files have previously been backed up to the Axion (aka Avamar) server. The presence of the file cache is one of the significant reasons why subsequent Axion backups (that is, after the initial backup) are generally very fast. Typically, the file cache screens out approximately 95% to 98% of the files. If the file cache is deleted, not used, or is undersized, then every file that is not a "hit" in the file cache must be opened, read, chunked, and hashed before the avtar process finds that the hashes were previously sent to the Axion server. If a file has a "hit" in the file cache, then the file is never accessed, saving significant time and CPU.

By default, the file cache can consume up to 1/8th of the physical RAM on the Axion client. For example, if the client has 4 GB of RAM, the file cache is limited to a maximum of 512 MB. The file cache doubles in size each time it must grow. The current file cache sizes are 5.5 MB, 11 MB, 22 MB, 44 MB, 88 MB, 176 MB, 352 MB, 704 MB, and so on. In this example where a client has 4 GB of RAM, the maximum size of the file cache is limited to 352 MB.

3. The second cache is the hash cache (p_cache.dat). The hash cache stores the hashes of the chunks that have been sent to the Axion server. As noted above, the avtar process gains the most efficiency by ensuring that the file cache is sized adequately. If the hash cache is deleted, not used, or is undersized then if the avtar process finds that a hash of a chunk that is not contained in the hash cache, the avtar process queries the Axion server for the presence of the hash.

By default, the hash cache could consume up to 1/16th of the physical RAM on the Axion client. In the previous example of the client with 4 GB of RAM, the hash cache is limited to a maximum of 256 MB. The hash cache also doubles in size each time it must grow. The current hash cache sizes are 24 MB, 48 MB, 96 MB, 192 MB, 384 MB, 768 MB, and so on. In this example where a client has 4 GB of RAM, the maximum size of the hash cache is limited to 194 MB.

From the above it can be deduced that if a client has more than 1/2 GB of RAM, the file and hash caches dominate the overall memory utilization. This is because both caches are read completely into memory at the start of the avtar backup. Also, seen above, by default, the overall memory used by avtar is limited to approximately 3/16th of the physical RAM on the Axion client.
 

The sizes of the file and hash caches are printed near the beginning of the avtar logs.

For example:

avtar Info <5573>: - Loaded cache file C:\ProgramFiles\Avamar\var\f_cache.dat (5767712 bytes)
avtar Info <5573>: - Loaded cache file C:\ProgramFiles\Avamar\var\p_cache.dat (25166368 bytes)

Here, the file cache is 5.5 MB, and the hash cache is 24 MB.

At the end of the avtar log, the following set of messages will be seen:

avtar Info <5587>: Updating cache files in C:\ProgramFiles\Avamar\var
avtar Info <5069>: - Writing cache file C:\ProgramFiles\Avamar\var\f_cache.dat
avtar Info <5546>: - Cache update complete C:\ProgramFiles\Avamar\var\f_cache.dat (5.5MB of 63MB max)
avtar Stats <6151>: File cache: 131072 entries,added/updated 140, booted 0
avtar Info <5069>: - Writing cache file C:\ProgramFiles\Avamar\var\p_cache.dat
avtar Info <5546>: - Cache update complete C:\ProgramFiles\Avamar\var\p_cache.dat (24.0MB of 31MB max)
avtar Stats <6152>: Hash cache: 1048576 entries,added/updated 1091, booted 0

From these lines, it indicates that the file cache has room to grow, but the hash cache is at its maximum. If either cache is undersized, the "booted" value will be non-zero and the log will include a warning that the cache is undersized. This is important to note because the size of the cache has a huge influence on the overall performance.

The default cache limits caches can be overridden using the following two flags:

--filecachemax=n

Where n is the number of MB, or a fraction (negative value = fraction of RAM)

Default value: 

--filecachemax=-8

Example limiting the file cache to a size of 100 MB:

--filecachemax=100

--hashcachemax=n

Where n is the number of MB, or a fraction (negative value = fraction of RAM)

Default value:

--hashcachemax=-16

Another implementation consideration is that if the cache size is limited on a set of clients, then the appropriate parameters should be added to each client's avtar.cmd file so that it is applied every time the client performs a backup, even manual backups. If no flag is in the avtar.cmd file, and a manual backup is performed without the appropriate options, the file, or hash cache could grow back to their default limits.

To tune performance, sometimes it is necessary to increase the cache sizes from the default values.

These conditions could exist in the following two diametrically opposed cases:

1. If the client has millions of small files, then the file cache may require an increase from the default size. The general guideline is that, for every million files on the Axion client, the client should have 512 MB of physical RAM. 

If a client has 1 million files, a minimum of 20 MB is required to store the file hashes for a single backup (each hash is 20 bytes). Since the file hashes for several backups must be stored, more is required meaning it will grow to approximately 44 MB. As 1/8th of the physical 512 MB of RAM is used by default, the cache is allowed to grow to a limit of 64 MB, so the value of 1/8th of RAM for the file cache is adequate.

2. If the client has a few large files, then the default of 1/16th for the hash cache is likely insufficient.  

For example, if there is a 200 GB database, then up to 10 million hashes must be cached. As each hash is 20 bytes, a minimum of 200 MB is required for a single backup.

The next increment available is 384 MB, therefore if this client has 4 GB of RAM, the hash cache must be allowed to grow up to 1/8th of the RAM. If the default of 1/16th of the RAM is used, then the hash cache is limited to 192 MB, and the hash cache will be undersized. In the case of databases, as few files are being backed up, the file cache is considerably smaller, so the net memory utilization is still approximately 1/8th to 3/16th of the RAM.

To address this issue more systematically in any given environment, determine from the avtar logs how large the file and hash caches are.
If the combined size of these caches must be smaller, then there are ways to limit the sizes.