Optimize for large file count

May I know if backup is failing or its unable to complete backup within your specified backup time limit ?

If backup is failing then what is the error ?

If its unable to complete backup then did you try to increase backup time limit ?

Regards,

Pawan

The file cache must be at least N x 44 MB, where N is the number of millions of files in the backup.

In your case, N is 2.5 so you should have at least 2.5 * 44 = 110 MB file cache.

The file cache doubles in size each time it needs to increase. The current file cache sizes are in megabytes: 5.5 MB, 11 MB, 22 MB, 44 MB, 88 MB, 176 MB, 352 MB, 704 MB, and 1,408 MB.

So you have to set file cache at least 176 MB.

But By default, value of file cache is 1/8 of total physical RAM. That means you already have file cache much higher than minimum value required for 2.5 million files.

I don’t think so you need to make any changes in file cache. But make sure you have file cache at default value.

Regards,

Pawan

This server has 6GB of memory in it. I would think something like the fcache at 1GB and the pcache at 512MB?

Thank you kindly,

Steven

Hello Steven,

What is the total physical RAM size on your client ?

If we have huge number of files, we should keep file cache higher and hash cache lower. The sum of these two should not be more than 25% of total physical RAM.

Regards,

Pawan

That's wonderful! Is there a way to set this in the avtar.cmd file?

I would recommend taking a look at the logs. If the caches are undersized, the logs will have messages about the cache being unable to grow and hashes being booted from the cache. If you do not see these messages, adjusting the cache sizing will not make any difference.

If the client is running version 6.0 or older and there are directories with large numbers of files in them, I would recommend upgrading the client to version 6.1 or newer since there is improved handling for large directories in newer versions of the client. This can make a significant performance difference.

Just for additional information (and for users who run database backup)

We should keep hash cache value higher and file cache lower for database backup.  Typically, the hash cache must be a minimum of N MB, where N is the size of the database being backed up in gigabytes. The current hash cache sizes are in megabytes: 24 MB, 48 MB, 96 MB, 192 MB, 384 MB, 768 MB, and so forth. If we have 4 GB physical RAM and backup 600 GB data base, we should have at least 600 MB hash cache, but we can not set 600 MB hash cache and we have to set it to 768 MB.

So first assign hash cache to required value and later assign file cache from total available RAM but the sum of these two should not be more than 1/4th of RAM.

Regards,

Pawan

The job was running for an extremely long time, sometimes around 24 hours. I will implement these changes and let you know how it effects the job.

Thanks,

Steven

I agree with IAN, logs may provide additional information to understand the issue.

Regards,

Pawan

Thanks for the feedback Ian, we're using Avamar 7.0. I'll take a look at the job log tonight after the job runs and update you on the status.

Thanks for all the great input,
Steven

So this server is taking about 8 hours to complete. They have a directory that has 2.8 million small files, the server it self is less than 70GB of data. Here is some of the log output:

2014-03-11 02:44:07 avtar Info <5069>: - Writing cache file "C:\Program Files\avs\var\f_cache.dat"

2014-03-11 02:44:12 avtar Info <5546>: Cache update complete C:\Program Files\avs\var\f_cache.dat (352.0MB of 1023MB max)

2014-03-11 02:53:29 avtar Info <8688>: Status 2014-03-11 02:53:29, 2,412,043 files, 63,870 folders, 62.11 GB (12,986 files, 213.1 MB, 0.34% new) 418MB 31% CPU E:\Signatures\81\2013\04\19\

2014-03-11 03:06:47 avtar Info <5548>: 2014-03-11 03:06:47 Eastern Daylight Time: Automatic cache flush of C:\Program Files\avs\var\f_cache.dat

2014-03-11 03:06:47 avtar Info <5069>: - Writing cache file "C:\Program Files\avs\var\f_cache.dat"

2014-03-11 03:06:51 avtar Info <5546>: Cache update complete C:\Program Files\avs\var\f_cache.dat (352.0MB of 1023MB max)

2014-03-11 03:08:42 avtar Info <8688>: Status 2014-03-11 03:08:42, 2,497,833 files, 65,530 folders, 62.55 GB (13,145 files, 213.8 MB, 0.33% new) 419MB 23% CPU E:\Signatures\83\2011\02\04\

2014-03-11 03:23:55 avtar Info <8688>: Status 2014-03-11 03:23:55, 2,582,345 files, 66,446 folders, 63.00 GB (13,145 files, 213.8 MB, 0.33% new) 419MB 21% CPU E:\Signatures\83\2013\12\14\

2014-03-11 03:39:07 avtar Info <8688>: Status 2014-03-11 03:39:07, 2,663,996 files, 67,705 folders, 63.42 GB (13,454 files, 215.1 MB, 0.33% new) 419MB 21% CPU E:\Signatures\84\2013\11\29\

2014-03-11 03:41:11 avtar Info <5548>: 2014-03-11 03:41:11 Eastern Daylight Time: Automatic cache flush of C:\Program Files\avs\var\f_cache.dat

2014-03-11 03:41:11 avtar Info <5069>: - Writing cache file "C:\Program Files\avs\var\f_cache.dat"

2014-03-11 03:41:15 avtar Info <5546>: Cache update complete C:\Program Files\avs\var\f_cache.dat (352.0MB of 1023MB max)

2014-03-11 03:54:21 avtar Info <8688>: Status 2014-03-11 03:54:21, 2,734,730 files, 69,392 folders, 63.80 GB (13,970 files, 217.4 MB, 0.33% new) 419MB 19% CPU E:\Signatures\94\2013\03\04\

2014-03-11 04:09:33 avtar Info <8688>: Status 2014-03-11 04:09:33, 2,798,407 files, 70,430 folders, 64.14 GB (14,154 files, 218.2 MB, 0.33% new) 419MB 15% CPU E:\Signatures\97\2013\07\19\

2014-03-11 04:14:29 avtar Info <5548>: 2014-03-11 04:14:29 Eastern Daylight Time: Automatic cache flush of C:\Program Files\avs\var\f_cache.dat

2014-03-11 04:14:29 avtar Info <5069>: - Writing cache file "C:\Program Files\avs\var\f_cache.dat"

2014-03-11 04:14:32 avtar Info <5546>: Cache update complete C:\Program Files\avs\var\f_cache.dat (352.0MB of 1023MB max)

2014-03-11 04:14:38 avtar Info <5163>: Backup complete, wrapping-up session with Server

I know 2.8 million files is a lot to process, but is there any way to speed this up?

Thanks,

Steven

What version of the client is this? If it's running 6.0, upgrading to 6.1 will likely drop the runtime of the backup by several hours because of the improved handling for large directories.

We are on Avamar version 7.0. Any advice for that version?

Thanks,
Steven

Thanks Ian,

  I took a look through ops manager and the system isn't using hardly any resources. Disk I/O is only minimal, CPU, and Memory are still in the green. I'm going to request a reboot of the system and see what happens.

Thanks,

Steven