Unsolved
This post is more than 5 years old
2 Intern
•
132 Posts
0
1150
Avamar f_cache2 size discrepancy between servers
Working with a customer on a performance issue and we ran across something that I am not sure is related, but did cause me to wonder.
The customer has 2x servers - one is their "active" file server at their primary location, and the second is a "alternate version" of the active server that is kept in sync via a product called "DoubleTake". So at any given time, each server effectively has "the same amount" of data, files, directories, etc. on them.
At present, we are backing up the active file server to an Avamar/DD at the primary location - and we are also backing up the "alternate" file server to an Avamar/DD at the secondary location. FWIW, the performance issue we are seeing is that the "alternate" file server backs up "twice as fast" as the active file server does - or more accurately, since the amount of changed data is not actually that much, the "alternate" file server is "processing" the 20 million files that need to be backed up twice as fast as the active file server is. So we are looking at various aspects of what might be causing one server to scan through all those files faster than another.
Which brings me to the f_cache2 files - which I'm not sure have anything to do with this, but did look odd to me.
On the active file server, from what I can tell from the Avamar session logs and elsewhere, the f_cache2 file is just under 9GB, and there are around "980 pages in all backups in cache".
On the alternate file server, from what I can tell from the Avamar session logs and elsewhere, the f_cache2 file is just under 47GB, and there are around "550 pages in all backups in cache".
Can someone help me understand how one f_cache file is 5 times the size of another, but seems to contain only half the amount of "pages", which as far as I understood were all the same size?
The only thing that I can think of is that we did have some issues in the past on the active file server that caused the Avamar cache files to be recreated a couple of times - so those files have not "been around" and been active for the same amount of backups that the cache files have on the alternate server (which as far as I know have not been recreated and have just been growing since the agent was installed on that server).
All comments/feedback appreciated - thanks.
Avamar Exorcist
462 Posts
0
March 28th, 2018 01:00
Regarding the first point (and without the benefit of primary information), I’d imagine that if the ‘alternate’ server is merely a receptacle for receiving and backing up data from the primary location then it is most likely going to be experiencing less contention than the primary, which, being an active file server, will have clients who wish to read and write data.
Given the relatively large number of files, the backup activity is likely going to encroach on file serving activity during the business day and hamper the performance of both .
In addition to this you’re also going to have the DoubleTake activity occurring and presumably creating an extra disk read work to copy the data written to the primary client over to the remote / alternate client. Since this will cause extra read contention on the primary and write contention to the alternate client’s storage, the backup (read) operations on the primary might be more affected by that particular workload than the alternate client.
There are lots of "it depends" factors of course..
Regarding the relative file caches, if you were looking for commentary on those it would really require primary information (log files).
connolly
2 Intern
2 Intern
•
132 Posts
0
March 28th, 2018 06:00
Speaking to the performance aspect - the one thing that the customer keeps bringing up is that if there is contention for the data on the active file server, they are not hearing about it - and FWIW, due to the amount of time the active file server takes to back up, we have been running backups during "working hours" and they still are not hearing about any performance issues.
Speaking to the f_cache2 aspect, I'd love to post all the log file information - but given that folks rarely respond to a lot of posts on this forum anyway, and given that there are about 10 log files for each server per backup iteration (because that is how we have the backups structured, for various other reasons), I have my doubts as to how much more information they would provide and whether anyone would even take the time to look at so much data in this kind of more informal forum (we're actually at the point of logging an SR because we've gone through pretty much everything in the KBs that talk about troubleshooting performance issues and we haven't found anything). Also, I may be missing something somewhere, but I don't believe that the session logs have info about the total size of the f_cache2 files - unless you mean some other "primary log files", in which case, please specify which ones.
And I've already posted summary info from the session log files, but if you really want more, tell me which parts and I'll post those. But what I have posted is summary info of the entries like this (both jobs effectively running outside of "business hours" as far as I know):
Primary server, f_cache2 file is around 9GB, backup for a directory set of a drive that is prohibitively large to back up in one job:
2018-03-28 05:18:25 avtar Info <5156>: Backup #182 timestamp 2018-03-28 05:18:26, 1,093,669 files, 109,853 folders, 869.3 GB (4,394 files, 463.4 MB, 0.05% new)
2018-03-28 05:18:25 avtar Info <7539>: Label "CC_Test_0345_WeFrSu-CC Test BPFILE alt tgt E T_Z-1522226700007", scheduled to expire after 05/28/18 (2018-05-28 08:45:00 UTC), daily backup
2018-03-28 05:18:25 avtar Info <6083>: Backed-up 869.3 GB in 84.47 minutes: 617 GB/hour (776,831 files/hour)
2018-03-28 05:18:25 avtar Info <5587>: Updating cache files in S:\Program Files\avs\var
2018-03-28 05:18:25 avtar Info <5069>: - Writing cache file "S:\Program Files\avs\var\f_cache2.dat"
2018-03-28 05:18:25 avtar Info <18895>: Cache update complete S:\Program Files\avs\var\f_cache2.dat (770 pages in all backups in cache)
2018-03-28 05:18:26 avtar Info <5069>: - Writing cache file "S:\Program Files\avs\var\p_cache.dat"
2018-03-28 05:18:28 avtar Info <5546>: Cache update complete S:\Program Files\avs\var\p_cache.dat (768.0 MiB of 2047 MiB max)
2018-03-28 05:18:36 avtar Info <7883>: Finished at 2018-03-28 05:18:36 Central Daylight Time, Elapsed time: 0001h:24m:38s
Secondary server, f_cache2 file is around 47GB, backup for the same directory set:
2018-03-28 06:36:18 avtar Info <5156>: Backup #270 timestamp 2018-03-28 06:34:52, 1,096,061 files, 110,239 folders, 867.9 GB (4,450 files, 490.4 MB, 0.06% new)
2018-03-28 06:36:18 avtar Info <7539>: Label "CC Test CGfile1 E Groups T_Z Schedule 0535-CC Test CGfile1 E Groups T_Z Group-1522233300006", scheduled to expire after 05/27/18 (2018-05-27 10:35:00 UTC), daily backup
2018-03-28 06:36:18 avtar Info <6083>: Backed-up 867.9 GB in 34.51 minutes: 1,509 GB/hour (1,905,581 files/hour)
2018-03-28 06:36:18 avtar Info <5587>: Updating cache files in C:\Program Files\avs\var
2018-03-28 06:36:18 avtar Info <5069>: - Writing cache file "C:\Program Files\avs\var\f_cache2.dat"
2018-03-28 06:36:18 avtar Info <18895>: Cache update complete C:\Program Files\avs\var\f_cache2.dat (544 pages in all backups in cache)
2018-03-28 06:36:18 avtar Info <5069>: - Writing cache file "C:\Program Files\avs\var\p_cache.dat"
2018-03-28 06:36:20 avtar Info <5546>: Cache update complete C:\Program Files\avs\var\p_cache.dat (768.0 MiB of 2047 MiB max)
2018-03-28 06:36:25 avtar Info <7883>: Finished at 2018-03-28 06:36:25 Central Daylight Time, Elapsed time: 0000h:34m:37s
Also, FWIW, here is the log info I originally referenced in my first post, for the same directory set on the primary active file server - FYI, the info below is from a backup over the weekend before we moved the f_cache2 files to another drive on the server to see if access to the cache files was affecting the performance at all; now I'm also curious why this backup references 956 pages and the other one references 770 pages? unless the number of pages referenced doesn't have anything to do with the actual size of the f_Cache2 file, but more to do with how many pages were needed during the backup session - that seems like something that could be commented on by the right level of SME?)
2018-03-26 16:51:52 avtar Info <5156>: Backup #169 timestamp 2018-03-26 16:51:52, 1,093,438 files, 109,835 folders, 869.2 GB (520 files, 130.4 MB, 0.01% new)
2018-03-26 16:51:52 avtar Info <7539>: Label "CC Test BPFILE Mon tgt E T_Z-1522096053323", scheduled to expire after 05/26/18 (2018-05-26 20:27:33 UTC), daily,weekly,monthly,yearly backup
2018-03-26 16:51:52 avtar Info <6083>: Backed-up 869.2 GB in 84.31 minutes: 619 GB/hour (778,154 files/hour)
2018-03-26 16:51:52 avtar Info <5587>: Updating cache files in V:\Program Files\avs\var
2018-03-26 16:51:52 avtar Info <5069>: - Writing cache file "V:\Program Files\avs\var\f_cache2.dat"
2018-03-26 16:51:52 avtar Info <18895>: Cache update complete V:\Program Files\avs\var\f_cache2.dat (956 pages in all backups in cache)
2018-03-26 16:51:52 avtar Info <5069>: - Writing cache file "V:\Program Files\avs\var\p_cache.dat"
2018-03-26 16:51:55 avtar Info <5546>: Cache update complete V:\Program Files\avs\var\p_cache.dat (768.0 MiB of 2047 MiB max)
2018-03-26 16:52:01 avtar Info <7883>: Finished at 2018-03-26 16:52:01 Central Daylight Time, Elapsed time: 0001h:24m:27s