avamar utilization difference between main site to replication site

Firstly, welcome to the forums, and above all, thank you for being an EMC customer.

Please consider moving this question as-is (no need to recreate) to the proper forum for maximum visibility.  Questions written to the users' own "Discussions" space don't get the same amount of attention and can go unanswered for a long time.

You can do so by selecting "Move" under ACTIONS along the upper-right.  Then search for and select: "Avamar Support Forum".

Avamar Support Forum

Hey,

I just found a solution for your reference. You will get the answer from the solution.

Avamar replication systems - source and target differ in capacity utilization

https://emc--c.na5.visual.force.com/apex/KB_BreakFix_1?id=kA1700000000Tse

Hope this helps.

that appears to be and EMC interal doc that we cannot get to.

looks like a call to support is required to get the info.

Hey,

The KB is for customer, not a EMC internal KB. I will post out the KB contents for your reference.

(Customer)Avamar replication systems - source and target differ in capacity utilisation

Symptoms

There is an Avamar source and an Avamar target system.  Capacity utilisation is higher on one system than the other.

Scope:

This solution is written for Avamar system administrators to help identify why capacity utilisation may differ between a source and a target system.  It is expected that the reader is familiar with the Avamar system administration guide, that they fully understand how to configure replication and that they are competent at performing Linux system administration tasks.

This solution has been written specifically for Avamar versions 4 & 5.

Cause

An Avamar source replicates selected data asynchronously to the target system.  Replication occurs daily.  Provided that replication completes fully, each day we would expect that the data on the source system to be roughly a day 'behind' the data stored on the target system.  This could mean a difference of up to 2 or 3% in capacity, depending on how much data change is occuring on the source system.

Replication is additive - systems are not kept synchronised, therefore, it is possible that the amount of data stored on the source and target systems may differ

Reasons for possible differences between the 'server utilization' values could be:

Physical / logical differences between the actual grids

• There may be a different number of data nodes on the source and target system
  
• The data nodes on the source system may not have the same disk configuration (1TB, 2TB, 3.3TB, and so on) as on the target system
  
• Adequately balanced distribution of stripes across the data nodes within each system (to within 2%)
  
• Storage and parity requirements may differ between Avamar versions.  A difference in the utilisation level may be observed if the grids are not running the same software revision.

Replication configuration

• Backups replicated to the target system may be configured to have a different retention policy than the same backups which exist on the source (refer to the 'expiredelta' flag for more information) or the replicated backups may only cover a particular timespan (for example, the last 4 weeks of backups from the source).
  
• Replication may be configured to replicate only a subset of clients from the source system to the target system if include or exclude settings are used within the replication configuration.
  
• Clients and their associated backups may have been deleted from the source system.   The deletion of a client or of backups on the source will not remove the same backups from the target system.  The backups will remain on the target system until they expire according to their retention settings.
  
• Retention policies may be changed for backups or clients on the source system.  The change in retention policies will affect new backups only.  Any new backups will be replicated to the target and will adhere to the updated retention policy but any backups already existing on the target will retain the retention policy which was applied to them at the time they were replicated to the target.

Other behaviours

• Cross replication may be occurring between multiple Avamar systems.  The source Avamar grid may also be receiving replication data from yet another source grid or the target grid may be the target of more than one Avamar source.
  
• Replication jobs may not be completing reliably and data sent to the target may may be 'lagging behind' the source by multiple days.
  
• Both systems contain the same amount of deduplicated data but the amount of parity overhead on each of them is different.  This could occur in the following scenario.  An Avamar source system is almost full.  A large amount of backups are deleted from the source system to lower its capacity level.  Replication of the deduplicated data then occurs from source to target.  The amount of deduplicated data is the same on both systems but the source system will initially be storing a lot more parity overhead than the target.

Resolution

Investigate the list of 'causes' given above to determine why there is a difference in utilisation between source and target.Below are tips to achieve this.  Note that they may not exhaustive and should be taken as guidelines.

1. Compare the two systems using 'status.dpn'
   
2. For each grid, examine the total size of the physical data partitions on each data node.  This can be obtained using the command 'df -h " grep data'.  If you are familiar with mapall commands you can query all data nodes simultaneously from the utility node.
   
3. For each grid run 'status.dpn' and count the total number of stripes on each data node.  The number of stripes is identified in backets (for example onl:xxx) .  There should be less than 2% difference between the total number of stripes on each data node.
   
4. Run 'status.dpn' and compare the version of Avamar running on each grid.
   
5. Check the replication configuration to see if flags such as '--expiredelta', '--before' or '--after' have been set.
   
6. Check the replication configuration to see if any ' --include' or '--exclude' flags are set. 
   
7. The 'replcnt.sh' script is a utility which can help identify which backups are stored on a source and not on a target system.  The script is available here. ftp://avamar_ftp:anonymous@ftp.avamar.com/software/scripts/replcnt.sh.  Run the script from the Avamar source system(s).  The script includes online help but usage is beyond the scope of this solution.
   
8. Use replcnt.sh and manually check backup retention in the Avamar Administrator GUI.
   
9. Examine all Avamar systems in the environment to understand which systems replicate data and to where.
   
10. Consult the /usr/local/avamar/var/cron/replicate.log on the Avamar source system(s) and examine for errors, failures or time outs which indicate replication has been anything other than successful.
    
11. This is expected behaviour and no action is necessary.  Over time the capacity levels of the two systems will equalise (the system with the higher capacity can be expected to become lower and vice versa).

Hey,

How is the issue now?

Hi,

thank you very much,

the replcnt.sh show me that i have 500 of 590 backups that missing in the main site(source site)

i also saw that even the retention is for 3 month (by the networker), i still have servers in avamar that keep versions more then 5 month.

i want and need delete all the backups that are over 3 month manually.

i will use this command:

expire-snapups --before='2013-09-01' --days=30--domain=/ > do-expire.sh chmod +x do-expire.sh ./do-expire.sh > do-expire.log 2>&1 &

i will run it first in the main site, then in the secound site.

i have 88% utilization in the secound site, i have to free some space soon as possible!

and i will check with the support why retention from the networker not working well.(for some backup with the same policy)

OK. Thank you for your followup.

Please mark my answer as "correct/helpful answer" if you think they are helpful. Thanks.

How can i delete the identify backups that stored on the target but not stored on the source?  (the Replcnt.sh result)

about the "Expire-snapups" it's not work for me, because the backups have no Tag. 

the retention comes from the Networker...

It's working for me:
ftp://avamar_ftp:anonymous@ftp.avamar.com/software/scripts/replcnt.sh

i went to this ftp site to get the 'replcnt' script. it is not there. how can I download?

Thank you

thank you i have it

sorry... what directory path should this be placed in

I usually place it in /home/admin and run it with:

chmod +x replcnt.sh ./replcnt.sh