connolly
3 Argentium

Very poor Avamar replication performance

Still working on replication issues for a customer, but now focusing on one particular client that is backing up using the Avamar file system client agent (Avamar version 7.5.1-101). Have been trying to replicate outstanding backups for the client, but lately, the replication performance has been very poor:

2019-03-24 08:49:09 avtar Info <7925>: Restored 9.278 KB from selection(s) with 9.278 KB in 1 files
2019-03-24 08:49:09 avtar Info <6090>: Restored 9.278 KB in 1197.28 minutes: 476 bytes/hour (0 files/hour)
2019-03-24 08:49:09 avtar Info <7883>: Finished at 2019-03-24 08:49:09 EDT, Elapsed time: 0019h:57m:16s

FWIW, the backups for this client are around 1.6TB each - however, I am seeing other file system clients complete replication of 800GB of data in 10 hours, so even if the full 1.6TB couldn't be replicated in the 20 hour window I have set up, I would at least expect to see better progress than "bytes an hour".

I have managed to replicate some backups for this client in the past, so I can't see this being a case of "no existing data so every chunk has to be verified" - and those backups have taken some time, which I accepted because those were the initial replication iterations. But once some client data has been replicated, I would expect to see better performance for the replication of a client's backups.

Again, other file system clients seem to be replicating fine, and VM backups are also replicating fine. It's just this particular client that is having these issues. 

I could certainly just "wait and see", but when replication has run twice for 20 hours (so 40 hours total), and the logs tell me it hasn't even moved 10K of data, there's something wrong.

All comments/feedback appreciated - thanks.

 

Reply
1 Reply
Highlighted
pmrossi
1 Copper

Re: Very poor Avamar replication performance

Hi,

did you check network performance and bandwidth using the iperf command on both sides.

Here what is suggested from the avamar manual:

Network bandwidth recommendations

Before you add a Data Domain system to an Avamar configuration, ensure that the

infrastructure provides adequate network bandwidth for backups and Avamar

maintenance activities.

Network bandwidth in an Avamar configuration has the most impact on Avamar client

backups to a Data Domain system and Avamar server maintenance activities. The

process that sends Avamar client metadata to the Avamar server has less impact on

the network bandwidth.

To measure the network bandwidth between the Avamar server and Data Domain

system, use the iperf utility. The iperf utility is available on the Avamar server, on

the Data Domain system, and from the Internet:

l On an Avamar server, the Linux operating system includes the iperf utility

in /usr/local/avamar/bin.

l On a Data Domain system, the Data Domain Operating System (DD OS) includes

the iperf utility.

l For Avamar clients, download the iperf utility from the Internet.

Use the iperf utility to test the network bandwidth

For the most comprehensive results, run the iperf utility in server mode on the

Avamar server and in client mode on the Data Domain system. Then run the iperf

utility in server mode on the Data Domain system and in client mode on the Avamar

server. Run the iperf utility several times to verify the consistency of the results.

Procedure

1. Run the iperf utility in server mode on the Avamar server by typing the

following command:

iperf -s -w 256k

2. Run the iperf utility in client mode on the Avamar server by typing the

following command:

iperf -c iperf-server-name -w 256k

where iperf-server-name is the iperf server.

To view statistics for every second, add the -i 1 option.

3. Run the iperf utility in server mode on the Data Domain system by typing the

following command:

net iperf server window-size 256K

4. Run the iperf utility in client mode on the Data Domain system by typing the

following command:

net iperf client iperf-server-name window-size 256K

where iperf-server-name is the iperf server.

To view statistics for every second, add the interval 1 option.

bye

 

Reply