Avamar: Troubleshooting network issue that cause slow Windows backup performance

Issue:

The backup performance is slow. Network issues suspected.

Troubleshooting performed:

The backup logs show the following events (indicating a network problem):

2014-04-19 05:20:23 avtar Info <5387>: Network error occurred: Rebuilding connection 0
2014-04-19 09:39:40 avtar Error <5774>: Internal Error: backtree: ADD_HASH timeout error 

The long delay below indicates that the network bandwidth or throughput between the client and server must be checked:
(The "usual" acceptable delay is ~0.008, however here there is a high value of 0.141 seconds.

avtar Stats <15102>: 2023-04-19 07:02:47 COMSTATS:0 sent=  34 recv[0]=  37  pending=  3/ 10 int= 0/50 send=  0 bytes=  3976+   13518 sleepms= 0 delay=(0.141 [0.001..0.237] sd=0.101 n=  34) (0.021 [0.007..0.030] sd=0.010 n=   3) 

Bandwidth and throughput testing between the client and server:

Iperf test:

1. Start a test backup on the Windows client.

2. Log in to the Avamar Utility Node as admin.

3. Run the following command (leave it running):

iperf -s -i 1

4. Log in to the windows client.

5. Download the Iperf tool from the following link: http://linhost.info/2010/02/iperf-on-windows/ (External Link)

6. Open a command prompt on the windows client, change to the iperf software location, and run the command:

iperf -c <ip address of the avamar server> -i1 

7. Leave this command running for a while and then terminate it.

8. Collect the generated logs from both the Avamar and the windows client to measure the available bandwidth.

Wireshark test with TCPdump:
(Required to measure the network throughput and stability if the iperf results show no issue)

1. Download the Wireshark software from the following link: http://www.wireshark.org/download.html (External Link)

2. Install the Wireshark software on the Avamar client.

3. Start a test backup on the Windows client.

4. Start a Wireshark TCPdump capture while applying the following filter, modifying the "< >" values with proper data:

ip.addr==<ipaddress of Avamar Utility node> || ip.addr==<ipaddress of Avamar storage node#1> || ip.addr==<ipaddress of Avamar storage node#2> || ... || ip.addr==<ipaddress of Avamar storage node#n> || tcp.port==28001 || tcp.port==27000 || tcp.port==28002 || tcp.port==29000

5. Log in to the Avamar Utility Node as admin, elevate to root, and load the Open ssh keys per Avamar: How to Log in to an Avamar Server and Load Various Keys.

6. Run the following command:

mapall --noerror --all --parallel --bg --user=root 'tcpdump -w /root/tcpdump.pcap host <ip address of the client>'

7. Allow the backup to run for one hour, then stop it, and save the Wireshark TCPdump from the client (as a .pcap file).

8. Terminate the TCPdump on Avamar using the following command:

mapall --noerror --all --user=root 'pkill tcpdump' 

9. Run the following commands to get the generated tcpdump.pcap file from all storage nodes:

mkdir /tmp/tcpdumplogs

cd /tmp/tcpdumplogs 

mapall --noerror --all --user=root get '/root/tcpdump.pcap'

10. Retrieve the logs from the Avamar Utility Node /tmp/tcpdumplogs directory using a program such as WinSCP.

11. Provide the collected logs to the network admin for analysis.

Check for TCP retransmissions on Avamar ports.

• Wireshark general documentation: www.wireshark.org (External Link)
• TCPdump official man page: www.tcpdump.org/tcpdump_man.html (External Link)
• Avamar: Troubleshooting slow backup performance
• Avamar: Optimizing performance when backing up Outlook archive .pst files