Avamar: Data Domain Integration: Restore from DD-Boost Performance Troubleshooting Tips

1. Avamar Server Settings: Backup and Restore Options

2. The bandwidth between the client and Data Domain

3. Data Domain (DD) Performance

4. Client configuration

Avamar Server Settings: Backup and Restore Options:

By default, data transfer between Avamar clients and Data Domain (DD) has a high level of encryption. Setting the encryption level to none can improve the backup/restore performance three times or more.

To verify the settings of the Data Transfer between Avamar client and Data Domain, examine backup and restore logs for the following messages:

Establishing a connection to the Data Domain system with encryption (Connection mode: A:3 E:2). 

This indicates that encryption is enabled during the backup or recovery session.

Establishing a connection to the Data Domain with basic authentication (Connection mode: A:0 E:0)

This indicates that encryption is not enabled during the backup or recover session.

To verify the default settings of Data Transfer between Avamar clients and Data Domain, review the following values from the Avamar Management Console Service (MCS) preferences file /usr/local/avamar/var/mc/server_data/prefs/mcserver.xml:

• <entry key="default_dd_client_restore_encryption" value=" high " />
• <entry key="default_dd_client_backup_encryption" value="high" />

The following guidelines are recommended for encryption best practices:

• For large backups or restores within the data center, set the encryption to Medium or None to improve the performance.
• For desktop/laptop clients backing up over a WAN, set the encryption to High.

To change the server default behavior, edit the key values by editing the mcserver.xml followed by MCS server restart. The available values are:

• High
• Medium
• None

For backups, the default data transfer mode between client and Data Domain can be modified from the plug-in options. Encryption methods to the Data Domain available options are:

• Default
• High
• Medium
• None

For restore tasks, the default data transfer mode between clients and DD can be modified from "Restore Options," "More Options," "Encryption method from Data Domain System". The available options are:

• Default
• High
• Medium
• None

Bandwidth between client and Data-Domain: 

It is important to find the bandwidth between Avamar client and Data Domain to set the right expectation for backup and restore performance.

1. Identify the DD Boost IP address used by the client during the restore (or backup) operation:

While a restore is running, identify the DD Boost IP by running the following command on the target client:

UNIX or Linux:

netstat -an |grep 2049

Windows:

netstat -an |findstr 2049 

If the backup or restore was started in debug mode, the IP address can be identified during the restore by searching for  port=2049.

2. Identify the DD interface speed (use the DD UI or run the DD commands below from the command line).

Net Show Settings

From the results, identify the port using the assigned IP address. 

System Show Ports

From the results, identify the port speed.

3. Use iPerf or ddpconnchk to verify the bandwidth between the client and DD Boost IP address.

The iPerf tool is available for Linux and Windows. More details about iPerf can be found in the following article: Data Domain: iperf Tool to Test Network Bandwidth -- version 2

The ddpconnchk tool is available for all platforms supported by DD Boost. It is not meant for the bandwidth verification only but for the overall DD Boost read and write performance from the client. More details about ddpconnchk can be found in Data Domain: DDPCONNCHK How to troubleshoot Data Domain DD Boost connectivity and performance.

For ddpconnchk, identify the Avamar storage unit in Data Domain (found by searching for "Setting default storage unit to" in the backup or restore log).

The following are ddpconnchk examples:

./ddpconnchk -s ip_adress -u ddboost_user -p <ddb_user_password> -l avamar-1234567890 -T writeimage -i 1g

This command reports the single stream write and read performance.

./ddpconnchk -s ip_adress -u ddboost_user -p <ddb_user_password> -l avamar-1234567890 -T writeimage -i 1g -n 10 

This command reports 10 streams write and read performance.

Data-Domain Performance:

The ddpconnchk test reports the expected read/write performance from the client. If performance is less than expected (based on the available bandwidth), open a Service Request with the Dell Technologies Data Domain support team to examine the read and write performance.

Client configuration:

If encryption is not enabled between the client and DD and there are no DD read or write performance issues, use the validate option to perform a restore without writing to the client:

Add the parameter --validate to the client avtar.cmd file (located in client var directory).

If the restore performance with the validate option is high, while restore performance is low, then it is likely a client-related issue, such as low memory, slow disk, or file system options.

When investigating restore performance behavior for Avamar backups stored on Data Domain, be sure to gather the information discussed in the following article. This is required in order to escalate such issues to Engineering.

Avamar: Information required by Support when investigating Avamar or Avamar/DataDomain client restore performance issues.