Avamar: How to enable and interpret avtar COMSTATS logging to diagnose communication issues

Communication statistics (COMSTATS) output is useful for diagnosing avtar to Avamar Server communication issues. This information can help to prove whether a communication bottleneck is occurring at the Avamar server.

Comstats provides statistics on data that is transferred between the Avamar client and server. It cannot be used to diagnose issues where a backup is sent to Data Domain.

This is because where backups are sent to Data Domain, only metadata is sent to the Avamar server. Thie size of metadata is tiny relative to the backup data which avtar sends to Data Domain.


How to apply "COMSTATS" output to avtar logging:

COMSTATS adds many lines to the logs, so it should be used sparingly.
Only use it for performance issue scenarios where Avamar server response time or client queue time is suspected to be a cause.
 

1. Edit or create a text file on the client system named avtar.cmd. The file must be placed in the Avamar installation directory (mentioned earlier in this article).
2. Use a text editor to create the file and add these lines before saving the file.

--comstats

It is also possible to enable comstats logging through the user interface for a single backup or enable it for an entire dataset. 
 

• Select the 'More Options' button.
• Choose 'Advanced Options'. 
• Mark the 'Enable Debugging Messages' checkbox while performing a single backup or adding the --comstats=true option to a dataset. 
• This is documented in the Avamar System Administrator guide.

General information:
 

• The client-side TCP Socket buffer is filled with "maxpending" requests to the Avamar server.
• Avtar monitors Avamar server response time, and, by default, adjusts "maxpending" to match the Avamar server response rate.
• If avtar does not receive an Avamar server response in 60 s, it resends the request. This is an application-level retry, not TCP. In this case, if the Avamar server has already replied (ACK), avtar will receive a second ACK, and reports UNECESSARY.
• WorkQ0 keeps the responses from the Avamar server. This number should always be a low, single digit. If not, it indicates a client-side problem.

Anatomy of a COMSTATS line

If --comstats is set, avtar outputs COMSTATS messages every second into the log.

avtar Stats <0000>: 2010-10-11 06:18:20 COMSTATS:0 sent= 84 recv[0]= 84 pending= 1/ 5 int= 0/50 send= 0 bytes= 9408+ 17711 sleepms= 0 delay=(0.008 [0.000..0.210] sd=0.030 n= 53) (0.022 [0.000..0.324] sd=0.066 n= 31)

COLOR CODED EXPLANATION OF THE LOG LINE:

COMSTATS:0 --> The "0" is the DPN index and refers to the GSAN. It is only useful for replication jobs. 0=Source, 1=Target.

sent= 84 recv[0]= 84 --> 84 requests were sent and 84 responses received from the server since the previous comstat message.

pending= 1/ 5 --> avtar has 1 message awaiting server response out of a queue max depth of 5=<current value of maxpending>. 
Large PENDING values indicate slow server response. Avtar will try to increase maxpending to match.

int= 0/50 --> Internal Pending messages / <Avtar FLAG value of MAXPENDING>

send= 0 --> Send queue length

bytes= 9408+ 17711 --> "Message" bytes is 9409; (Xmit-bytes + Rcv-bytes) is 17711
"Message" bytes are defined as GETHASH + ADDHASH + ADDCOMP.

sleepms= 0 --> Throttle delay = 0

The following line shows statistics for Avamar server response and delay times in the interval since the previous message:

• Average delay was 0.008 s.
• [Range 0 to 0.21 sec].
• Standard deviation = 0.03.
• n=<Number of messages since last "TUNE">.

The first set of numbers is for delay of "Non-data" type messages. The second set of data is for delay of "Data" type messages.

• MSG_CMD_LOGIN - Log in to the Server with provided credentials.
• TICKETLOGIN - log in to the Server with the provided Session Ticket.
• HASH_IS_PRESENT (hash)- Is the hash already stored on the server?
• MSG_CMD_MOD_BACKUP_LIST (time, label, size, expiration, root hash, LSTR_ADD) -- Record a new backup root hash.

• ADD_HASH_DATA (hash, type, data)- Add new data of kind type with address hash.
• GET_HASH_DATA (hash)- Retrieve data for hash returning type and data.

More information about MAXPENDING