Avamar: Avagent logs show "CR_ERROR" while registering or during backup

Scenario 1 - Client registration fails:

avagent.log:

2018/02/20-07:42:30.26300 [agent_main]  agent_message::lowrecv() Unable to recv message size!

2018/02/20-07:42:30.26300 [agent_main]  agent_message::recvcapsule capsule: server_avail:1, ret.size=0
2018/02/20-07:42:30.26300 [agent_main]  agent_message::receive_message after recv
2018-02-20 13:12:30 avagent Error <5061>: No response to message
2018/02/20-07:42:30.26300 [agent_main]  agent_message::hangup closing socket
2018/02/20-07:42:30.26300 [agent_main]  agent_message::hangup closed connection
2018/02/20-07:42:30.26399 [agent_main]  agent_commands::check_mc_response received an bad status code 'CR_ERROR'
2018/02/20-07:42:30.26399 [agent_main]  agent_commands::check_mc_response ret=9. 

-- Or --

2019-01-28 12:52:53 avagent Info <6601>: Hostnamelist: client1.avamar.com,171.16.100.200,client1.avamar.com,127.0.0.1
2019-01-28 12:52:54 avagent Info <5472>: Registering with the MCS (10.18.100.200)
2019-01-28 12:52:54 avagent Error <5320>: Received from MCS#100003:client certificate verification failed 100.168.100.20.
2019-01-28 12:52:54 avagent Error <5061>: No response to message
2019-01-28 12:52:54 avagent Error <6210>: Unable to register '/clients\client1' with MCS at 10.18.100.200:30001
2019-01-28 12:52:54 avagent Error <6164>: Registration error: Client ID(CID) missing from response. (< msgver="1" />
2019-01-28 12:52:54 avagent Error <5036>: Unable to register CR_ERROR

Scenario 2 - Registration completes, but the backups get stuck in "Waiting client" or "Timed-Out End" with no data transfer:

avagent.log:

2017/06/14-15:17:43.99699 [agent_main]  agent_message::receive_message message sent, recv ack, timeout 300
2017/06/14-15:18:02.95099 [agent_main]  agent_message::lowrecv() Unable to recv message size!
2017/06/14-15:18:02.95099 [agent_main]  agent_message::recvcapsule capsule: server_avail:1, ret.size=0
2017/06/14-15:18:02.95099 [agent_main]  agent_message::receive_message after recv
2017-06-14 08:18:02 avagent Error <5061>: No response to message
2017/06/14-15:18:02.95099 [agent_main]  agent_message::hangup closing socket
2017/06/14-15:18:02.95099 [agent_main]  agent_message::hangup closed connection
2017-06-14 08:18:02 avagent Error <5038>: Error sending message CR_ERROR    

-- Or --

avagent Info <7502>: Registration of client /Data_Center1/SharePoint2013/prod_ext/client2.emc.local with MCS avamar-01emc.local:28001 successful.
2017-06-14 18:41:57 avagent Info <5928>: Registration of plugin 3015 VSS successful.
2017-06-14 18:41:57 avagent Info <5928>: Registration of plugin 3023 update successful.
2017-06-14 18:41:57 avagent Info <5928>: Registration of plugin 3001 Windows successful.
2017-06-14 18:41:57 avagent Info <5619>: Registration of client and plugins complete.
2017-06-14 18:41:57 avagent Info <5964>: Requesting work from avamar-01.emc.local
2017-06-14 18:42:16 avagent Error <5061>: No response to message 2017-06-14 18:42:16 avagent Error <5038>: Error sending message CR_ERROR

If the avagent log reports "No response to message" or "CR_ERROR during the initial handshake, there is a communication issue blocking the send/ack messages between the Avamar grid and the client.

The following steps can be used to assist with finding bottlenecks:

1. Check that all the required Avamar ports are open in the firewall and can send and receive data.
(Avamar server to client and client to Avamar server)

2. Check if there is any antivirus or malware detection software blocking the data on Avamar ports.

3. If the client is behind a NAT, verify that route tables are set up correctly.

4. If the same error is seen on all clients, ensure that no changes were made to the Avamar network.

5. Engage the network team to trace the network traffic from the failing client to the Avamar grid and check for the packet drops in the network.

6. If no issues can be found, collect tcpdump and Wireshark or Netmon information and Create a Service Request.

• Information about how to collect Wireshark can be found in the following article: Avamar: Troubleshoot network issues causing Windows backup performance to be very slow
• Wireshark general documentation and downloads: www.wireshark.org (External link)
• How to use Wireshark to capture, filter and inspect packets (External link)

• https://linuxtechlab.com/learn-use-tcpdump-command-examples  (External link)