Avamar: Linux backup fails with "INTERNAL ERROR: <0001> assert error" and "RECV_CRC_ERROR"

This behavior was observed on multiple Linux virtual machines hosted by the same physical host.

A backup fails with the following messages:

2023/02/23-09:06:56.59015 [ms[0]recv]  WARN: <0441> sock::recv RECV_CRC_ERROR 105c838 != 905c838 HASHLIST_IS_PRESENT=406 serial=1185 seq=0 flags=R:N:0 kind=0 rsp=MSG_ERR_NONE
2023/02/23-09:11:56.32974 [avtar]  INTERNAL ERROR: <0001> assert error (false), /local/abs2/work/src/avclient/backtree.cpp line 296
2023/02/23-09:17:33.56426 [ms[0]recv]  WARN: <0441> sock::recv RECV_CRC_ERROR e7ffce6e != efffce6e ADD_COMP_DATA=368 serial=2758 seq=0 flags=R:N:0 kind=0 rsp=MSG_ERR_ENTER_INCOMPLETE

The RECV_CRC_ERROR messages imply an issue transmitting data over the socket used to connect with Avamar.

The Virtual Machine (VM) host had a faulty physical network interface which was assigned to the affected virtual machines, and this resulted in packet loss.

There were no faults logged in the /var/log/messages on the client close to the time of the backup failure.

Check the machine acting as the hypervisor for any hardware issues and correct them.

In this scenario, a faulty physical network interface card on the machine hosting the virtual machines had to be replaced.

After the physical hardware was fixed, backups succeeded without any corrective action on the client itself.