Avamar: Client ID (CID) mismatches with Goav remediation
Summary: Client ID (CID) mismatches and path mismatches occur on Avamar servers when the Global Storage Area Network (GSAN) and Management Console Service Database (MCDB) differ when storing client accounting information. ...
Symptoms
Browsing a client in the Avamar User Interface (AUI) or Avamar Administrator shows the error message: Internal Administrator Client Error.
A client does not show in the AUI.
The browse subdomain checkbox in the AUI returns no results and may possibly display an error.
The following error is reported: 0 ERROR! Exit code 74: Account not found in the Avamar database.
Cause
Overview
Client ID (CID) mismatches and path mismatches occur on Avamar servers when the Global Storage Area Network (GSAN) and Management Console Service Database (MCDB) differ when storing client accounting information.
Client accounting information is stored in two places: GSAN and MCDB.
There are two important things to store for a client.
- The full path of the client:
-
/clients/w11ofsd.lab.com
-
- The unique client id (CID):
-
c7d083d712377955d84a52d2d618b47e33fa1ef
-
A healthy match between GSAN and MCDB would be that the client path and CID match.
In a replication environment, the destination Avamar server has the domain called "/REPLICATE."
This is a GSAN only domain. MCDB does not store information about replicated clients in that domain.
There are a few scenarios where GSAN and MCDB can differ when storing the client accounting information:
- A client path exists in one store but not the other, and the CID does not match.
- A client path exists in one store but not the other, and the CID matches a client with a different path in the other store.
- A client path exists in both stores, but the Client ID does not match.
Manual verification of the client accounting information can be done in GSAN and MCDB.
GSAN
avmgr getl --path=<domain_of_client_path> | grep -i <client_name>
Example:
admin@avamar:~/>: avmgr getl --path=/clients | grep -i w11ofsd.lab.com 1 w11ofsd.lab.com location: c7d083d712377955d84a52d2d618b47e33fa1ef1 pswd: 89e28a1932a8d1b4a4f9e2883e03a9f33f5082d1
From the example, c7d083d712377955d84a52d2d618b47e33fa1ef1 is the CID, and the full path is /clients/w11ofsd.lab.com.
In GSAN, a similar lookup can be done to resolve the client's full path from a CID:
admin@avamar:~/>: avmgr resf --acnt=ref{c7d083d712377955d84a52d2d618b47e33fa1ef1}
1 Request succeeded
/clients/w11ofsd.lab.com c7d083d712377955d84a52d2d618b47e33fa1ef1
MCDB
psql -p 5555 mcdb -U admin -c "select cid,descr from clients where cid='<CID>'"
Example:
admin@avamar:~/>: psql -p 5555 mcdb -U admin -c "select cid,descr from clients where cid='c7d083d712377955d84a52d2d618b47e33fa1ef1'" cid | descr ------------------------------------------+--------------------------------------- c7d083d712377955d84a52d2d618b47e33fa1ef1 | /clients/w11ofsd.lab.com (1 row)
From the example, c7d083d712377955d84a52d2d618b47e33fa1ef1 is the CID, and the full path is /clients/w11ofsd.lab.com.
If there is not a healthy match between the GSAN and MCDB, this is a CID mismatch.
Resolution
The GoAV tool is used to automatically fix CID mismatches.
Download and install the Goav tool.
000192151 | Avamar: Goav tool
Run the following command to check for CID mismatches and generate two report files:
./goav client cid-mismatch check
There is a safe and unsafe report file. Review them both for an explanation.
Example:
admin@avamar:~/>: ./goav client cid-mismatch check =========================================================== GoAv : 1.80 Avamar : 19.10 Date : 19 Aug 2024 16:20 -03 =========================================================== COMMAND : ./goav client cid-mismatch check NOTE: This is not an official tool =========================================================== Client CID & Path Mismatch Script --------------------------------- Load GSAN clients from avmgr cpdb Load MCS clients from mcdb Analyze GSAN mismatches GSAN path /MC_DELETED/DPADatastoreServer_bgVYLK0WONuiFagzVngqBQ2021.09.01.13.38.57 does not exist in MCDB, but the GSAN cid b8429c8a089567b7a1aa8cfa7608b735413630ab exists in MCDB at path /avamarVC/VirtualMachines/DPADatastoreServer_bgVYLK0WONuiFagzVngqBQ, and has 0 backups GSAN Mismatch Count: 1 Analyze MCDB mismatches MCDB path /avamarVC/VirtualMachines/DPADatastoreServer_bgVYLK0WONuiFagzVngqBQ does not exist in GSAN, but the MCDB cid b8429c8a089567b7a1aa8cfa7608b735413630ab exists in GSAN at path MCDB Mismatch Count: 1 Report Files Generated ---------------------- Safe Report: /home/admin/goav_clients/client_cid_mismatch_report_SAFE_delete_2024-08-19-1724095237 Unsafe Report: /home/admin/goav_clients/client_cid_mismatch_report_UNSAFE_delete_2024-08-19-1724095237
At the bottom of the output in the example, there is information about the location of the report files.
CID mismatches which were designated as safe to remove do not require SME review and approval and can be resolved without Dell support intervention.
CID mismatches which were designated as unsafe require SME review and approval.
The customer or support engineer provides the report files and the Goav secure access key request to an SME to fix unsafe mismatches.
Unsafe example:
./goav client cid-mismatch fix
Process CID Mismatch Reports ---------------------------- Parsing reports for mismatches to fix No safe mismatches found to fix Provide the following CID mismatch unsafe auto-fix key request to an SME --> 1709325232,1724098448 <--- >
In the unsafe mismatches example above, the SME grants a secure access key based on the request 1709325232,1724098448.
What makes a CID mismatch safe or unsafe to remove?
Safe CID mismatches are where the client path exists in one store but not the other. The CID does not match the CID of a client with a different path in the other store and has no backups in GSAN.
Unsafe CID mismatches are considered through the following scenarios:
- The client still has backups in GSAN to remove.
- The client path does not exist in the other store but has a matching CID to a different path.
- The client paths match in both stores and have a different CID.