Avamar: Data Domain Does Not Show in Avamar Web or Administrator Console

• Data Domain is not shown in the Avamar User Interface (UI) or the Avamar Administrator console.
• Restarting the Management Console Service (MCS) does not fix the issue.

admin@av-server:~/>: mccli dd show-prop --name=ddve01.lab.com
1,30943,Data Domain system does not exist.
admin@av-server:~/>: 

• Restarting with the command-line interface (CLI) using mcddrsnmp does not fix the issue.

root@av-server:~/#: mcddrsnmp restart
INFO: Stopping Data Domain SNMP Manager....
INFO: Data Domain SNMP Manager is not running
INFO: Starting Data Domain SNMP Manager....
INFO: Connecting to MCS Server: av-server.emn.com at port: 7778...
INFO: Successfully connected to MCS Server: av-server.lab.com at port: 7778.
INFO: No trap listeners were started, Data Domain SNMP Manager didn't start.
root@av-server:~/#: 

• ddrmaint command works fine:

admin@av-server:~/>: ddrmaint read-ddr-info --format=full
====================== Read-DDR-Info ======================

 System name        : ddve01.lab.com
 System ID          : 608023E84E6C40CFB5F6608023E84E6123456789
 DDBoost user       : ddboost
 System index       : 1
 Replication        : True
 CP Backup          : True
 Model number       : DD VE
 Serialno           : AUDVHS64123456
 DDOS version       : 7.10.1.20-1090468
 System attached    : 2023-05-26 12:33:29 (1685104409)
 System max streams : 16
 RL Mode            : -1
 RL Status          : -1
admin@av-server:~/>:

• During MCS start, the following message appears in /usr/local/avamar/var/mc/server_log/mcserver.log.0:

INFO: Collection of DdrInfo is null or empty.

• Proactive health check script shows that MCS flush was failing for 6 days:

ERROR: No MCS flush in past 24 hours. Last one is 2024-08-30 08:03:43
RESOLUTION: Try 'mcserver.sh --flush' to see why MCS flushes are failing
# --> MC flush in past 24 hours      *FAILED*

• List MC_BACKUPS confirms the issue:

admin@av-server:~/>: avtar --backups --path=/MC_BACKUPS --count=5
avtar Info <5551>: Command Line: /usr/local/avamar/bin/avtar.bin --flagfile=/usr/local/avamar/etc/usersettings.cfg --server=av-server --vardir=/usr/local/avamar/var --bindir=/usr/local/avamar/bin --id=root --password=**************** --vardir=/usr/local/avamar/var --bindir=/usr/local/avamar/bin --sysdir=/usr/local/avamar/etc --backups --account=/MC_BACKUPS --count=5
avtar Info <7977>: Starting at 2024-09-05 22:50:48 IST [avtar Dec  8 2023 07:07:43 19.10.100-135 Linux-x86_64]
avtar Info <6555>: Initializing connection
avtar Info <5552>: Connecting to Avamar Server (av-server)
avtar Info <5554>: Connecting to one node in each datacenter
avtar Info <5583>: Login User: "root", Domain: "default", Account: "/MC_BACKUPS"
avtar Info <5580>: Logging in on connection 0 (server 0)
avtar Info <5582>: Avamar Server login successful
avtar Info <10632>: Using Client-ID='51947b9a14df3111ca78e435008cde1234567890'
avtar Info <5550>: Successfully logged into Avamar Server [19.10.0-135]
avtar Info <19849>: Selecting 5 backups
avtar Info <7377>: Backups for /MC_BACKUPS as of 2024-09-05 22:50:48 IST
    Date      Time    Seq       Label           Size     Plugin    Working directory         Targets
 ---------- -------- ----- ----------------- ---------- -------- --------------------- -------------------
 2024-08-30 08:03:43   904                      565563K Linux    /usr/local/avamar     var/mc/server_data
 2024-08-30 07:46:27   903                      565518K Linux    /usr/local/avamar     var/mc/server_data
 2024-08-29 08:02:17   902                      566738K Linux    /usr/local/avamar     var/mc/server_data
 2024-08-29 07:46:44   901                      566693K Linux    /usr/local/avamar     var/mc/server_data
 2024-08-28 08:02:39   900                      567929K Linux    /usr/local/avamar     var/mc/server_data
avtar Info <5314>: Command completed (exit code 0: success)
admin@av-server:~/>:

• The /usr/local/avamar/var/mc/server_log/flush.log shows the following error:

...
2024-09-05 08:01:01 avtar Info <5552>: Connecting to Avamar Server (av-server)
2024-09-05 08:01:01 avtar Info <5554>: Connecting to one node in each datacenter
2024-09-05 08:01:01 avtar Info <5993>: - Connect: Connected to 123.123.1.113:29000, Priv=0, SSL Cipher=ECDHE-RSA-AES256-GCM-SHA384
2024-09-05 08:01:01 avtar Info <5993>: - Datacenter 0 has 1 nodes: Connected to 123.123.1.113:29000, Priv=0, SSL Cipher=ECDHE-RSA-AES256-GCM-SHA384
2024-09-05 08:01:01 avtar Info <5583>: Login User: "root", Domain: "default", Account: "/MC_BACKUPS"
2024-09-05 08:01:01 avtar Info <5580>: Logging in on connection 0 (server 0)
2024-09-05 08:01:01 avtar Error <5126>: Login error 69: Authentication failure  (Invalid password?  User disabled?)
2024-09-05 08:01:01 avtar FATAL <8941>: Fatal server connection problem, aborting initialization. Verify correct server address and login credentials.
2024-09-05 08:01:01 avtar Info <6149>: Error summary: 2 errors: 8941, 5126
2024-09-05 08:01:01 avtar Info <6645>: Not sending wrapup anywhere.
2024-09-05 08:01:01 avtar Info <5314>: Command failed (2 errors, exit code 10008: cannot establish connection with server (possible network or DNS failure))
--------------------------------------------------------------------------------------------------------
----- END avtar log 2024-09-05 08:01:01 IST  (0 warnings, 2 errors, 1 fatal error)
--------------------------------------------------------------------------------------------------------

• The change-passwords ran six days ago:

root@av-server:/home/admin/#: history | grep change
    1  2023-06-01 19:55:53 change-passwords
  191  2024-08-30 19:42:59 change-passwords
  555  2024-09-05 22:56:13 history | grep change
root@av-server:/home/admin/#:

• The password in /usr/local/avamar/var/mc/server_data/prefs/mcserver.xml shows the root and MCUser using the same password: 

root@av-server:/home/admin/proactive_check/#: grep AP /usr/local/avamar/var/mc/server_data/prefs/mcserver.xml
                <entry key="backuprestoreAP" value="{AES}hMKAX9yMuHxbyvltADTR0NALyFSXDMMTCQIqYkry5Bk=" />
                <entry key="backuponlyAP" value="{AES}hMKAX9yMuHxbyvltADTR0NALyFSXDMMTCQIqYkry5Bk=" />
                <entry key="rootAP" value="{AES}cBqSwCN8Grcb/f735SS9Yw==" />
                <entry key="MCUSERAP" value="{AES}cBqSwCN8Grcb/f735SS9Yw==" />
                <entry key="restoreonlyAP" value="{AES}hMKAX9yMuHxbyvltADTR0NALyFSXDMMTCQIqYkry5Bk=" />
                <entry key="viewuserAP" value="{AES}cBqSwCN8Grcb/f735SS9Yw==" />
              <entry key="monarch_license_feature_codes" value="POWER_PROTECT_CAPACITY" />
              <entry key="rabbitmqAP" value="{AES}UhnKR3q+4LSjgqTF5SSquw==" />
root@av-server:/home/admin/proactive_check/#:

• However, log in to Avamar UI using MCUser and root, the password is different.
• At GSAN level, the password for root and MCUser are correct:

admin@av-server:~/>: avmgr logn --id=root --ap=*******
1  Request succeeded
-1  privilege level  (enabled,create,read,backup,access,move,delete,maint,manage,fullmanage,noticketrequired,superuser,ignoreacls,readdir,mclogin,opt1,opt2)
2  block type  (directory)
admin@av-server:~/>:

admin@av-server:~/>: avmgr logn --id=MCUser --ap=xxxxxxx
1  Request succeeded
7161  privilege level  (enabled,create,read,backup,access,move,delete,maint,fullmanage,noticketrequired)
2  block type  (directory)
admin@av-server:~/>:

To address this issue, encrypt the root server password in the mcserver.xml file and restart the MCS. This starts the Data Domain SNMP Manager, making the Data Domain visible in both the Avamar UI and MCCLI.

• Put the root server password in plain text on /usr/local/avamar/var/mc/server_data/prefs/mcserver.xml and encrypt again.
• Backup mcserver.xml:

cp -p /usr/local/avamar/var/mc/server_data/prefs/mcserver.xml /usr/local/avamar/var/mc/server_data/prefs/x-mcserver.xml

• Edit the /usr/local/avamar/var/mc/server_data/prefs/mcserver.xml and put the root server password as plain text in rootAP value:

admin@av-server:/home/admin/#: vi /usr/local/avamar/var/mc/server_data/prefs/mcserver.xml
                <entry key="backuprestoreAP" value="{AES}hMKAX9yMuHxbyvltADTR0NALyFSXDMMTCQIqYkry5Bk=" />
                <entry key="backuponlyAP" value="{AES}hMKAX9yMuHxbyvltADTR0NALyFSXDMMTCQIqYkry5Bk=" />
                <entry key="rootAP" value="change-password-here" />
                <entry key="MCUSERAP" value="{AES}cBqSwCN8Grcb/f735SS9Yw==" />
                <entry key="restoreonlyAP" value="{AES}hMKAX9yMuHxbyvltADTR0NALyFSXDMMTCQIqYkry5Bk=" />
                <entry key="viewuserAP" value="{AES}cBqSwCN8Grcb/f735SS9Yw==" />
              <entry key="monarch_license_feature_codes" value="POWER_PROTECT_CAPACITY" />
              <entry key="rabbitmqAP" value="{AES}UhnKR3q+4LSjgqTF5SSquw==" />
admin@av-server:/home/admin/#:

• Encrypt the password:

root@av-server:/home/admin/#: mccipher encrypt --all
**********************************************************************************
* EMC Avamar Management Console (MC).                                            *
* MC Security Tool for Secret Key generation, encryption, decryption and digest. *
**********************************************************************************
Ignore to encrypted preference, since value is NULL or EMPTY: mcserver:/usr/local/avamar/lib:com/avamar/mc/cac/ldap_login_ap
encrypted processedValue is null, with prefKey: mcserver:/usr/local/avamar/lib:com/avamar/mc/cac/ldap_login_ap, will not be stored
Ignore to encrypted preference, since value is NULL or EMPTY: mcserver:/usr/local/avamar/var/mc/server_data/prefs:com/avamar/mc/cac/ldap_login_ap
encrypted processedValue is null, with prefKey: mcserver:/usr/local/avamar/var/mc/server_data/prefs:com/avamar/mc/cac/ldap_login_ap, will not be stored
encrypted processedValue is null, with prefKey: mcserver:/usr/local/avamar/var/mc/server_data/prefs:com/avamar/mc/dpn/users/MCUSERAP, will not be stored
encrypted processedValue is null, with prefKey: mcserver:/usr/local/avamar/var/mc/server_data/prefs:com/avamar/mc/dpn/users/backuponlyAP, will not be stored
encrypted processedValue is null, with prefKey: mcserver:/usr/local/avamar/var/mc/server_data/prefs:com/avamar/mc/dpn/users/backuprestoreAP, will not be stored
encrypted processedValue is null, with prefKey: mcserver:/usr/local/avamar/var/mc/server_data/prefs:com/avamar/mc/dpn/users/restoreonlyAP, will not be stored
encrypted processedValue is null, with prefKey: mcserver:/usr/local/avamar/var/mc/server_data/prefs:com/avamar/mc/dpn/users/viewuserAP, will not be stored
encrypted processedValue is null, with prefKey: mcserver:/usr/local/avamar/var/mc/server_data/prefs:com/avamar/mc/msg_broker/rabbitmqAP, will not be stored
WARNING: Can not encrypt some or all MCS Preferences.
All MC DB Tables have been encrypted successfully.
All MCCLI config files have been encrypted successfully.
See MCCipher log for details.
? 2012 EMC Corporation. All rights reserved.
root@av-server:/home/admin#:

• Restart MCS. During the restart, the Data Domain SNMP Manager starts:

INFO: Starting Data Domain SNMP Manager....
INFO: Connecting to MCS Server: av-server.emn.com at port: 7778...
INFO: Successfully connected to MCS Server: av-server.lab.com at port: 7778.
INFO: Trap listeners status:
INFO: Listening to port 163 for traps from [ddve01.lab.com]
INFO: Data Domain SNMP Manager started.

• MCS flush works fine.
• The Data Domain appears again in Avamar UI and MCCLI:

admin@av-server:~/>: mccli dd show-prop --name=ddve01.lab.com
0,23000,CLI command completed successfully.
Attribute                                     Value
--------------------------------------------- ---------------------------------
IPv4 Hostname                                 ddve01.lab.com
IPv6 Hostname                                 N/A
Total Capacity (post-comp size)               3038.2 GiB
Server Utilization (post-comp use%)           38%
Bytes Protected (client pre-comp size)        1.1 TB
File System Available (post-comp avail)       1879.3 GiB
File System Used (post-comp used)             1158.9 GiB
User Name                                     ddboost
Default Replication Storage System            Yes
Target For Avamar Checkpoint Backups          Yes
Maximum Streams For Avamar Checkpoint Backups 1
Maximum Streams                               16
Maximum Streams Limit                         20
Instant Access Limit                          1
DDOS Version                                  7.10.1.20-1090468
Serial Number                                 AUDVHS64123456
Model Number                                  DD VE
Encryption Strength                           none
Authentication Mode                           none
Monitoring Status                             SNMP trap manager is not running.
Cloud Tier                                    disabled
DDBoost Licensed                              true
DDBoost Enabled                               true
DDBoost User Enabled                          true
DDBoost User Status                           User Valid
DDBoost Option Status                         Option Enabled
SNMP status                                   Enabled
File System Status                            File System Running
Synchronization of maintenance operations     on
admin@av-server:~/>: