Avamar: Data Domain nie jest wyświetlana w Avamar Web Console lub Administrator Console

• Data Domain nie jest wyświetlana w interfejsie użytkownika (UI) Avamar ani w konsoli administratora Avamar.
• Ponowne uruchomienie usługi konsoli zarządzania (MCS) nie rozwiązuje problemu.

admin@av-server:~/>: mccli dd show-prop --name=ddve01.lab.com
1,30943,Data Domain system does not exist.
admin@av-server:~/>: 

• Ponowne uruchamianie przy użyciu interfejsu wiersza poleceń (CLI) przy użyciu mcddrsnmp Nie rozwiązuje problemu.

root@av-server:~/#: mcddrsnmp restart
INFO: Stopping Data Domain SNMP Manager....
INFO: Data Domain SNMP Manager is not running
INFO: Starting Data Domain SNMP Manager....
INFO: Connecting to MCS Server: av-server.emn.com at port: 7778...
INFO: Successfully connected to MCS Server: av-server.lab.com at port: 7778.
INFO: No trap listeners were started, Data Domain SNMP Manager didn't start.
root@av-server:~/#: 

• ddrmaint Polecenie działa poprawnie:

admin@av-server:~/>: ddrmaint read-ddr-info --format=full
====================== Read-DDR-Info ======================

 System name        : ddve01.lab.com
 System ID          : 608023E84E6C40CFB5F6608023E84E6123456789
 DDBoost user       : ddboost
 System index       : 1
 Replication        : True
 CP Backup          : True
 Model number       : DD VE
 Serialno           : AUDVHS64123456
 DDOS version       : 7.10.1.20-1090468
 System attached    : 2023-05-26 12:33:29 (1685104409)
 System max streams : 16
 RL Mode            : -1
 RL Status          : -1
admin@av-server:~/>:

• Podczas uruchamiania MCS pojawia się następujący komunikat w /usr/local/avamar/var/mc/server_log/mcserver.log.0:

INFO: Collection of DdrInfo is null or empty.

• Proaktywny skrypt kontroli poprawności działania pokazuje, że opróżnianie MCS nie powiodło się przez 6 dni:

ERROR: No MCS flush in past 24 hours. Last one is 2024-08-30 08:03:43
RESOLUTION: Try 'mcserver.sh --flush' to see why MCS flushes are failing
# --> MC flush in past 24 hours      *FAILED*

• List MC_BACKUPS Potwierdza problem:

admin@av-server:~/>: avtar --backups --path=/MC_BACKUPS --count=5
avtar Info <5551>: Command Line: /usr/local/avamar/bin/avtar.bin --flagfile=/usr/local/avamar/etc/usersettings.cfg --server=av-server --vardir=/usr/local/avamar/var --bindir=/usr/local/avamar/bin --id=root --password=**************** --vardir=/usr/local/avamar/var --bindir=/usr/local/avamar/bin --sysdir=/usr/local/avamar/etc --backups --account=/MC_BACKUPS --count=5
avtar Info <7977>: Starting at 2024-09-05 22:50:48 IST [avtar Dec  8 2023 07:07:43 19.10.100-135 Linux-x86_64]
avtar Info <6555>: Initializing connection
avtar Info <5552>: Connecting to Avamar Server (av-server)
avtar Info <5554>: Connecting to one node in each datacenter
avtar Info <5583>: Login User: "root", Domain: "default", Account: "/MC_BACKUPS"
avtar Info <5580>: Logging in on connection 0 (server 0)
avtar Info <5582>: Avamar Server login successful
avtar Info <10632>: Using Client-ID='51947b9a14df3111ca78e435008cde1234567890'
avtar Info <5550>: Successfully logged into Avamar Server [19.10.0-135]
avtar Info <19849>: Selecting 5 backups
avtar Info <7377>: Backups for /MC_BACKUPS as of 2024-09-05 22:50:48 IST
    Date      Time    Seq       Label           Size     Plugin    Working directory         Targets
 ---------- -------- ----- ----------------- ---------- -------- --------------------- -------------------
 2024-08-30 08:03:43   904                      565563K Linux    /usr/local/avamar     var/mc/server_data
 2024-08-30 07:46:27   903                      565518K Linux    /usr/local/avamar     var/mc/server_data
 2024-08-29 08:02:17   902                      566738K Linux    /usr/local/avamar     var/mc/server_data
 2024-08-29 07:46:44   901                      566693K Linux    /usr/local/avamar     var/mc/server_data
 2024-08-28 08:02:39   900                      567929K Linux    /usr/local/avamar     var/mc/server_data
avtar Info <5314>: Command completed (exit code 0: success)
admin@av-server:~/>:

• Polecenie /usr/local/avamar/var/mc/server_log/flush.log Wyświetla następujący błąd:

...
2024-09-05 08:01:01 avtar Info <5552>: Connecting to Avamar Server (av-server)
2024-09-05 08:01:01 avtar Info <5554>: Connecting to one node in each datacenter
2024-09-05 08:01:01 avtar Info <5993>: - Connect: Connected to 123.123.1.113:29000, Priv=0, SSL Cipher=ECDHE-RSA-AES256-GCM-SHA384
2024-09-05 08:01:01 avtar Info <5993>: - Datacenter 0 has 1 nodes: Connected to 123.123.1.113:29000, Priv=0, SSL Cipher=ECDHE-RSA-AES256-GCM-SHA384
2024-09-05 08:01:01 avtar Info <5583>: Login User: "root", Domain: "default", Account: "/MC_BACKUPS"
2024-09-05 08:01:01 avtar Info <5580>: Logging in on connection 0 (server 0)
2024-09-05 08:01:01 avtar Error <5126>: Login error 69: Authentication failure  (Invalid password?  User disabled?)
2024-09-05 08:01:01 avtar FATAL <8941>: Fatal server connection problem, aborting initialization. Verify correct server address and login credentials.
2024-09-05 08:01:01 avtar Info <6149>: Error summary: 2 errors: 8941, 5126
2024-09-05 08:01:01 avtar Info <6645>: Not sending wrapup anywhere.
2024-09-05 08:01:01 avtar Info <5314>: Command failed (2 errors, exit code 10008: cannot establish connection with server (possible network or DNS failure))
--------------------------------------------------------------------------------------------------------
----- END avtar log 2024-09-05 08:01:01 IST  (0 warnings, 2 errors, 1 fatal error)
--------------------------------------------------------------------------------------------------------

• Zmiana haseł została przeprowadzona sześć dni temu:

root@av-server:/home/admin/#: history | grep change
    1  2023-06-01 19:55:53 change-passwords
  191  2024-08-30 19:42:59 change-passwords
  555  2024-09-05 22:56:13 history | grep change
root@av-server:/home/admin/#:

• Hasło w /usr/local/avamar/var/mc/server_data/prefs/mcserver.xml pokazuje katalog główny i MCUser używający tego samego hasła: 

root@av-server:/home/admin/proactive_check/#: grep AP /usr/local/avamar/var/mc/server_data/prefs/mcserver.xml
                <entry key="backuprestoreAP" value="{AES}hMKAX9yMuHxbyvltADTR0NALyFSXDMMTCQIqYkry5Bk=" />
                <entry key="backuponlyAP" value="{AES}hMKAX9yMuHxbyvltADTR0NALyFSXDMMTCQIqYkry5Bk=" />
                <entry key="rootAP" value="{AES}cBqSwCN8Grcb/f735SS9Yw==" />
                <entry key="MCUSERAP" value="{AES}cBqSwCN8Grcb/f735SS9Yw==" />
                <entry key="restoreonlyAP" value="{AES}hMKAX9yMuHxbyvltADTR0NALyFSXDMMTCQIqYkry5Bk=" />
                <entry key="viewuserAP" value="{AES}cBqSwCN8Grcb/f735SS9Yw==" />
              <entry key="monarch_license_feature_codes" value="POWER_PROTECT_CAPACITY" />
              <entry key="rabbitmqAP" value="{AES}UhnKR3q+4LSjgqTF5SSquw==" />
root@av-server:/home/admin/proactive_check/#:

• Jeśli jednak zalogujesz się do interfejsu użytkownika Avamar za pomocą MCUser i root, hasło jest inne.
• Na poziomie GSAN hasło dla użytkownika root i MCUser są poprawne:

admin@av-server:~/>: avmgr logn --id=root --ap=*******
1  Request succeeded
-1  privilege level  (enabled,create,read,backup,access,move,delete,maint,manage,fullmanage,noticketrequired,superuser,ignoreacls,readdir,mclogin,opt1,opt2)
2  block type  (directory)
admin@av-server:~/>:

admin@av-server:~/>: avmgr logn --id=MCUser --ap=xxxxxxx
1  Request succeeded
7161  privilege level  (enabled,create,read,backup,access,move,delete,maint,fullmanage,noticketrequired)
2  block type  (directory)
admin@av-server:~/>:

Aby rozwiązać ten problem, zaszyfruj hasło serwera głównego w mcserver.xml i uruchom ponownie MCS. Spowoduje to uruchomienie Data Domain SNMP Manager, dzięki czemu Data Domain będzie widoczna zarówno w interfejsie użytkownika Avamar, jak i MCCLI.

• Umieść hasło serwera głównego w postaci zwykłego tekstu na /usr/local/avamar/var/mc/server_data/prefs/mcserver.xml i ponownie zaszyfruj.
• Kopia zapasowa mcserver.xml:

cp -p /usr/local/avamar/var/mc/server_data/prefs/mcserver.xml /usr/local/avamar/var/mc/server_data/prefs/x-mcserver.xml

• Edytuj plik /usr/local/avamar/var/mc/server_data/prefs/mcserver.xml i umieść hasło serwera głównego jako zwykły tekst w wartości rootAP:

admin@av-server:/home/admin/#: vi /usr/local/avamar/var/mc/server_data/prefs/mcserver.xml
                <entry key="backuprestoreAP" value="{AES}hMKAX9yMuHxbyvltADTR0NALyFSXDMMTCQIqYkry5Bk=" />
                <entry key="backuponlyAP" value="{AES}hMKAX9yMuHxbyvltADTR0NALyFSXDMMTCQIqYkry5Bk=" />
                <entry key="rootAP" value="change-password-here" />
                <entry key="MCUSERAP" value="{AES}cBqSwCN8Grcb/f735SS9Yw==" />
                <entry key="restoreonlyAP" value="{AES}hMKAX9yMuHxbyvltADTR0NALyFSXDMMTCQIqYkry5Bk=" />
                <entry key="viewuserAP" value="{AES}cBqSwCN8Grcb/f735SS9Yw==" />
              <entry key="monarch_license_feature_codes" value="POWER_PROTECT_CAPACITY" />
              <entry key="rabbitmqAP" value="{AES}UhnKR3q+4LSjgqTF5SSquw==" />
admin@av-server:/home/admin/#:

• Zaszyfruj hasło:

root@av-server:/home/admin/#: mccipher encrypt --all
**********************************************************************************
* EMC Avamar Management Console (MC).                                            *
* MC Security Tool for Secret Key generation, encryption, decryption and digest. *
**********************************************************************************
Ignore to encrypted preference, since value is NULL or EMPTY: mcserver:/usr/local/avamar/lib:com/avamar/mc/cac/ldap_login_ap
encrypted processedValue is null, with prefKey: mcserver:/usr/local/avamar/lib:com/avamar/mc/cac/ldap_login_ap, will not be stored
Ignore to encrypted preference, since value is NULL or EMPTY: mcserver:/usr/local/avamar/var/mc/server_data/prefs:com/avamar/mc/cac/ldap_login_ap
encrypted processedValue is null, with prefKey: mcserver:/usr/local/avamar/var/mc/server_data/prefs:com/avamar/mc/cac/ldap_login_ap, will not be stored
encrypted processedValue is null, with prefKey: mcserver:/usr/local/avamar/var/mc/server_data/prefs:com/avamar/mc/dpn/users/MCUSERAP, will not be stored
encrypted processedValue is null, with prefKey: mcserver:/usr/local/avamar/var/mc/server_data/prefs:com/avamar/mc/dpn/users/backuponlyAP, will not be stored
encrypted processedValue is null, with prefKey: mcserver:/usr/local/avamar/var/mc/server_data/prefs:com/avamar/mc/dpn/users/backuprestoreAP, will not be stored
encrypted processedValue is null, with prefKey: mcserver:/usr/local/avamar/var/mc/server_data/prefs:com/avamar/mc/dpn/users/restoreonlyAP, will not be stored
encrypted processedValue is null, with prefKey: mcserver:/usr/local/avamar/var/mc/server_data/prefs:com/avamar/mc/dpn/users/viewuserAP, will not be stored
encrypted processedValue is null, with prefKey: mcserver:/usr/local/avamar/var/mc/server_data/prefs:com/avamar/mc/msg_broker/rabbitmqAP, will not be stored
WARNING: Can not encrypt some or all MCS Preferences.
All MC DB Tables have been encrypted successfully.
All MCCLI config files have been encrypted successfully.
See MCCipher log for details.
? 2012 EMC Corporation. All rights reserved.
root@av-server:/home/admin#:

• Uruchom ponownie MCS. Podczas ponownego uruchamiania uruchamia się Data Domain SNMP Manager:

INFO: Starting Data Domain SNMP Manager....
INFO: Connecting to MCS Server: av-server.emn.com at port: 7778...
INFO: Successfully connected to MCS Server: av-server.lab.com at port: 7778.
INFO: Trap listeners status:
INFO: Listening to port 163 for traps from [ddve01.lab.com]
INFO: Data Domain SNMP Manager started.

• Płukanie MCS działa dobrze.
• Data Domain pojawi się ponownie w interfejsie użytkownika Avamar i MCCLI:

admin@av-server:~/>: mccli dd show-prop --name=ddve01.lab.com
0,23000,CLI command completed successfully.
Attribute                                     Value
--------------------------------------------- ---------------------------------
IPv4 Hostname                                 ddve01.lab.com
IPv6 Hostname                                 N/A
Total Capacity (post-comp size)               3038.2 GiB
Server Utilization (post-comp use%)           38%
Bytes Protected (client pre-comp size)        1.1 TB
File System Available (post-comp avail)       1879.3 GiB
File System Used (post-comp used)             1158.9 GiB
User Name                                     ddboost
Default Replication Storage System            Yes
Target For Avamar Checkpoint Backups          Yes
Maximum Streams For Avamar Checkpoint Backups 1
Maximum Streams                               16
Maximum Streams Limit                         20
Instant Access Limit                          1
DDOS Version                                  7.10.1.20-1090468
Serial Number                                 AUDVHS64123456
Model Number                                  DD VE
Encryption Strength                           none
Authentication Mode                           none
Monitoring Status                             SNMP trap manager is not running.
Cloud Tier                                    disabled
DDBoost Licensed                              true
DDBoost Enabled                               true
DDBoost User Enabled                          true
DDBoost User Status                           User Valid
DDBoost Option Status                         Option Enabled
SNMP status                                   Enabled
File System Status                            File System Running
Synchronization of maintenance operations     on
admin@av-server:~/>: