Skip to main content
Sign Out

Welcome to Dell

My Account
  • Place orders quickly and easily
  • View orders and track your shipping status
  • Enjoy members-only rewards and discounts
  • Create and access a list of your products
  • Manage your Dell EMC sites, products, and product-level contacts using Company Administration.
Sign In Create an Account Dell Financial Services Premier Sign In Partner Program Sign In
Contact Us

Your Dell.com Carts

Article Number: 000203605

Avamar: Avamar Client Manager (/aam) loading webpage hangs on login

Summary: Avamar : Avamar Client Manager (/aam) loading webpage hangs on login. The login may never finish or take tens of minutes to complete.

This article may have been automatically translated. If you have any feedback regarding its quality, please let us know using the form at the bottom of this page.

Article Content

Symptoms

Avamar : Avamar Client Manager (/aam) loading webpage hangs on login.
The login may never finish or take tens of minutes to complete.
The page may not timeout but leave you waiting a long time.

Cause

The cause for this issue may be split into two:
1. A large amount of remote servers added to a single Avamar Client Manager causing the load on the tomcat webserver to increase.

We can see a large amount of remote servers added to this one Avamar from the tomcat database axion_server table 
psql -p 5556 emdb

select id, lastcontact, axionsystemname from axion_server;

      id       |     lastcontact     |       axionsystemname        
---------------+---------------------+------------------------------
 1629287858128 | 1969-12-31 19:00:00 | AVAMAR10.EX.COM              
 1629287980100 | 1969-12-31 19:00:00 | AVAMAR20.EX.COM
 1629287647775 | 1969-12-31 19:00:00 | AVAMAR30.EX.COM     
 1629287949115 | 1969-12-31 19:00:00 | AVAMAR19.EX.COM  
 1660563561277 | 1969-12-31 19:00:00 | AVAMAR40.EX.COM    
 1629287597140 | 1969-12-31 19:00:00 | AVAMAR50.EX.COM                                             
 1629287775889 | 1969-12-31 19:00:00 | AVAMAR60.EX.COM              
 1629287812033 | 1969-12-31 19:00:00 | AVAMAR70.EX.COM              
 1629288006387 | 1969-12-31 19:00:00 | AVAMAR80.EX.COM              
 1629287515457 | 1969-12-31 19:00:00 | AVAMAR90.EX.COM       
             1 | 1969-12-31 19:00:00 | AVAMAR100.EX.COM              
 1629287732083 | 1969-12-31 19:00:00 | AVAMAR200.EX.COM  
 1629288102775 | 1969-12-31 19:00:00 | AVAMAR300.EX.COM              
 1662042283060 | 1969-12-31 19:00:00 | AVAMAR400.EX.COM              
 1651676768316 | 1969-12-31 19:00:00 | AVAMAR500.EX.COM              
(15 rows)             


2. A remote server with MCUser password issue

From the table output above, Avamar Server AVAMAR300.EX.COM had an issue with the MCUser password.

On the AVAMAR300.EX.COM, the MCS service was running and a login test with MCUser passed fine in the GUI as well as command-line.
Example: 
avmgr logn --id=MCUser --ap=MyMCUserPass
1  Request succeeded
7161  privilege level  (enabled,create,read,backup,access,move,delete,maint,fullmanage,noticketrequired)
2  block type  (directory)

We can see that although this looks fine, on the remote server AVAMAR300.EX.COM the mcserver.out log shows there is a clear issue with the MCUser password. 
vi /usr/local/avamar/var/mc/server_data/prefs/mcserver.out

=== BEGIN === check.mcs (prestart)
check.mcs                        passed
=== PASS === check.mcs PASSED OVERALL (prestart)
Starting Administrator Server at Tue Aug 30 15:04:25 EDT 2022 ...
Upgrade MCS Preference: processing time = 325ms
Upgrade MCDB: processing time = 262ms
Check node list: processing time = 1527ms
Prepare MC SSL Properties: processing time = 2ms
Init node: processing time = 420ms
Install service: com.avamar.mc.prefs.MCSPreferencesService, processing time = 12ms
Install service: com.avamar.mc.message.MessageEventService, processing time = 15ms
Event Size: 18548
Install service: com.avamar.mc.event.EventService, processing time = 11388ms
Install service: com.avamar.mc.jms.push.JmsPushService, processing time = 40ms
Install service: com.avamar.mc.dpn.DPNProxyService, processing time = 2031ms
Install service: com.avamar.mc.datatap.MCDataTapService, processing time = 22ms
Install service: com.avamar.mc.cm.ClusterManagerService, processing time = 12ms
[output shortened...]
Install service: com.avamar.mc.ws.MCJettyService, processing time = 1272ms
Install service: com.avamar.mc.repl.ReplicationService, processing time = 85ms
Install service: com.avamar.mc.st.SessionTicketService, processing time = 1434ms
Install service: com.avamar.mc.ras.McEbmsService, processing time = 2051ms
Install service: com.avamar.mc.migration.MigrationService, processing time = 60ms
Start service container: processing time = 51304ms
log4j:WARN No appenders could be found for logger (org.apache.axiom.util.stax.dialect.StAXDialectDetector).
log4j:WARN Please initialize the log4j system properly.
Init crontab: processing time = 3800ms
Update axion system table: processing time = 3ms
Update connect emc email subject: processing time = 70ms
Administrator Server started at Tue Aug 30 15:05:26 EDT 2022.
Update connect emc email subject: processing time = 70ms
Administrator Server started at Tue Aug 30 15:05:26 EDT 2022.
MCS Preference has been decrypted failed: com/avamar/mc/dpn/users/MCUSERAP
MCS Preference has been decrypted failed: com/avamar/mc/dpn/users/MCUSERAP
MCS Preference has been decrypted failed: com/avamar/mc/dpn/users/MCUSERAP
MCS Preference has been decrypted failed: com/avamar/mc/dpn/users/MCUSERAP
MCS Preference has been decrypted failed: com/avamar/mc/dpn/users/MCUSERAP
ERROR: Invalid AES key length
MCS Preference has been decrypted failed: com/avamar/mc/dpn/users/MCUSERAP
MCS Preference has been decrypted failed: com/avamar/mc/dpn/users/MCUSERAP
MCS Preference has been decrypted failed: com/avamar/mc/dpn/users/MCUSERAP

Resolution

Steps

1. Remove ALL axion servers BESIDES the source server (the server that you are adding other servers to). The ID column for the source server that you will not touch has an ID = 1.

Original Table contents from cause section 
psql -p 5556 emdb

select id, lastcontact, axionsystemname from axion_server;

      id       |     lastcontact     |       axionsystemname        
---------------+---------------------+------------------------------
 1629287858128 | 1969-12-31 19:00:00 | AVAMAR10.EX.COM              
 1629287980100 | 1969-12-31 19:00:00 | AVAMAR20.EX.COM
 1629287647775 | 1969-12-31 19:00:00 | AVAMAR30.EX.COM     
 1629287949115 | 1969-12-31 19:00:00 | AVAMAR19.EX.COM  
 1660563561277 | 1969-12-31 19:00:00 | AVAMAR40.EX.COM    
 1629287597140 | 1969-12-31 19:00:00 | AVAMAR50.EX.COM                                         
 1629287775889 | 1969-12-31 19:00:00 | AVAMAR60.EX.COM              
 1629287812033 | 1969-12-31 19:00:00 | AVAMAR70.EX.COM              
 1629288006387 | 1969-12-31 19:00:00 | AVAMAR80.EX.COM              
 1629287515457 | 1969-12-31 19:00:00 | AVAMAR90.EX.COM       
             1 | 1969-12-31 19:00:00 | AVAMAR100.EX.COM              
 1629287732083 | 1969-12-31 19:00:00 | AVAMAR200.EX.COM  
 1629288102775 | 1969-12-31 19:00:00 | AVAMAR300.EX.COM              
 1662042283060 | 1969-12-31 19:00:00 | AVAMAR400.EX.COM              
 1651676768316 | 1969-12-31 19:00:00 | AVAMAR500.EX.COM              
(15 rows)         

Run delete command to delete each server except the id=1 server (AVAMAR100.EX.COM)

// connect to emdb
psql -p 5556 emdb

// delete each server besides source server
delete from axion_server where id!='1';


2. Confirm the other remote servers are all deleted.

select id, lastcontact, axionsystemname from axion_server;

      id       |     lastcontact     |       axionsystemname        
---------------+---------------------+------------------------------
             1 | 1969-12-31 19:00:00 | AVAMAR100.EX.COM              


3. Exit the emdb using postgresql syntax.

\q

 

4. Restart tomcat so we can identify when the servlet re launches and get the latest logs for the issue, as root user 
emwebapp.sh --restart

5. Log into Client Manager (/aam) webpage again and try to add remote servers again by hostname/MCUser credential. In this example, we tried to add AVAMAR300.EX.COM which threw an error. We can see in the ACMTrace.log after the tomcat server was restarted the following logs for trying to add AVAMAR300.EX.COM. 
vi /usr/local/avamar-tomcat/logs/ACMTrace.log

// latest logs post tomcat restart start with "Servlet Context Launched."
2022-09-21 16:03:57,802 INFO  [localhost-startStop-1]-listeners.ContextLauncher: Servlet Context Launched.
2022-09-21 16:04:00,623 INFO  [localhost-startStop-1]-sdk.ServiceConnection: Wait for task: STORAGE_INFO: Getting storage information
2022-09-21 16:04:01,625 INFO  [localhost-startStop-1]-sdk.ServiceConnection: Task done: STORAGE_INFO
2022-09-21 16:04:01,660 INFO  [localhost-startStop-1]-utils.ACMProperties: Cas Server Url Ssl Prefix: https://AVAMAR300.EX.COM
2022-09-21 16:04:06,669 INFO  [ajp-nio-0:0:0:0:0:0:0:1-8509-exec-1]-filters.ImageLocaleFilter: Locale in China: false
2022-09-21 16:04:07,926 INFO  [ajp-nio-0:0:0:0:0:0:0:1-8509-exec-8]-filters.ImageLocaleFilter: Locale in China: false
2022-09-21 16:04:10,172 WARN  [ajp-nio-0:0:0:0:0:0:0:1-8509-exec-1]-helper.LDAPManagerUserAuth: LDAP property 'ldap.url.*' is not defined
2022-09-21 16:04:10,176 WARN  [ajp-nio-0:0:0:0:0:0:0:1-8509-exec-1]-helper.LDAPManagerUserAuth: LDAP property 'ldap.url.*' is not defined
2022-09-21 16:04:10,178 WARN  [ajp-nio-0:0:0:0:0:0:0:1-8509-exec-1]-helper.LDAPManagerUserAuth: LDAP property 'ldap.url.*' is not defined
2022-09-21 16:04:10,179 WARN  [ajp-nio-0:0:0:0:0:0:0:1-8509-exec-1]-helper.LDAPManagerUserAuth: LDAP property 'ldap.url.*' is not defined
2022-09-21 16:04:10,180 INFO  [ajp-nio-0:0:0:0:0:0:0:1-8509-exec-1]-auth.AvamarAuthConfigurationUserAuth: Authentication Type: Web Restore Avamar
2022-09-21 16:04:11,037 WARN  [ajp-nio-0:0:0:0:0:0:0:1-8509-exec-1]-cli.MCInternalCLIUserAuth: MCCLI exec catch Execption: Unable to initialize MCCLI
2022-09-21 16:04:11,043 INFO  [ajp-nio-0:0:0:0:0:0:0:1-8509-exec-1]-cli.MCInternalCLIUserAuth: authenticate result out: 0,23000,CLI command completed successfully.

2022-09-21 16:04:11,310 WARN  [ajp-nio-0:0:0:0:0:0:0:1-8509-exec-1]-cli.MCInternalCLIUserAuth: MCCLI exec catch Execption: Unable to initialize MCCLI
2022-09-21 16:04:11,312 INFO  [ajp-nio-0:0:0:0:0:0:0:1-8509-exec-1]-cli.MCInternalCLIUserAuth: get privilege command output: <CLIOutput>
  <Results>
    <ReturnCode>0</ReturnCode>
    <EventCode>23000</EventCode>
    <EventSummary>CLI command completed successfully.</EventSummary>
  </Results>
  <Data>
    <Row>
      <Name>erx1mxe</Name>
      <Role>Administrator</Role>
      <Domain>/</Domain>
      <Authenticator>Axion Authentication System</Authenticator>
      <Domain>/</Domain>
      <Authenticator>Axion Authentication System</Authenticator>
    </Row>
  </Data>
</CLIOutput>

2022-09-21 16:04:11,692 INFO  [ajp-nio-0:0:0:0:0:0:0:1-8509-exec-2]-sdk.ServiceConnection: Wait for task: STORAGE_INFO: Getting storage information
2022-09-21 16:04:12,693 INFO  [ajp-nio-0:0:0:0:0:0:0:1-8509-exec-2]-sdk.ServiceConnection: Task done: STORAGE_INFO
2022-09-21 16:04:32,272 WARN  [ajp-nio-0:0:0:0:0:0:0:1-8509-exec-1]-helper.LDAPManagerUserAuth: LDAP property 'ldap.url.*' is not defined
2022-09-21 16:04:32,273 WARN  [ajp-nio-0:0:0:0:0:0:0:1-8509-exec-1]-helper.LDAPManagerUserAuth: LDAP property 'ldap.url.*' is not defined
2022-09-21 16:04:32,273 WARN  [ajp-nio-0:0:0:0:0:0:0:1-8509-exec-1]-helper.LDAPManagerUserAuth: LDAP property 'ldap.url.*' is not defined
2022-09-21 16:04:32,273 WARN  [ajp-nio-0:0:0:0:0:0:0:1-8509-exec-1]-helper.LDAPManagerUserAuth: LDAP property 'ldap.url.*' is not defined
2022-09-21 16:04:32,274 INFO  [ajp-nio-0:0:0:0:0:0:0:1-8509-exec-1]-auth.AvamarAuthConfigurationUserAuth: Authentication Type: Web Restore Avamar
2022-09-21 16:04:32,528 WARN  [ajp-nio-0:0:0:0:0:0:0:1-8509-exec-1]-cli.MCInternalCLIUserAuth: MCCLI exec catch Execption: Unable to initialize MCCLI
2022-09-21 16:04:32,529 INFO  [ajp-nio-0:0:0:0:0:0:0:1-8509-exec-1]-cli.MCInternalCLIUserAuth: authenticate result out: 0,23000,CLI command completed successfully.

2022-09-21 16:04:32,791 WARN  [ajp-nio-0:0:0:0:0:0:0:1-8509-exec-1]-cli.MCInternalCLIUserAuth: MCCLI exec catch Execption: Unable to initialize MCCLI
2022-09-21 16:04:32,792 INFO  [ajp-nio-0:0:0:0:0:0:0:1-8509-exec-1]-cli.MCInternalCLIUserAuth: get privilege command output: <CLIOutput>
  <Results>
    <ReturnCode>0</ReturnCode>
    <EventCode>23000</EventCode>
    <EventSummary>CLI command completed successfully.</EventSummary>
  </Results>
  <Data>
    <Row>
      <Name>root</Name>
      <Role>Administrator</Role>
      <Domain>/</Domain>
      <Authenticator>Axion Authentication System</Authenticator>
    </Row>
  </Data>
</CLIOutput>

2022-09-21 16:04:33,073 INFO  [ajp-nio-0:0:0:0:0:0:0:1-8509-exec-2]-sdk.ServiceConnection: Wait for task: STORAGE_INFO: Getting storage information
2022-09-21 16:04:34,074 INFO  [ajp-nio-0:0:0:0:0:0:0:1-8509-exec-2]-sdk.ServiceConnection: Task done: STORAGE_INFO

6. Now we know there is an issue with the MCUser credential on the AVAMAR300.EX.COM, refer to the Cause #2 section for details. Like mentioned before, MCUser was unable to be decrypted so when the source Avamar tries to register the remote Avamar using its MCUser credential it fails. 
We must run change-passwords as root user to udpate MCUser on AVAMAR300.EX.COM.
The password does not need to be changed from what it currently is, the same password can be entered in the script but the script must be run to update the password in all of the configuration files with the proper encryption/decryption key applied. 
change-passwords

7. After updating MCUser password on the remote server, refresh the source server Client Manager (/aam) webpage (AVAMAR100.EX.COM). Then attempt to successfully re-add AVAMAR300.EX.COM to the source server.

Article Properties

Affected Product

Avamar

Last Published Date

21 Feb 2023

Version

8

Article Type

Solution

Back to Top