Avamar server showing offline in AER GUI with error "An MCS Fault Message Exception with m essage 'E22801: User login failure.' was caught"

Summary: Avamar server showing offline in AER GUI with error "An MCS Fault Message Exception with message 'E22801: User login failure.' was caught"

This article applies to This article does not apply to This article is not tied to any specific product. Not all product versions are identified in this article.
Check out other resources

Symptoms



External Avamar server showing offline in Avamar Extended Retention GUI although the gridservice is running.

Error reported in Ada Grid Trace logs as below: 
root@svchgva1-avmr001:/opt/EMC/TransportSystemService/log/#:less Adagridtrace.log
2018-05-03 13:39:45,888 INFO  [AvamarSysConfigPolling]-c.e.a.m.u.MCCipherUtils: com.emc.avamar.modelx.util.MCCipherUtils.getPreferenceAttributesByUser(MCUser, null, LOCAL): restoring 2 prefe
rences for user MCUser
2018-05-03 13:39:55,609 INFO  [AvamarSysConfigPolling]-o.a.c.s.f.ReflectionServiceFactoryBean: Creating Service {http://sdk10.mc.avamar.com}mcsdk10 from class com.emc.avamar.modelx.ws.McServ
icePortType
2018-05-03 13:39:56,107 WARN  [AvamarSysConfigPolling]-c.e.a.m.u.MCHostServerConfiguration: MCHostServerConfiguration.getWebServicesMCRegistryService(): An MCS Fault Message Exception with m
essage 'E22801: User login failure.' was caught
com.emc.avamar.modelx.ws.McsFaultMsg_Exception: E22801: User login failure.

Gridservice running correctly:

root@svchgva1-avmr001:~/#: service zzgridservice status
Checking Status of Transport System Service state: running

 

Cause

Communication between gridservice and Avamar MCS failed due to an authentication issue with MCUSERAP password.

Run the following command from mcserver.xml and check if all the passwords are encrypted by default: 
root@svchgva1-avmr001:~/#: grep AP /usr/local/avamar/var/mc/server_data/prefs/mcserver.xml
                <entry key="backuprestoreAP" value="{AES}R4cW/x3fVIo4GSBZJhjHlQ==" />
                <entry key="backuponlyAP" value="{AES}EBnkN1YRr4ZfyP9kSAM9Vg==" />
                <entry key="rootAP" value="{AES}/WsuFhewvKy4XqRnr3qiEoQuLd5s1AIXtURY/fT4Ins=" />
                <entry key="MCUSERAP" value="MCUSer1" />
                <entry key="restoreonlyAP" value="{AES}Fsc/Hdj7FaLUhequP56ZXg==" />
                <entry key="viewuserAP" value="{AES}IWp1I30YOs2Wpa5clVbWQQ==" />

Resolution

Log in as root to the Avamar utility node.

Decrypt and encrypt the passwords: 
root@svchgva1-avmr001:~/#: mccipher decrypt --all

**********************************************************************************
* EMC Avamar Management Console (MC).                                            *
* MC Security Tool for Secret Key generation, encryption, decryption and digest. *
**********************************************************************************


root@svchgva1-avmr001:~/#: mccipher encrypt --all

**********************************************************************************
* EMC Avamar Management Console (MC).                                            *
* MC Security Tool for Secret Key generation, encryption, decryption and digest. *
**********************************************************************************

Check mcserver.xml file again:

root@svchgva1-avmr001:~/#: grep AP /usr/local/avamar/var/mc/server_data/prefs/mcserver.xml
                <entry key="backuprestoreAP" value="{AES}R4cW/x3fVIo4GSBZJhjHlQ==" />
                <entry key="backuponlyAP" value="{AES}EBnkN1YRr4ZfyP9kSAM9Vg==" />
                <entry key="rootAP" value="{AES}/WsuFhewvKy4XqRnr3qiEoQuLd5s1AIXtURY/fT4Ins=" />
                <entry key="MCUSERAP" value="{AES}/WsuFhewvKy4XqRnr3qiEoQuLd5s1AIXtURY/fT4Ins=" />
                <entry key="restoreonlyAP" value="{AES}Fsc/Hdj7FaLUhequP56ZXg==" />
                <entry key="viewuserAP" value="{AES}IWp1I30YOs2Wpa5clVbWQQ==" />

Log in with MCUser password as a test:

root@svchgva1-avmr001:~/#:avmgr logn --id=MCUser --ap=MCUser1
1  Request succeeded
7161  privilege level  (enabled,create,read,backup,access,move,delete,maint,fullmanage,noticketrequired)
2  block type  (directory)

Restart gridservice from Avamar server:

root@svchgva1-avmr001:~/#: service zzgridservice start
Starting TransportSystemService started.
root@svchgva1-avmr001:~/#: service zzgridservice status
Checking Status of Transport System Service state: running

Check the Adagridtrace logs again from the below path , you will find Gridservice responding and the status FULL_ACCESS:

root@svchgva1-avmr001:/opt/EMC/TransportSystemService/log/#:less Adagridtrace.log
2018-05-03 13:44:54,636 INFO  [main]-c.e.a.m.u.MCRegistryService: com.emc.avamar.modelx.util.MCRegistryService.<init>(URL, 172.18.8.197, a9GuCmeLXAHo97hqzIi4VEO5oqvTjds4y7V6RCpHRsE=): Using
Web Services MCRegistry Instance MCRegistryService(https://172.18.8.197:9443/services/mcsdk10?wsdl, 172.18.8.197, a9GuCmeLXAHo97hqzIi4VEO5oqvTjds4y7V6RCpHRsE=)
2018-05-03 13:44:54,701 INFO  [main]-c.e.a.m.u.MCHostServerConfiguration: MCHostServerConfiguration.getPreferences(): MC Ports:
2018-05-03 13:44:54,702 INFO  [main]-c.e.a.m.u.MCHostServerConfiguration: MCHostServerConfiguration.getPreferences():   Web Services --> 9443
2018-05-03 13:44:54,702 INFO  [main]-c.e.a.m.u.MCHostServerConfiguration: MCHostServerConfiguration.getPreferences():   RMI --> 7778
2018-05-03 13:44:54,703 INFO  [main]-o.a.c.s.f.ReflectionServiceFactoryBean: Creating Service {http://sdk10.mc.avamar.com}mcsdk10 from class com.emc.avamar.modelx.ws.McServicePortType
2018-05-03 13:44:55,398 INFO  [main]-c.e.a.m.u.MCRegistryService: MCRegistryService.connect(null, null): Number of headers: 1
2018-05-03 13:44:55,398 INFO  [main]-c.e.a.m.u.MCRegistryService: MCRegistryService.connect(null, null): Session Key: urn:uuid:F46F10F9B1C7ED9D541525347898450
2018-05-03 13:44:55,399 INFO  [main]-c.e.a.m.u.MCRegistryService: MCRegistryService.connect(null, null): Session Value: SERVICE
2018-05-03 13:44:55,399 INFO  [main]-c.e.a.m.u.MCRegistryService: MCRegistryService.connect(null, null): Session Description: Service Id: urn:uuid:F46F10F9B1C7ED9D541525347898450
2018-05-03 13:44:55,409 INFO  [main]-c.e.a.m.u.MCRegistryService: com.emc.avamar.modelx.util.MCRegistryService.<init>(URL, 172.18.8.197, a9GuCmeLXAHo97hqzIi4VEO5oqvTjds4y7V6RCpHRsE=): Using
Web Services MCRegistry Instance MCRegistryService(https://172.18.8.197:9443/services/mcsdk10?wsdl, 172.18.8.197, a9GuCmeLXAHo97hqzIi4VEO5oqvTjds4y7V6RCpHRsE=)
2018-05-03 13:44:55,485 WARN  [main]-c.e.a.m.u.AvamarModelCoreServerConfiguration: com.emc.avamar.modelx.util.AvamarModelCoreServerConfiguration.init(): 2 backend storage references found.
Only the first one identified as DPN storage will be processed
2018-05-03 13:44:55,486 WARN  [main]-c.e.a.m.u.AvamarModelCoreServerConfiguration: com.emc.avamar.modelx.util.AvamarModelCoreServerConfiguration.init(): Backend Storage Reference that will b
e used for this session is svchgva1-avmr001.trading.aog.grp[DPN storage on svchgva1-avmr001.trading.aog.grp]
2018-05-03 13:44:55,522 INFO  [AvamarSysConfigPolling]-c.e.i.s.g.AvamarGridPolling: Avamar Grid polling running, updating system status.
2018-05-03 13:45:01,092 WARN  [AvamarSysConfigPolling]-c.e.a.m.u.AvamarModelCoreServerConfiguration: com.emc.avamar.modelx.util.AvamarModelCoreServerConfiguration.init(): 2 backend storage r
eferences found.  Only the first one identified as DPN storage will be processed
2018-05-03 13:45:01,093 WARN  [AvamarSysConfigPolling]-c.e.a.m.u.AvamarModelCoreServerConfiguration: com.emc.avamar.modelx.util.AvamarModelCoreServerConfiguration.init(): Backend Storage Ref
erence that will be used for this session is svchgva1-avmr001.trading.aog.grp[DPN storage on svchgva1-avmr001.trading.aog.grp]
2018-05-03 13:45:01,115 INFO  [AvamarSysConfigPolling]-c.e.i.s.g.AvamarGridPolling: Avamar Grid status changed, or sending "heart beat" update.
2018-05-03 13:45:01,133 INFO  [AvamarSysConfigPolling]-c.e.i.s.g.AvamarGridPolling: Sending status event with message:
com.emc.imf.service.grid.messages.MAvamarSystem@3e72ff[
  id=1401879676@00:50:56:BE:6F:5D
  name=SVCHGVA1-AVMR001.TRADING.AOG.GRP
  ipaddress=172.18.8.197
  isonline=true
  totalCapacity=575897
  usedCapacity=40755
  status=FULL_ACCESS
  owningComponentId=0
  activity=<null>
  version=7.2.1-32_HF256506_6






 

Additional Information

If you face the same error of MCS authentication and found all passwords are encrypted correctly as below you can follow the same steps of decrypt then encrypt the passwords again then restart the gridservice: 
root@svchgva1-avmr001:~/#: grep AP /usr/local/avamar/var/mc/server_data/prefs/mcserver.xml
                <entry key="backuprestoreAP" value="{AES}R4cW/x3fVIo4GSBZJhjHlQ==" />
                <entry key="backuponlyAP" value="{AES}EBnkN1YRr4ZfyP9kSAM9Vg==" />
                <entry key="rootAP" value="{AES}/WsuFhewvKy4XqRnr3qiEoQuLd5s1AIXtURY/fT4Ins=" />
                <entry key="MCUSERAP" value="{AES}/WsuFhewvKy4XqRnr3qiEoQuLd5s1AIXtURY/fT4Ins=" />
                <entry key="restoreonlyAP" value="{AES}Fsc/Hdj7FaLUhequP56ZXg==" />
                <entry key="viewuserAP" value="{AES}IWp1I30YOs2Wpa5clVbWQQ==" />

 

Affected Products

Avamar Extended Retention

Products

Avamar Extended Retention, Avamar Media Access Node
Article Properties
Article Number: 000061786
Article Type: Solution
Last Modified: 11 Oct 2024
Version:  3
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.