IDPA Upgrade to 2.3 fails on Avamar server with error "Avamar EBR Server initialization failed" caused by Lockbox reset failure on Avamar Server

Summary: This KB walks you through an upgrade failure issue on Avamar while upgrading IDPA to version 2.3. The issue may happen when ACM tries to reset the lockbox on Avamar during the Avamar Upgrade process. ...

This article applies to This article does not apply to This article is not tied to any specific product. Not all product versions are identified in this article.
Check out other resources

Symptoms

IDPA Upgrade to 2.3 fails on Avamar server with error " Avamar EBR Server initialization failed". Lockbox reset failure on Avamar Server causes this on Avamar while installing Avsecurity rollup.

In "avAppLevelUpgrade.log" log file, we see the below error message:

RUNNING,17,Package AvPlatformOsRollup_2018-R4-v5 is ready for installation
RUNNING,18,Started installation of Backup server packages. This may take 40 minutes
RUNNING,19,AvamarUpgrade1820,AvPlatformOsRollup_2018-R4-v5 avp installation failed
RUNNING,20,Initializing Avamar EBR server
RUNNING,21,Avamar EBR Server initialization Failed
RUNNING,25,Successfully upgraded Backup Server
RUNNING,25,Preparing to upgrade Backup Server hotfix
RUNNING,26,Ready to upgrade Backup Server hotfix
RUNNING,26,Upgrading Backup Server hotfix


In "upgradeutility.log" log file,we see the following error:

2019-02-06 21:13:04,114 INFO  [upgrade-workflow-5]-util.ProcessBuilderExecutor: ERROR: main: another package is already active
2019-02-06 21:13:04,114 INFO  [upgrade-workflow-5]-av.AvamarUpgradeOperations: Failed to execute hot fix script. responseCode: 1
2019-02-06 21:13:04,114 INFO  [upgrade-workflow-5]-upgradeutil.UpgradeUtil: Getting latest progress percent from: /data01/tmp/patch/logs/status/avAppLevelUpgrade.log
2019-02-06 21:13:04,114 INFO  [upgrade-workflow-5]-upgradeutil.UpgradeUtil: prevPercent: 52 currentPercent: 44
2019-02-06 21:13:04,114 ERROR [upgrade-workflow-5]-av.AvamarRetryTask: Failed to upgrade Avamar.
com.emc.vcedpa.upgradeutil.common.exception.UpgradeException: Unable to upgrade Backup Server.
        at com.emc.vcedpa.upgradeutil.configure.av.AvamarUpgradeOperations.aveHotFix(AvamarUpgradeOperations.java:222)
        at com.emc.vcedpa.upgradeutil.configure.av.AvamarUpgradeOperations.upgrade(AvamarUpgradeOperations.java:166)
        at com.emc.vcedpa.upgradeutil.configure.av.AvamarRetryTask.run(AvamarRetryTask.java:56)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)


On ACM GUI, we see the below Warning messages:

kA2f100000008EYCAY_3_0

Cause

This failures happens due to a lockbox reset failure on Avamar server. 

On the Avamar server, avinstaller.log.0 under path (/usr/local/avamar/var/avi/server_log) shows the below error:

INFO: PollMessageSender sent: <UpdateMessage><ProcessKey>AvamarUpgrade20190206Feb021549504018</ProcessKey><ProcessInstanceId>AvamarUpgrade20190206Feb021549504018.60013</ProcessInstanceId><Timestamp>2019/02/06-21:05:56.00982</Timestamp><Content>Unable to reset lockbox - operation failed</Content></UpdateMessage>

Feb 06, 2019 9:05:57 PM com.avamar.avinstaller.process.handler.ServerScriptHandler$TaskProcessor run
WARNING: From err out: 02/06 21:05:56 error: Unable to reset lockbox - operation failed


Additionally, the failure can be checked on Avamar Installation manager page to confirm the error message.

Resolution

Login to the Avamar server as admin user.

Switch to root user (su - root)

Step 1.
    Only delete or move /etc/vcs/cst/csp.clb* files.

Step 2.
    /usr/local/rsa_cst/cst/builds/linux_gcc34_x64_r/lib/cstadmin initialize /etc/vcs/cst
    Enter in the passphrase twice
    The above command will create a new lockbox with a specific passphrase.

    Note: The current passphrase can be fetched using the below command:
    dmidecode | grep -i uuid | awk -F 'UUID: ' '{print "v"$2"*"}'
 
Step 3.
    java -jar /usr/local/rsa_cst/cst/bin/lockbox-service.jar create 'mc_root@127.0.0.1.pw' '<enter root password to mc here>'
    The above command will create an entry in the lockbox for the mc root user.

Note: The root MC password is the Avamar GSAN root password (usually common) found in mcserver.xml, you may need to decrypt this 

Step 4.
   Restart emwebapp (as root): 
root@idpaave:~/#: emwebapp.sh --restart
			
			
			Click on Retry button on ACM to reattempt the Upgrade on Avamar
 

Additional Information

This content is translated in other languages: 
https://downloads.dell.com/TranslatedPDF/PT-BR_KB530270.pdf
https://downloads.dell.com/TranslatedPDF/ZH-CN_KB530270.pdf
https://downloads.dell.com/TranslatedPDF/DE_KB530270.pdf
https://downloads.dell.com/TranslatedPDF/ES_KB530270.pdf
https://downloads.dell.com/TranslatedPDF/FR_KB530270.pdf
https://downloads.dell.com/TranslatedPDF/JA_KB530270.pdf
https://downloads.dell.com/TranslatedPDF/IT_KB530270.pdf
https://downloads.dell.com/TranslatedPDF/KO_KB530270.pdf
https://downloads.dell.com/TranslatedPDF/NL_KB530270.pdf
https://downloads.dell.com/TranslatedPDF/PT_KB530270.pdf
https://downloads.dell.com/TranslatedPDF/RU_KB530270.pdf
https://downloads.dell.com/TranslatedPDF/SV_KB530270.pdf

Note: Only registered Dell Customers can access the content on the above link, via Dell.com/support.

Affected Products

Integrated Data Protection Appliance Family

Products

PowerProtect Data Protection Software, Integrated Data Protection Appliance Family, Integrated Data Protection Appliance Software
Article Properties
Article Number: 000170421
Article Type: Solution
Last Modified: 09 Sep 2024
Version:  4
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.