Highlighted
connolly
3 Argentium

Avamar PDM fails with internal error, logs show "lockbox" issue?

Trying to run the Proxy Deployment Manager on an Avamar v7.5 system and it throws an "Internal Error" when trying to create a recommendation - log has the following entries:

2018-02-19T09:58:42.399-0600 TRACE [TaskExecutor-10][ impl.ConnectionManagerService#    isValidTicket]: Checking if connection manager ticket is valid: b287fcab-7278-4729-809e-2923fbbf2e68

2018-02-19T09:58:42.401-0600 TRACE [TaskExecutor-10][ impl.ConnectionManagerService# waitForMcSession]: Requesting connection manager mc session

2018-02-19T09:58:42.404-0600 TRACE [TaskExecutor-10][ impl.ConnectionManagerService# waitForMcSession]: Acquired MC session: ...f3a17fc39615

2018-02-19T09:58:42.405-0600 INFO  [TaskExecutor-10][              util.LockboxUtil#    initializeCST]: Initializing CST Lockbox: /etc/vcs/cst

2018-02-19T09:58:42.410-0600 ERROR [TaskExecutor-10][           impl.LockboxService#DefaultMcPassword]: Unexpected exception

com.emc.csp.error.IOException: The Lockbox stable value threshold was not met because the system fingerprint has changed. To reset the system fingerprint, open the Lockbox using the passphrase. : File : /etc/vcs/cst/csp.clb

I can't figure out what "lockbox" they are talking about. And I thought Avamar got rid of its lockbox?

We have used the PDM at this site to deploy proxies before without any issues.

There hasn't been anything done recently that I can think of to change anything siginificant with respect to the Avamar and its vCenter integration that would cause this - not sure what might have been done in the VMware environment.

FWIW - VM backups completed successfully over the weekend, and I was able to browse the vCenter by "pretending" to add a new VM client using the same Avamar GUI I am running the PDM from.

Anyone seen this before?

All comments/feedback appreciated - thanks.

Reply
3 Replies
Highlighted
connolly
3 Argentium

Re: Avamar PDM fails with internal error, logs show "lockbox" issue?

Potentially related question - does anyone know of any Avamar side authentication or certificate dependency that would affect how the Proxy Deployment Manager worked, but would not affect regular backup or replication operations?

The log entries seem to indicate that this issue arose over the last week - and we were doing some troubleshooting of Avamar replication and backups, but most of it was focused on the Avamar integration with the Data Domain.

Just trying to figure out what if any common thread there might be here.

Reply
Highlighted
Danny101
1 Copper

Re: Avamar PDM fails with internal error, logs show "lo

Hello,

Did you ever get past this issue? I have the same problem with the same error in the log

Reply
genecent
1 Copper

Re: Avamar PDM fails with internal error, logs show "lo

I had this issue and resolved it with the steps here: https://community.emc.com/docs/DOC-79612

Login to the Avamar server as admin user.
Switch to root user (su - root)

Step 1. Backup your previous lockbox files
Only delete or move /etc/vcs/cst/csp.clb* files.

Step 2. Create a new lockbox (I would recommend sticking with the current passphrase as otherwise you may have new troubles)
/usr/local/rsa_cst/cst/builds/linux_gcc34_x64_r/lib/cstadmin initialize /etc/vcs/cst
Enter in the passphrase twice
The above command will create a new lockbox with a specific passphrase.

Note: The current passphrase can be fetched using the below command:
dmidecode | grep -i uuid | awk -F 'UUID: ' '{print "v"$2"*"}'

Step 3. Add your mc_root PW to the lockbox, hopefully you know what it is
java -jar /usr/local/rsa_cst/cst/bin/lockbox-service.jar create 'mc_root@127.0.0.1.pw' '<enter root password to mc here>'
The above command will create an entry in the lockbox for the mc root user.

Note: The root MC password is the Avamar GSAN root password (usually common) found in mcserver.xml, you may need to decrypt this following https://support.emc.com/kb/332566.

You can test a password that you think it may be with this command: mccipher verify -p mcserver:/usr/local/avamar/var/mc/server_data/prefs:com/avamar/mc/dpn/users/rootAP -t <your best PW guess>


Step 4. Restart emwebapp (as root):

root@idpaave:~/#: emwebapp.sh --restart

Try creating your PDM recommendation again. 🙂


-William

Reply