NVE: Upgrade fails during "Configuring AuthC service for NetWorker"

The NVE upgrade process fails when configuring the authc service for NetWorker:

The NVE upgrade process finds the authc_config.txt.preserved but fails with an unknown error. The NVE /data01/avamar/repo/temp/*/tmp/workflow.log shows:

YYYY-MM-DD HH:MM:SS (-TZ) 47133243251360 INFO: ---------- BEGIN action.reconfigure-networker-authc-service::do ----------
YYYY-MM-DD HH:MM:SS (-TZ) 47133243251360 INFO: >>> [make_temp_script] 222 very_current_user=root, as_user=root, ssh command is: (sed -i 's/EMC_JAVADIR=\/usr\/java\/latest/EMC_JAVADIR=\/opt\/nre\/java\/latest/Ig' /nsr/authc_config.txt.preserved) >/usr/local/avamar/var/run_command-sysout.13101.47133243251360 2>&1
YYYY-MM-DD HH:MM:SS (-TZ) 47133243251360 INFO: no output seen from "sed -i 's/EMC_JAVADIR=\/usr\/java\/latest/EMC_JAVADIR=\/opt\/nre\/java\/latest/Ig' /nsr/authc_config.txt.preserved"
YYYY-MM-DD HH:MM:SS (-TZ) 47133243251360 INFO: "sed -i 's/EMC_JAVADIR=\/usr\/java\/latest/EMC_JAVADIR=\/opt\/nre\/java\/latest/Ig' /nsr/authc_config.txt.preserved", exit status=0 (success)
YYYY-MM-DD HH:MM:SS (-TZ) 47133254743080 INFO: >>> [make_temp_script] 222 very_current_user=root, as_user=root, ssh command is: (/opt/nsr/authc-server/scripts/authc_configure.sh -silent /nsr/authc_config.txt.preserved) >/usr/local/avamar/var/run_command-sysout.13101.47133254743080 2>&1
YYYY-MM-DD HH:MM:SS (-TZ) 47133243251360 INFO: - - - - - - - BEGIN(output) - - - - - - -
YYYY-MM-DD HH:MM:SS (-TZ) 47133243251360 INFO: 
YYYY-MM-DD HH:MM:SS (-TZ) 47133243251360 INFO: WARNING: Port 9090 is already in use.
YYYY-MM-DD HH:MM:SS (-TZ) 47133243251360 INFO: - - - - - - - END
YYYY-MM-DD HH:MM:SS (-TZ) 47133243251360 ERROR: Configuring AuthC service for NetWorker, exit status=1 (error)
YYYY-MM-DD HH:MM:SS (-TZ) 47133243251360 INFO: post: Configuring AuthC service for NetWorker, exit status=1 (error)
YYYY-MM-DD HH:MM:SS (-TZ) 47133243251360 INFO: >>> [make_temp_script] 222 very_current_user=root, as_user=, ssh command is: (rpm -q dpnserver --last | head -1) >/usr/local/avamar/var/run_command-sysout.13101.47133243251360 2>&1
YYYY-MM-DD HH:MM:SS (-TZ) 47133243251360 INFO: >>> [make_temp_script] 222 very_current_user=root, as_user=admin, ssh command is: (/usr/local/avamar/bin/avhostname -f) >/usr/local/avamar/var/run_command-sysout.13101.47133243251360 2>/usr/local/avamar/var/run_command-syserr.13101.47133243251360

The /nsr/authc_config.txt.preserved contains an incorrect entry for EMC_TCUSER. For example:

nve:~ # cat /nsr/authc_config.txt.preserved
EMC_JAVADIR=/opt/nre/java/latest
EMC_TCUSER=tomcat
EMC_TCPORT=9090
EMC_CREATETCKEYSTORE=n
EMC_TCKEYSTORE=/nsr/authc/conf/authc.keystore
EMC_DATADIR=/nsr/authc/data
EMC_KEEPBOOT=y
nve:~ #

During an NVE upgrade, the upgrade workflow runs the NetWorker authc_configure.sh script using the values defined in /nsr/authc_config.txt.preserved. The expected value for EMC_TCUSER is nsrtomcat.

nve:~ # cat /etc/passwd | grep tomcat
nsrtomcat:x:1003:100::/home/nsrtomcat:/bin/bash

Setting any other user account causes the upgrade to fail if the user does not exist; if the user account exists, changing authc ownership to another account can break authc. 

1. Log in to the NVE over SSH as the user admin.
2. Switch to root: sudo su -
3. Modify the /nsr/authc_config.txt.preserved to have the correct EMC_TCUSER account:

nve:~ # vi /nsr/authc_config.txt.preserved

EMC_JAVADIR=/opt/nre/java/latest
EMC_TCUSER=nsrtomcat
EMC_TCPORT=9090
EMC_CREATETCKEYSTORE=n
EMC_TCKEYSTORE=/nsr/authc/conf/authc.keystore
EMC_DATADIR=/nsr/authc/data
EMC_KEEPBOOT=y

4. Save the file: [ESC] :wq!
5. Return to the NVE web UI and retry the task.

Debugging:

Append -x to the authc_configure.sh #!/bin/bash line, example:

nve:~ # head /opt/nsr/authc-server/scripts/authc_configure.sh
#! /bin/bash -x
#
# $Id$ Copyright (c) 2015-2023 EMC Corporation.
#

#
# Copyright (c) 2015-2023 EMC Corporation.
#
# All rights reserved.  This is an UNPUBLISHED work, and
# comprises proprietary and confidential information of EMC.
nve:~ #

Retry the task and review the following logs on the NVE:

• /data01/avamar/repo/temp/*/tmp/workflow.log
• /data01/avamar/repo/temp/*/tmp/err.log
• /space/avamar/var/avi/server_log/avinstaller.log

For the cause defined in this KB, the following was logged in the workflow.log once -x debug was enabled:

nve:~ # cat /data01/avamar/repo/temp/*/tmp/workflow.log | grep "username\|TCUSER"
2024-07-11 12:56:40 (-0400) 47274291182240 INFO: cmd=/space/avamar/repo/temp/NveUpgrade-19.11.0-12.avp_1720716526720/scripts/workflow run tools/webmonitor.rb --server=localhost --username=root --password=*********** --validate_only=true
2024-07-11 12:56:40 (-0400) 47274291182240 INFO: >>> [make_temp_script] 222 very_current_user=root, as_user=root, ssh command is: (/space/avamar/repo/temp/NveUpgrade-19.11.0-12.avp_1720716526720/scripts/workflow run tools/webmonitor.rb --server=localhost --username=root --password=*********** --validate_only=true) >/usr/local/avamar/var/run_command-sysout.15932.47274291182240 2>&1
2024-07-11 12:56:41 (-0400) 47436079741600 NOTICE: Utility: tools/webmonitor.rb --server=localhost --username=root --password=********** --validate_only=true
2024-07-11 12:56:42 (-0400) 47274291182240 INFO: "/space/avamar/repo/temp/NveUpgrade-19.11.0-12.avp_1720716526720/scripts/workflow run tools/webmonitor.rb --server=localhost --username=root --password=*********** --validate_only=true", exit status=0 (success)
2024-07-11 13:04:59 (-0400) 46986367916700 INFO: + DEFAULT_TCUSER=nsrtomcat
2024-07-11 13:04:59 (-0400) 46986367916700 INFO: + EMC_CREATETCUSER=n
2024-07-11 13:04:59 (-0400) 46986367916700 INFO: ++ awk /EMC_TCUSER/ /nsr/authc_config.txt.preserved
2024-07-11 13:04:59 (-0400) 46986367916700 INFO: + username=tomcat
2024-07-11 13:04:59 (-0400) 46986367916700 INFO: ++ awk /EMC_CREATETCUSER/ /nsr/authc_config.txt.preserved
2024-07-11 13:04:59 (-0400) 46986367916700 INFO: + TCUSER_HOME=

Extra Resources:

• NetWorker Virtual Edition Upgrade to 19.8.0.108 Fails at Reconfiguring NetWorker Management Console Settings
• NetWorker Virtual Edition upgrade task fails during "Reconfiguring NetWorker Authc Service Settings (36 of 47)"
• NVE: Upgrade or Install Failure Triage Guide