Avamar: EMT Shows Down Due to /etc/profile Missing

The EMT shows down using dpnctl status emt although the tomcat shows up:

root@ave-srv:~/#: dpnctl status emt
Identity added: /home/admin/.ssh/admin_key (/home/admin/.ssh/admin_key)
dpnctl: INFO: emt status: down.
dpnctl: INFO: [see log file "/usr/local/avamar/var/log/dpnctl.log"]
root@ave-srv:~/#: 
root@ave-srv:~/#:  tomcatctl start
tomcatctl: INFO: EM Tomcat start succeeded
root@ave-srv:~/#: 

admin@ave-srv:~/>: emserver.sh --test
INFO: Enterprise Database status: up
admin@ave-srv:~/>:

admin@ave-srv:~/>: emwebapp.sh --test
INFO: Enterprise Manager web application status: up
admin@ave-srv:~/>:

root@ave-srv:~/#: dpnctl status emt
Identity added: /home/admin/.ssh/admin_key (/home/admin/.ssh/admin_key)
dpnctl: INFO: emt status: down.
dpnctl: INFO: [see log file "/usr/local/avamar/var/log/dpnctl.log"]
root@ave-srv:~/#: 

The following error message is seen during EMT start with the following command as admin:

admin@ave-srv:~/>: sudo -A -H /usr/local/avamar/bin/tomcatctl status --verbose
tomcatctl: ERROR: missing required subsystem control for "EM Tomcat"
admin@ave-srv:~/>:

admin@ave-srv:~/>: dpnctl start emt --verbose
Using /usr/local/avamar/var/probe.xml
dpnctl: INFO: dpnctl_util::set_verbose 1
dpnctl: INFO: dpnctl_cleanup::set_verbose 1
        /usr/bin/find /tmp -maxdepth 1 -type f \( -name 'dpnctl-*.[0-9]*' \) -mtime +1 -uid 500 -exec rm -f {} \;
dpnctl: INFO: dpnctl_node::set_verbose 1
dpnctl: INFO: interactive context = true
dpnctl: INFO: dpnctl_identity::set_verbose 1
        (sudo -A -H /usr/local/avamar/lib/dpnutils/unattended-restart-ctl supported --verbose ; echo $? >/tmp/dpnctl-subsystem-control-action-status-23375) >/tmp/dpnctl-subsystem-control-action-output-23375 2>&1
        /bin/cat /tmp/dpnctl-subsystem-control-action-output-23375
[ "/bin/cat /tmp/dpnctl-subsystem-control-action-output-23375" exit status = 0 ]
        /bin/cat /tmp/dpnctl-subsystem-control-action-status-23375 2>&1
[ "/bin/cat /tmp/dpnctl-subsystem-control-action-status-23375 2>&1" exit status = 0 ]
        rm -f /tmp/dpnctl-subsystem-control-action-status-23375 /tmp/dpnctl-subsystem-control-action-output-23375
        (sudo -A -H /usr/local/avamar/lib/dpnutils/unattended-restart-ctl mark --verbose ; echo $? >/tmp/dpnctl-subsystem-control-action-status-23375) >/tmp/dpnctl-subsystem-control-action-output-23375 2>&1
        /bin/cat /tmp/dpnctl-subsystem-control-action-output-23375
[ "/bin/cat /tmp/dpnctl-subsystem-control-action-output-23375" exit status = 0 ]
        /bin/cat /tmp/dpnctl-subsystem-control-action-status-23375 2>&1
[ "/bin/cat /tmp/dpnctl-subsystem-control-action-status-23375 2>&1" exit status = 0 ]
output of "sudo -A -H /usr/local/avamar/lib/dpnutils/unattended-restart-ctl mark --verbose":
- - - - - - - - - - - - - - - BEGIN
- - - - - - - - - - - - - - - END
        rm -f /tmp/dpnctl-subsystem-control-action-status-23375 /tmp/dpnctl-subsystem-control-action-output-23375
Using /usr/local/avamar/var/probe.xml
dpnctl_ssh_context::set_verbose 1
[ dpnctl_ssh_context::have_ssh_context(): checking agent status ]
        /usr/bin/ssh-add -l
521 SHA256:uyzWbemBqkPJMI4rFPwJaJIgjRK7EmHq83DirzHGURQ /home/admin/.ssh/id_rsa (ECDSA)
[ dpnctl_ssh_context::have_ssh_context(): checking agent status ]
        /usr/bin/ssh-add -l
521 SHA256:uyzWbemBqkPJMI4rFPwJaJIgjRK7EmHq83DirzHGURQ /home/admin/.ssh/id_rsa (ECDSA)
        /usr/bin/ssh-add -l
521 SHA256:uyzWbemBqkPJMI4rFPwJaJIgjRK7EmHq83DirzHGURQ /home/admin/.ssh/id_rsa (ECDSA)
        /usr/bin/ssh-add /home/admin/.ssh/admin_key
Identity added: /home/admin/.ssh/admin_key (/home/admin/.ssh/admin_key)
        /usr/bin/ssh-add -l
521 SHA256:uyzWbemBqkPJMI4rFPwJaJIgjRK7EmHq83DirzHGURQ /home/admin/.ssh/admin_key (ECDSA)
dpnctl: INFO: dpnctl_notification::set_verbose 1
        (sudo -A -H /usr/local/avamar/bin/tomcatctl status --verbose --verbose ; echo $? >/tmp/dpnctl-subsystem-control-action-status-23375) >/tmp/dpnctl-subsystem-control-action-output-23375 2>&1
        /bin/cat /tmp/dpnctl-subsystem-control-action-output-23375
[ "/bin/cat /tmp/dpnctl-subsystem-control-action-output-23375" exit status = 0 ]
        /bin/cat /tmp/dpnctl-subsystem-control-action-status-23375 2>&1
[ "/bin/cat /tmp/dpnctl-subsystem-control-action-status-23375 2>&1" exit status = 0 ]
        rm -f /tmp/dpnctl-subsystem-control-action-status-23375 /tmp/dpnctl-subsystem-control-action-output-23375
gsan status check:
- - - - - - - - - - - - - - - BEGIN
server up: idle
- - - - - - - - - - - - - - - END
dpnctl: INFO: interactive context = true
        (sudo -A -H /usr/local/avamar/bin/tomcatctl status --verbose --verbose ; echo $? >/tmp/dpnctl-subsystem-control-action-status-23375) >/tmp/dpnctl-subsystem-control-action-output-23375 2>&1
        /bin/cat /tmp/dpnctl-subsystem-control-action-output-23375
[ "/bin/cat /tmp/dpnctl-subsystem-control-action-output-23375" exit status = 0 ]
        /bin/cat /tmp/dpnctl-subsystem-control-action-status-23375 2>&1
[ "/bin/cat /tmp/dpnctl-subsystem-control-action-status-23375 2>&1" exit status = 0 ]
        rm -f /tmp/dpnctl-subsystem-control-action-status-23375 /tmp/dpnctl-subsystem-control-action-output-23375
dpnctl: INFO: To monitor progress, run in another window: tail -f /tmp/dpnctl-subsystem-control-action-output-23375
        (sudo -A -H /usr/local/avamar/bin/tomcatctl start --verbose --verbose ; echo $? >/tmp/dpnctl-subsystem-control-action-status-23375) >/tmp/dpnctl-subsystem-control-action-output-23375 2>&1
        /bin/cat /tmp/dpnctl-subsystem-control-action-output-23375
[ "/bin/cat /tmp/dpnctl-subsystem-control-action-output-23375" exit status = 0 ]
        /bin/cat /tmp/dpnctl-subsystem-control-action-status-23375 2>&1
[ "/bin/cat /tmp/dpnctl-subsystem-control-action-status-23375 2>&1" exit status = 0 ]
output of "sudo -A -H /usr/local/avamar/bin/tomcatctl start --verbose --verbose":
- - - - - - - - - - - - - - - BEGIN
command: if [[ -r /etc/profile && ( -z $JAVA_HOME || ! $PATH =~ "avamar/bin" ) ]] ; then source /etc/profile ; fi ; which emwebapp.sh
output:
which: no emwebapp.sh in (/usr/bin:/bin:/usr/sbin:/sbin)
exit status: 1
tomcatctl: ERROR: missing required subsystem control for "EM Tomcat"
- - - - - - - - - - - - - - - END
dpnctl: ERROR: error return from "sudo -A -H /usr/local/avamar/bin/tomcatctl start --verbose --verbose" - exit status 1
dpnctl: ERROR: 1 error seen in output of "sudo -A -H /usr/local/avamar/bin/tomcatctl start --verbose --verbose"
        rm -f /tmp/dpnctl-subsystem-control-action-status-23375 /tmp/dpnctl-subsystem-control-action-output-23375
dpnctl: ERROR: error return from starting EM Tomcat component - exit status 1
dpnctl: INFO: [user "admin"] for exit status 1, incrementing dpnctl program exit status to 1
dpnctl: INFO: No /usr/local/avamar/var/dpn_service_status exist.
dpnctl: INFO: [see log file "/usr/local/avamar/var/log/dpnctl.log"]
        rm -f /usr/local/avamar/var/dpnctl.lck
dpnctl: ERROR: traceback on exit:
dpnctl_util::quit (/usr/local/avamar/bin/dpnctl line 6495)
        rm -f /usr/local/avamar/var/dpnctl.lck
dpnctl: ERROR: [user "admin"] program exit status = 1 (error)
admin@ave-srv:~/>: 

Checking some lines in the above error message, it shows:

command: if [[ -r /etc/profile && ( -z $JAVA_HOME || ! $PATH =~ "avamar/bin" ) ]] ; then source /etc/profile ; fi ; which emwebapp.sh

The echo $PATH and which emwebapp.sh commands show correct:

admin@ave-srv:~/>: echo $PATH
/opt/emc-tools/bin:/usr/local/avamar/bin:/sbin:/usr/sbin:/usr/local/sbin:/home/admin/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/apache/bin:/usr/local/ssl/bin
admin@ave-srv:~/>:

admin@ave-srv:~/>: which emwebapp.sh
/usr/local/avamar/bin/emwebapp.sh
admin@ave-srv:~/>:

However, the echo "JAVA_HOME=$JAVA_HOME" shows blank:

admin@ave-srv:~/>: echo "JAVA_HOME=$JAVA_HOME"
JAVA_HOME=
admin@ave-srv:~/>:

The java directory is correct:

admin@ave-srv:~/>: ls -ld /usr/java/latest
lrwxrwxrwx 1 root root 31 Jun 17 14:07 /usr/java/latest -> /usr/lib/jvm/jre-1.8-oracle-x64
admin@ave-srv:~/>:

Checking the JAVA_HOME in /etc/profile and found that the file is empty:

admin@ave-srv:~/>: grep -i JAVA_HOME /etc/profile
admin@ave-srv:~/>:
admin@ave-srv:/etc/>: cat profile
admin@ave-srv:/etc/>:

The issue was caused due to /etc/profile and JAVA_HOME variable missing.

1. Check the Avamar type and version.
2. If Avamar is Multi-Node, contact Dell Technical support and provide this article 000484149 to support follows the internal notes section, otherwise proceed to next step
3. Check if there is another environment with the same type and version
4. Copy or create the /etc/profile file
5. Run the following commands as root:

chown root:root /etc/profile
chmod 644 /etc/profile
grep -i JAVA_HOME /etc/profile
export JAVA_HOME=/usr/java/latest/
echo $JAVA_HOME

6. Output shows:

root@ave-srv:~/#: grep -i JAVA_HOME /etc/profile
export JAVA_HOME=/usr/java/latest/
root@ave-srv:~/#: 
root@ave-srv:~/#: export JAVA_HOME=/usr/java/latest/
root@ave-srv:~/#: 
root@ave-srv:~/#: echo $JAVA_HOME
/usr/java/latest/
root@ave-srv:~/#: 

7. The EMT should show as up:

admin@ave-srv:~/>: dpnctl status
Identity added: /home/admin/.ssh/admin_key (/home/admin/.ssh/admin_key)
dpnctl: INFO: gsan status: up
dpnctl: INFO: MCS status: up.
dpnctl: INFO: emt status: up.
...
admin@ave-srv:~/>: