Avamar: Gen4T Hardware: Avamar upgrade workflow fails while installing Gen4T system tools

The upgrade process fails with the following error in the Avamar Installer (AVinstaller) Web interface: 

Install Gen4T system tools (nn of nnn)

There currently exists process(es) running. Try to re-run the task again in a later time!

Reviewing the log shows that ipmiutil_evt service stopped responding during the startup: 

1. Log in to the Avamar Utility Node as admin

2. Elevate to root privilege.

3. View the process tree using the following command: 

ps -ef --forest

Sample output:

root     20050     1  0 08:51 ?        00:00:00 sh -c sudo -H -u root bash -c "ssh-agent bash /usr/local/avamar/var/run_command-tmpscript.19018.21382508" /usr/local/avamar/var/run_command-outer-output.19018.21382508 2>&1
root     20052 20050  0 08:51 ?        00:00:00  \_ sudo -H -u root bash -c ssh-agent bash /usr/local/avamar/var/run_command-tmpscript.19018.21382508
root     20053 20052  0 08:51 ?        00:00:00      \_ bash /usr/local/avamar/var/run_command-tmpscript.19018.21382508
root     20054 20053  0 08:51 ?        00:00:00          \_ ssh-agent bash /usr/local/avamar/var/run_command-tmpscript.19018.21382508
root     20082 20053  0 08:51 ?        00:00:00          \_ /usr/bin/perl /data01/avamar/repo/temp/AvamarUpgrade-18.1.0-33.avp_1535632333454/tools/mapall 
root     20089 20082  0 08:51 ?        00:00:00              \_ ssh -x -o GSSAPIAuthentication=no root@x.x.x.30 /usr/local/avamar/src/gen4t-sys-1.9/avsetup.sh
...
root     20090 17879  0 08:51 ?        00:00:00  \_ sshd: root@notty
root     20098 20090  0 08:51 ?        00:00:00      \_ /bin/bash /usr/local/avamar/src/gen4t-sys-1.9/avsetup.sh
root     20619 20098  0 08:52 ?        00:00:00          \_ /bin/bash /usr/local/avamar/src/gen4t-sys-1.9/sys/avsetup.sh
root     20730 20619  0 08:52 ?        00:00:00          |   \_ /bin/rpm -Uvh /usr/local/avamar/src/gen4t-sys-1.9/sys/ipmiut
root     20734 20730  0 08:52 ?        00:00:00          |       \_ /bin/sh /var/tmp/rpm-tmp.31162 1
root     20738 20734  0 08:52 ?        00:00:00          |           \_ /bin/sh /sbin/service ipmiutil_evt start
root     20740 20738  0 08:52 ?        00:00:00          |               \_ /bin/sh /etc/init.d/ipmiutil_evt start
root     20743 20740  0 08:52 ?        00:00:01          |                   \_ /usr/sbin/ipmiutil getevt -s -b -t 0
root     20620 20098  0 08:52 ?        00:00:00          \_ /usr/bin/tee -a /tmp/gen4t-sys_20098.log

In the above output, the Avamar upgrade starts the avsetup.sh script to install the gen4T system tools. The avsetup is waiting for the ipmiutil command to return. 

The process tree also gives the location of the log for the Gen4T system tool installation (In this scenario /tmp/gen4t-sys_20098.log). 

4. The log contents shows that ipmiutil_evt service stopped responding during the startup: 

* ipmiutil-emc
**********
Signature   : (none)
INFO: Uninstalling unsigned ipmiutil-emc-2.7.9-8 RPM ...
INFO: Uninstalled ipmiutil-emc-2.7.9-8 successfully.
INFO: Installing ipmiutil-emc-2.7.9-8.x86_64.rpm ...
Preparing...                ##################################################
ipmiutil-emc                ##################################################
ipmiutil_evt              0:off  1:off  2:off  3:off  4:off  5:off  6:off
Starting ipmiutil_evt:

The root cause is unknown.

Create a Serice Request for assistance from the Dell Technologies Avamar Support team referencing this article.