NVE: Installed Binary Versions in Appliance Do Not Match Installed Upgrade or Rollup Versions

• A NetWorker Virtual Edition Appliance (NVE) is deployed and runs successfully.
• Several NveUpgrade and NvePlatformOsRollup packages are installed.
• The NetWorker Installation Manager user interface displays all the NveUpgrade and NvePlatformOsRollup packages as complete. 
• The below command indicates that all the updates are complete.

avi-cli --verbose localhost --password 'xxxxxxxx' --listhistory

• The system continues to function successfully after each update. 
• When checking the version of binaries, it is observed that the versions in the NveUpgreade and NvePlatformOsRollup packages did not take effect. 
• The NVE displays the version of binaries as the base installation as evidenced by the below command:

rpm -qa |grep <binary_name>

rpm -qa | grep kernel
displays 4.12.14-122.12.1.x86_64: While it should be 4.12.14-122.173.1.x86_64 which was embedded in the latest Rollup.

rpm -qa | grep sqlite3
displays 3.8.10.2-9.15.1.x86_64: While it should be 3.39.3-9.26.1.x86_64 which was embedded in the latest Rollup.

rpm -qa | grep openssl
displays 1.0.2p-3.14.1.x86_64: While it should be 1_1-1.1.1d-2.98.1.x86_64 which was embedded in the latest Rollup.

Further investigation noted that the err.log reports the following error message:

Path to err.log:

/space/avamar/var/avi/server_data/package_data/NvePlatformOsRollup_YYYY-R#-v#.avp_xxxxxxxxxx/err.log

The below command is run during the update process to get the version of the current kernel from the output of the command:

uname -a

YYYY-MM-DD HH:mm:SS (-TZ) 47198950636200 ERROR: "ssh -q -i /root/.ssh/rootid -x -q -o ConnectTimeout=5 -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o GSSAPIAuthentication=no root@<NVE_IP_Address> uname -a", exit status=255 (error)

Since it errors out with "exit status=255 (error)," none of the packages in OsRollup (which consists of base operating system packages mostly) are upgraded, even though the upgrade workflow is successful.

The root cause of this misbehavior is that the PermitRootLogin value for the local host is modified from the default value of "yes" to the incorrect value of "no." This misconfiguration prevented the SSH command (above) to display the installed kernel version.

LogLevel INFO
kexalgorithms ecdh-sha2-nistp384,ecdh-sha2-nistp521
MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-512,hmac-sha2-256-etm@openssh.com,hmac-sha2-256,umac-128-etm@openssh.com,umac-128@openssh.com
PermitEmptyPasswords no
PermitRootLogin no
Match Address ::1,127.0.0.1,127.0.0.1,127.0.0.2,::1,<NVE_IPv4_Address>,<NVE_IPv6_Address>
PermitRootLogin no

To address this issue, do the following:

1. Modify the /etc/ssh/sshd_config file to its default format. The change entails only one line, and the ending section should be changed as follows.

LogLevel INFO
kexalgorithms ecdh-sha2-nistp384,ecdh-sha2-nistp521
MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-512,hmac-sha2-256-etm@openssh.com,hmac-sha2-256,umac-128-etm@openssh.com,umac-128@openssh.com
PermitEmptyPasswords no
PermitRootLogin no
Match Address ::1,127.0.0.1,127.0.0.1,127.0.0.2,::1,<NVE_IPv4_Address>,<NVE_IPv6_Address>
PermitRootLogin yes

2. After the above change is done, reboot the NVE.
3. Run the following command which to display the kernel version:

ssh -q -i /root/.ssh/rootid -x -q -o ConnectTimeout=5 -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o GSSAPIAuthentication=no root@<NVE_IP_ADDRESS> uname -a

nve:~ # ssh -q -i /root/.ssh/rootid -x -q -o ConnectTimeout=5 -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o GSSAPIAuthentication=no root@<NVE_IP_ADDRESS> uname -a
Linux nve 4.12.14-122.153-default #1 SMP Tue Mar 7 14:13:19 UTC 2023 (9f7af45) x86_64 x86_64 x86_64 GNU/Linux

4. Apply the latest NveUpgreade or NvePlatformOsRollup packages to update the version of binaries to the latest versions.
5. Check the version of installed binaries and compare with the versions in the applied NveUpgrade or NvePlatformOsRollup packages using the below command.

rpm -qa

If the symptoms in this KB were observed but the cause is not applicable, see the following KBs for other potential causes which may have resulted in NVE OsRollup not updating packages correctly:

NVE: OS Rollup Reports as Successful but Did Not Update OS Packages when /root/.bashrc Is Modified
NVE: OS Rollup Completed But Did Not Upgrade Kernel after /root permissions were modified
NVE: OS Rollup Reports as Successful but Did Not Update OS Packages After NVE IP Address Change