NVE: OS Rollup Reports as Successful but Did Not Update OS Packages After NVE IP Address Change

• Security vulnerabilities (CVEs) may be reported against the NVE. An OS Rollup was already applied to address the vulnerabilities.
• OS Rollup packages are listed as completed in the avinstaller History tab:

There are two potential causes for this issue. One or both of these causes may be present on the NVE.

Cause 1:

The rollup's workflow log reports that it fails to get the system information. It attempts to use the wrong IP address to connect to the NVE and check the operating system details prior to updating:

nve:~ # cat /usr/local/avamar/var/avi/server_data/package_data/NvePlatformOsRollup_2024-R2-v3.avp_*/workflow.log | grep -A2 "uname"
2024-09-20 13:24:27 (-0400) 47106469567160 INFO: post: "ssh -q -i /root/.ssh/rootid -x -q -o ConnectTimeout=5 -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o GSSAPIAuthentication=no  root@123.123.9.16  uname -a", exit status=255 (error)
2024-09-20 13:24:27 (-0400) 47106469567160 INFO: address=123.123.9.16 remote=ssh -q -i /root/.ssh/rootid -x -q -o ConnectTimeout=5 -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o GSSAPIAuthentication=no  root@123.123.9.16  os=unknown version=unknown update=unknown arch=unknown
2024-09-20 13:24:27 (-0400) 47106469567160 INFO:  - One or more nodes are not running SLES succeeded

The IP address shown is not the current IP address of the NVE appliance. It may be an older IP address used previously.

nve:~ # ifconfig -a
eth0      Link encap:Ethernet  HWaddr 00:50:56:BB:B4:63
          inet addr:123.123.0.16  Bcast:123.123.0.255  Mask:255.255.255.0
          inet6 addr: fe80::250:56ff:febb:b463/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:2713 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1309 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:492699 (481.1 Kb)  TX bytes:1463814 (1.3 Mb)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:21317 errors:0 dropped:0 overruns:0 frame:0
          TX packets:21317 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:5696590 (5.4 Mb)  TX bytes:5696590 (5.4 Mb)

The rollup is getting the wrong IP address from the /usr/local/avamar/var/probe.xml.

nve:~ # cat /usr/local/avamar/var/probe.xml
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<dpn>
   <module name="nve" userinput_domain="networker.lan" userinput_search="networker.lan">
      <node type="single-node server">
         <network-interface id="0" userinput_ifname="eth0">
            <address value="123.123.9.16" userinput_netmask="255.255.255.0" userinput_customhostname="nve.networker.lan"/>
            <uses allow="replication,internal,backup,management"/>
         </network-interface>
      </node>
   </module>
</dpn>

When running the same command from the workflow, it does not return uname -a output.

ssh -q -i /root/.ssh/rootid -x -q -o ConnectTimeout=5 -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o GSSAPIAuthentication=no  root@WRONG_IP_FROM_WORKFLOW.LOG  uname -a

Example:

nve:~ # ssh -q -i /root/.ssh/rootid -x -q -o ConnectTimeout=5 -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o GSSAPIAuthentication=no  root@123.123.9.16  uname -a
nve:~ #

Cause 2:

The rollup's workflow log reports that it fails to get the system information. It attempts to use the correct IP address to connect to the NVE and check the operating system details prior to updating but does not get the right OS details:

2024-10-16 10:44:08 (-0400) 47438611020440 INFO: post: "ssh -q -i /root/.ssh/rootid -x -q -o ConnectTimeout=5 -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o GSSAPIAuthentication=no  root@123.123.0.16  uname -a", exit status=255 (error)
2024-10-16 10:44:08 (-0400) 47438611020440 INFO: address=123.123.0.16 remote=ssh -q -i /root/.ssh/rootid -x -q -o ConnectTimeout=5 -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o GSSAPIAuthentication=no  root@123.123.0.16  os=unknown version=unknown update=unknown arch=unknown
2024-10-16 10:44:08 (-0400) 47438611020440 INFO:  - One or more nodes are not running SLES succeeded

The /etc/ssh/sshd_config contains the old (wrong IP) in the Match Address line:

Match Address ::1,127.0.0.1,127.0.0.1,::1,123.123.9.16,fe80::250:56ff:febb:8bdb
        PermitRootLogin yes

Running the uname -a command from the workflow, repeatedly asks for the root password, even when the correct IP address and root password are used:

nve:~ # ssh -q -i /root/.ssh/rootid -x -q -o ConnectTimeout=5 -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o GSSAPIAuthentication=no  root@123.123.0.16  uname -a
Password:
Password:
Password:

Logging in as root from an IP address not defined in the Match Address line for PermitRootLogin is not allowed.

Unless an NVE snapshot was made prior to applying the rollup, there is no option to re-apply the rollup. If a VMware snapshot is available, revert the NVE to the snapshot, perform the steps outlined below, and reapply the rollup.

If no snapshot is available, a newer rollup can be applied (if available) after making the below changes. If the latest OS Rollup was applied, wait for the next rollup to be released and apply it after performing the steps below. The following steps can be used to correct both of the issues outlined in the Cause section.

1. Create a VMware snapshot of the NVE.
2. Open an SSH session to the NVE, log in as admin.
3. Switch to root:

sudo su -

4. If /usr/local/avamar/var/probe.xml exists, open it with a text editor and ensure that the configuration values (IP address, search domain, hostname) are all correct:

vi /usr/local/avamar/var/probe.xml

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<dpn>
   <module name="SERVER_SHORT_NAME" userinput_domain="DOMAIN.DOMAIN" userinput_search="DOMAIN.DOMAIN">
      <node type="single-node server">
         <network-interface id="0" userinput_ifname="eth0">
            <address value="IPv4_ADDRESS" userinput_netmask="SUBNET_ADDRESS" userinput_customhostname="FULLY_QUALIFIED_DOMAIN_NAME"/>
            <uses allow="replication,internal,backup,management"/>
         </network-interface>
      </node>
   </module>
</dpn>

Example:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<dpn>
   <module name="nve" userinput_domain="networker.lan" userinput_search="networker.lan">
      <node type="single-node server">
         <network-interface id="0" userinput_ifname="eth0">
            <address value="123.123.0.16" userinput_netmask="255.255.255.0" userinput_customhostname="nve.networker.lan"/>
            <uses allow="replication,internal,backup,management"/>
         </network-interface>
      </node>
   </module>
</dpn>

1. Ensure that the file is owned by admin: chown admin:admin /usr/local/avamar/var/probe.xml
2. Ensure that the file permissions are correct: chmod 644 /usr/local/avamar/var/probe.xml

5. Open the sshd_config file using vi:

vi /etc/ssh/sshd_config

6. Go to the Match Address line and ensure that the current IP address is defined in the list of IP addresses which PermtRootLogin.

Match Address ::1,127.0.0.1,127.0.0.1,::1,123.123.9.16,fe80::250:56ff:febb:8bdb
        PermitRootLogin yes

Match Address ::1,127.0.0.1,127.0.0.1,::1,123.123.0.16,fe80::250:56ff:febb:8bdb
        PermitRootLogin yes

7. Restart the sshd service:

systemctl restart sshd

8. Ensure that the SSH command from the workflow returns the correct uname -a output:

ssh -q -i /root/.ssh/rootid -x -q -o ConnectTimeout=5 -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o GSSAPIAuthentication=no  root@NVE_IP_ADDRESS  uname -a

nve:~ # ssh -q -i /root/.ssh/rootid -x -q -o ConnectTimeout=5 -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o GSSAPIAuthentication=no  root@123.123.0.16  uname -a
Enter passphrase for key '/root/.ssh/rootid':
Linux nve 4.12.14-122.201-default #1 SMP Tue Mar 12 13:01:34 UTC 2024 (b92bd5a) x86_64 x86_64 x86_64 GNU/Linux

2024-09-20 15:14:58 (-0400) 47419850660520 INFO: "ssh -q -i /root/.ssh/rootid -x -q -o ConnectTimeout=5 -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o GSSAPIAuthentication=no  root@127.0.0.1  uname -a", exit status=0 (success)
2024-09-20 15:14:58 (-0400) 47419850660520 INFO: address=127.0.0.1 remote=ssh -q -i /root/.ssh/rootid -x -q -o ConnectTimeout=5 -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o GSSAPIAuthentication=no  root@127.0.0.1  os=SLES version=12 update=5 arch=x86_64
...
2024-09-20 15:15:23 (-0400) 47419850660520 INFO: Installing OS updates via sec_install_os_errata_sles.pl on node set 0.s
...
2024-09-20 15:15:26 (-0400) 47419862858080 INFO: >>> [make_temp_script] 222 very_current_user=root, as_user=root, ssh command is: (perl /space/avamar/repo/temp/NvePlatformOsRollup_2024-R2-v3.avp_1726859540790/tools/sec_install_os_errata_sles.pl /data01/avamar/src/rollup_tmp//sec_os_updates_SLES12SP5-2024-R2-v3.tgz* </dev/null >/tmp/sec_install_os_errata_sles.pl.log 2>&1) >/usr/local/avamar/var/run_command-sysout.7314.47419862858080 2>&1

The VMware snapshot can be removed once the rollup was confirmed successful and system updating is validated. 

If the cause in this article does not apply, see:

• NVE: OS Rollup Reports as Successful but did not Update OS packages when /root/.bashrc Is Modified
• NVE: OS Rollup Completed But Did Not Upgrade Kernel
• NVE: Installed Binary Versions in Appliance Do Not Match Installed Upgrade or Rollup Versions

========================

• NVE: Upgrade or Install Failure Triage Guide.
• Dell Avamar and NetWorker Operating System (OS) Rollup Security Update Cadence (Log in to Dell Support is required to view this article)
• NetWorker Virtual Edition (NVE): How to Install the Platform Security Rollup.