NVE: OS Rollup Fails "One or More Nodes Not Running SLES" when /root/.bashrc Is Modified

NetWorker Virtual Edition rollup fails, with the following error reported in /usr/local/avamar/var/avi/server_data/package_data/NvePlatformOsRollup_YYYY-R#-v#.avp_EPOCH_TIMESTAMP/workflow.log

nve:~ # cat /usr/local/avamar/var/avi/server_data/package_data/*/workflow.log | grep -A2 "uname"
YYYY-MM-DD HH:mm:SS (TZ) 47378741877400 INFO: >>> [make_temp_script] 222 very_current_user=root, as_user=root, ssh command is: (ssh -q -i /root/.ssh/rootid -x -q -o ConnectTimeout=5 -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o GSSAPIAuthentication=no  root@127.0.0.1  uname -a) >/usr/local/avamar/var/run_command-sysout.13521.47378741877400 2>&1
YYYY-MM-DD HH:mm:SS (TZ) 47378741877400 INFO: - - - - - - - BEGIN(output) - - - - - - -
YYYY-MM-DD HH:mm:SS (TZ) 47378741877400 INFO:
---        
YYYY-MM-DD HH:mm:SS (TZ) 47378741877400 INFO: "ssh -q -i /root/.ssh/rootid -x -q -o ConnectTimeout=5 -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o GSSAPIAuthentication=no  root@127.0.0.1  uname -a", exit status=0 (success)
YYYY-MM-DD HH:mm:SS (TZ) 47378741877400 INFO: address=::1 remote=ssh -q -i /root/.ssh/rootid -x -q -o ConnectTimeout=5 -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o GSSAPIAuthentication=no  root@127.0.0.1  os=unknown version=unknown update=unknown arch=x86_64
YYYY-MM-DD HH:mm:SS (TZ) 47378741877400 INFO:  - One or more nodes are not running SLES succeeded

The OsRollup workflow logs can be found under the rollups package_data directory:

/usr/local/avamar/var/avi/server_data/package_data/NvePlatformOsRollup_YYYY-R#-v#.avp_EPOCH_TIMESTAMP/workflow.log

In this instance, the NVE's /root/.bashrc has been modified to include some post login functions for the root user. When the workflow attempts to run the SSH command as root, it is returning the custom script output from /root/.bashrc when it is only expecting the output of uname -a. This results in the workflow failing to detect the correct OS.

/root/.bashrc contains user specified commands or scripts.

nve:~ # cat /root/.bashrc
/etc/dynmotd

Running the same command reported in the workflow, other output is returned from /root/.bashrc instead of only the uname -a output.

nve:~ # ssh -q -i /root/.ssh/rootid -x -q -o ConnectTimeout=5 -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o GSSAPIAuthentication=no  root@127.0.0.1  uname -a

===========================================================================

 - DESC:       NetWorker Virtual Edition Appliance
 - NAME:       nve.networker.lan
 - PRIVATE IP: 123.123.0.16/24
 - OS:         SUSE Linux Enterprise Server 12 (x86_64)
 - NSR:        19.10.0.2.Build.77
 - NRE:        8.0.21
 - JAVA:       jre1.8.0_411

===========================================================================
 - Current user........: root
 - Login time..........: Fri Sep 20 11:47:34 EDT 2024
 - Users...............: Currently 1 user(s) logged on
 - CPU usage...........: 0.10, 0.03, 0.01 (1, 5, 15 min)
 - Memory used.........: 2946 MB / 15776 MB
 - / Disk usage........: 38%
 - Processes...........: 210 running
 - System uptime.......: 0 days 19 hours 0 minutes 22 secondsg
===========================================================================
 - NVE History..........:

Feb 27 2020 NveInstall-19.1.1-90.avp_1582793396969
Sep 19 08:58 NveConfig-19.1.1-90.avp_1726750488800
Sep 19 09:19 NvePlatformOsRollup_2020-R3-v5.avp_1726751519503
Sep 19 10:06 NveUpgrade-19.3.0-11.avp_1726754527160
Sep 19 10:49 NveUpgrade-19.4.0-84.avp_1726756738443
Sep 19 12:04 NveUpgrade-19.5.0-82.avp_1726760392985
Sep 19 12:30 NvePlatformOsRollup_2021-R3-v6.avp_1726763311547
Sep 19 12:53 NveUpgrade-19.6.1-51.avp_1726764299579
Sep 19 13:13 NveUpgrade-19.8.0-81.avp_1726765443481
Sep 19 13:20 NvePlatformOsRollup_2023-R1-v7.avp_1726766343118
Sep 19 13:24 NvePlatformOsRollup_2023-R2-v5.avp_1726766619289
Sep 19 13:44 NveUpgrade-19.9.0-518.avp_1726767035124
Sep 19 13:51 NvePlatformOsRollup_2023-R4-v4.avp_1726768210200
Sep 19 14:13 NveUpgrade-19.10.0-42.avp_1726768964793
Sep 19 16:51 NvePlatformOsRollup_2024-R2-v3.avp_1726778983486
===========================================================================

Linux nve 4.12.14-122.37-default #1 SMP Sun Sep 6 05:00:36 UTC 2020 (fe8cacf) x86_64 x86_64 x86_64 GNU/Linux

The command should only return the-a output.
Example:

nve:~ # ssh -q -i /root/.ssh/rootid -x -q -o ConnectTimeout=5 -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o GSSAPIAuthentication=no  root@127.0.0.1  uname -a
Linux nve 4.12.14-122.37-default #1 SMP Sun Sep 6 05:00:36 UTC 2020 (fe8cacf) x86_64 x86_64 x86_64 GNU/Linux
nve:~ #

Remove the contents of /root/.bashrc or delete the file.

1. Create a VMware snapshot of the NVE.
2. Open an SSH session to the NVE as admin
3. Switch to root:

sudo su -

4. Delete or rename the /root/.bashrc file

rm -rf /root/.bashrc

OR

mv /root/.bashrc /root/backup-bashrc​​

5. Verify that the following command only returns results for uname:

ssh -q -i /root/.ssh/rootid -x -q -o ConnectTimeout=5 -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o GSSAPIAuthentication=no  root@127.0.0.1  uname -a

6. Return to the NVE installation manager and retry the current task

The VMware snapshot can be removed once the rollup was confirmed successful and system updating is validated. 

If the cause in this KB does not apply, the see the following KB:  

• NVE: OS Rollup Fails "One or More Nodes Not Running SLES" when /root ownership is changed
• NVE: OS rollup reported as successful but did not Update OS packages after NVE IP address change
• NVE: Installed Binary Versions in Appliance Do Not Match Installed Upgrade or Rollup Versions

========================

• NVE: Upgrade or Install Failure Triage Guide.
• Dell Avamar and NetWorker Operating System (OS) Rollup Security Update Cadence (Log in to Dell Support is required to view this article.)
• NetWorker Virtual Edition (NVE): How to Install the Platform Security Rollup.