NVE: OS Rollup Fails "One or More Nodes Not Running SLES" when /root ownership is changed
Summary: A NetWorker Virtual Edition (NVE) operating system rollup fails, reporting "One or More Nodes Not Running SLES". The ownership of /root was changed to another user.
Symptoms
NetWorker Virtual Edition rollup fails, with the following error reported in /usr/local/avamar/var/avi/server_data/package_data/NvePlatformOsRollup_YYYY-R#-v#.avp_EPOCH_TIMESTAMP/workflow.log
YYYY-MM-DD HH:mm:SS (-TZ) 4107124 INFO: no output seen from "ssh -q -i /root/.ssh/rootid -x -q -o ConnectTimeout=5 -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o GSSAPIAuthentication=no root@127.0.0.1 uname -a"
YYYY-MM-DD HH:mm:SS (-TZ) 4107124 INFO: post: "ssh -q -i /root/.ssh/rootid -x -q -o ConnectTimeout=5 -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o GSSAPIAuthentication=no root@127.0.0.1 uname -a", exit status=255 (error)
YYYY-MM-DD HH:mm:SS (-TZ) 4107124 INFO: address=127.0.0.1 remote=ssh -q -i /root/.ssh/rootid -x -q -o ConnectTimeout=5 -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o GSSAPIAuthentication=no root@127.0.0.1 os=unknown version=unknown update=unknown arch=unknown
YYYY-MM-DD HH:mm:SS (-TZ) 47378741877400 INFO: - One or more nodes are not running SLES succeeded
The OsRollup workflow logs can be found under the rollups package_data directory:
/usr/local/avamar/var/avi/server_data/package_data/NvePlatformOsRollup_YYYY-R#-v#.avp_EPOCH_TIMESTAMP/workflow.log
Cause
The ownership of the /root directory was changed by the backup admin at some point. For example it shows:
nve:~ # ls -ld /root drwx------ 8 admin root 4096 May 16 09:27 root
The permissions of /root are as such that no other user or group has access to the directory, only the owner (in this case admin) does.
During the rollup, the following command is run:
ssh -q -i /root/.ssh/rootid -x -q -o ConnectTimeout=5 -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o GSSAPIAuthentication=no root@127.0.0.1 uname -a
It tries to run as_root, but since only the admin user has access, the command returns nothing and skips the kernel update. For example:
nve:~ # ssh -q -i /root/.ssh/rootid -x -q -o ConnectTimeout=5 -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o GSSAPIAuthentication=no root@127.0.0.1 uname -a nve:~ #
When /root is owned by root:root that it should return something like:
nve:~ # ssh -q -i /root/.ssh/rootid -x -q -o ConnectTimeout=5 -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o GSSAPIAuthentication=no root@127.0.0.1 uname -a Linux nve 4.12.14-122.153-default #1 SMP Tue Mar 7 14:13:19 UTC 2023 (9f7af45) x86_64 x86_64 x86_64 GNU/Linux nve:~ #
If /root is owned by another user, the ssh command to get uname -a returns nothing, causing the workflow to report the OS as "unknown" and skip the NVE’s OS package updates.
Resolution
Correct the /root ownership:
sudo chown root:root /root
Return to the NVE installation manager and retry the current task.
/root directory. Modifying default system files and settings on the NVE can cause issues during rollups/upgrades, potentially corrupting the NVE, and requiring redeployment and NetWorker Server Disaster Recovery (nsrdr). Therefore, it is not recommended to change default system files or settings.
To permit SSH access to root, see: NVE: How to Allow root SSH Access On a NetWorker Virtual Edition
Additional Information
The symptoms and error messaging reported in this KB may occur due to other causes. If /root ownership and permissions were not modified on the NVE, see the following articles:
- NVE: OS Rollup Fails "One or More Nodes Not Running SLES" when /root/.bashrc Is Modified
- NVE: OS rollup reported as successful but did not Update OS packages after NVE IP address change
- NVE: Installed Binary Versions in Appliance Do Not Match Installed Upgrade or Rollup Versions
- NVE: OS Rollup Fails "Clean up kernels (7 of 18) Failed"
========================
NVE: Upgrade or Install Failure Triage Guide
Dell Avamar and NetWorker Operating System (OS) Rollup Security Update Cadence
NetWorker Virtual Edition (NVE): How to Install the Platform Security Rollup