ECS: OBS: xDoctor: RAP099: xDoctor not uniform across all nodes
Summary: This KB explains why XDR_OBS_099 is reported due to xDoctor not uniform across all nodes.
Symptoms
xDoctor is a support and diagnostic tool designed to identify and help resolve known configuration, software, and hardware issues that may negatively impact ECS and OBS systems. Key features include:
- Troubleshooting: Helps support teams and customers identify root causes of ECS and OBS issues.
- Proactive Monitoring: Detects early signs of problems
- Support Engagement: Some actions and resolutions require Dell support involvement.
xDoctor is reporting the following error:
------------------------------------------------------
ERROR - xDoctor not uniform across all nodes
------------------------------------------------------
Extra = Not allowed to use the SYSTEM scope, use LOCAL scope instead or reinstall xDoctor -> xdr_versions={'4.8-85.0': ['169.254.1.3'], '4.8-100.1': ['169.254.1.4', '169.254.1.2', '169.254.1.1']}
RAP = RAP099
Solution = KB 91703
Timestamp = 2025-10-09_120102
PSNT = CKM00000000000 @ 4.8-100.1
As a result ECS diagnosis is limited to local scope. This means that only a small set of symptom detectors can be tested, and not all telemetry reports can be collected and sent home.
To confirm the xDoctor version is not uniform, SSH to a node to verify as below:
# sudo xdoctor -s xDoctor not uniform across all nodes ... Trying xDoctor Resync ... Resync failed: No xDoctor package found for re-installation [4.8-85.0] -> ['169.254.1.3'] [4.8-100.1] -> ['169.254.1.4', '169.254.1.2', '169.254.1.1']
In the above example, node 3 is running xDoctor 4.8-85.0 while nodes 1, 2 and 4 are running xDoctor 4.8-100.1.
Cause
- A node is reinstalled or was offline during xDoctor updates.
- A node is replaced, and the new node xDoctor version is not uniform with the other nodes.
- Capacity expansion with new nodes having an older xDoctor version
Resolution
It is highly recommended to address this as this leads to incomplete topology views of the install base and being unaware for known symptoms.
While syncing up the node is the solution, the best recommendation is to upgrade xDoctor on all nodes to the latest for all symptom detectors and auto healers enabled. Auto healers were introduced in later xDoctor versions.
Note when downloading the xDoctor package, this depends on your code version in ECS and OBS - also take note of the file formats below:
- ECS (3.8 and below): xDoctor package starts with 4.x. (Example: 4.8-105.0) - ECS Product Support webpage (Sign in required to download package)
- OBS (3.9 and above): xDoctor package starts with 5.x (Example: 5.1-105.0) - OBS Product Support webpage (Sign in required to download package)
|
To upgrade xDoctor to the latest, follow the steps in KB ECS: OBS: xDoctor: Manual Upgrade Procedure:
See Example 1 in the Additional Info section to upgrade xDoctor to the latest version. |
|
If there are any restrictions with updating xDoctor to the latest version, we can sync the last version on the local system. Follow the steps in KB ECS: OBS: xDoctor: Manual Upgrade Procedure:
See Example 2 in the Additional Info section to make all nodes xDoctor uniform to the last version. |
Once the xDoctor is upgraded to the latest, confirm that the xDoctor version is uniform on all nodes in the rack:
# sudo xdoctor -s xDoctor Uniform on all nodes: 4.8-105.0.
If any assistance is required with the above steps, reach out to Dell technical support referencing this KB.
Additional Information
Below are examples provided as per the resolution section:
Example 1 - Upgrade xDoctor to the latest version
Confirm symptom that xDoctor is not uniform on all nodes - below nodes 1-5 are on xDoctor 4.8-102.0 while nodes 6-10 are on 4.8-95.0:
# sudo xdoctor -s xDoctor not uniform across all nodes ... Trying xDoctor Resync ... Resync failed: No xDoctor package found for re-installation [4.8-102.0] -> ['169.254.2.1', '169.254.2.2', '169.254.2.3', '169.254.2.4', '169.254.2.5'] [4.8-95.0] -> ['169.254.2.10', '169.254.2.11', '169.254.2.8', '169.254.2.9', '169.254.2.6', '169.254.2.7']
Once latest version of xDoctor is uploaded on node 1 to /home/admin, confirm it exists:
# ls -l /home/admin | grep xDoctor -rw-r--r-- 1 admin users 43793804 Apr 14 08:21 xDoctor4ECS-4.8-102.0.noarch.rpm -rw-r--r-- 1 root root 121382804 Oct 9 11:34 xDoctor4ECS-4.8-105.0.noarch.rpm
Upgrade xDoctor to the latest and enable all auto healers:
# sudo xdoctor --upgrade --local=/home/admin/xDoctor4ECS-4.8-105.0.noarch.rpm
This new xDoctor RPM has the following Auto Healers:
┌──────────────┐
│ Auto Healers │
└───┬──────────┘
│
│ time_zone = Enabled
│ pmon_crontab_check = Disabled
│ pmon_swapiness_check = Disabled
│ rsyslogd_check = Disabled
│ task_md_cleanup_status = Disabled
│ ntpd_not_running (New) = Disabled
│ cron_not_running (New) = Disabled
│ machines_file_error (New) = Disabled
│ non_uniform_psnt (New) = Disabled
│ racadm_stale_pid (New) = Disabled
│ obj_control_svc_check (New) = Disabled
In order to have them active, they need to be enabled ...
You can do this during this upgrade or later post upgrade via `xdoctor --config`
Would you like to enable (A)ll, only the (N)ew ones or (I)gnore them [I]: A
2025-10-09 11:36:15,663: xDoctor_4.8-102.0 - INFO : User selected to enable all Auto Healers ...
2025-10-09 11:36:15,663: xDoctor_4.8-102.0 - INFO : Local Upgrade (/home/admin/xDoctor4ECS-4.8-105.0.noarch.rpm)
2025-10-09 11:36:15,696: xDoctor_4.8-102.0 - INFO : Current Installed xDoctor version is 4.8-102.0
2025-10-09 11:36:15,712: xDoctor_4.8-102.0 - INFO : Requested package version is 4.8-105.0
2025-10-09 11:36:15,713: xDoctor_4.8-102.0 - INFO : Updating xDoctor RPM Package (RPM)
2025-10-09 11:36:15,935: xDoctor_4.8-102.0 - INFO : - Distribute package
2025-10-09 11:36:17,402: xDoctor_4.8-102.0 - INFO : - Install new rpm package
2025-10-09 11:36:33,562: xDoctor_4.8-102.0 - INFO : Enabling Auto Healer: pmon_crontab_check ...
2025-10-09 11:36:33,563: xDoctor_4.8-102.0 - INFO : Enabling Auto Healer: pmon_swapiness_check ...
2025-10-09 11:36:33,563: xDoctor_4.8-102.0 - INFO : Enabling Auto Healer: rsyslogd_check ...
2025-10-09 11:36:33,563: xDoctor_4.8-102.0 - INFO : Enabling Auto Healer: task_md_cleanup_status ...
2025-10-09 11:36:33,563: xDoctor_4.8-102.0 - INFO : Enabling Auto Healer: ntpd_not_running ...
2025-10-09 11:36:33,563: xDoctor_4.8-102.0 - INFO : Enabling Auto Healer: cron_not_running ...
2025-10-09 11:36:33,563: xDoctor_4.8-102.0 - INFO : Enabling Auto Healer: machines_file_error ...
2025-10-09 11:36:33,563: xDoctor_4.8-102.0 - INFO : Enabling Auto Healer: non_uniform_psnt ...
2025-10-09 11:36:33,563: xDoctor_4.8-102.0 - INFO : Enabling Auto Healer: racadm_stale_pid ...
2025-10-09 11:36:33,563: xDoctor_4.8-102.0 - INFO : Enabling Auto Healer: obj_control_svc_check ...
2025-10-09 11:36:33,828: xDoctor_4.8-102.0 - INFO : Auto Healer Settings saved and distributed ...
┌──────────────────────┐
│ Updated Auto Healers │
└───┬──────────────────┘
│
│ time_zone = Enabled
│ pmon_crontab_check = Enabled
│ pmon_swapiness_check = Enabled
│ rsyslogd_check = Enabled
│ task_md_cleanup_status = Enabled
│ ntpd_not_running (New) = Enabled
│ cron_not_running (New) = Enabled
│ machines_file_error (New) = Enabled
│ non_uniform_psnt (New) = Enabled
│ racadm_stale_pid (New) = Enabled
│ obj_control_svc_check (New) = Enabled
2025-10-09 11:36:33,829: xDoctor_4.8-102.0 - INFO : xDoctor successfully updated to version 4.8-105.0
Confirm xDoctor is now uniform on all nodes:
# sudo xdoctor -s xDoctor Uniform on all nodes: 4.8-105.0
Example 2 - Make all nodes xDoctor uniform to the last version on the local system.
Confirm symptom that xDoctor is not uniform on all nodes - below nodes 1, 2 and 4 are on xDoctor 4.8-100.1 while nodes 3 are on 4.8-85.0:
# sudo xdoctor -s xDoctor not uniform across all nodes ... Trying xDoctor Resync ... Resync failed: No xDoctor package found for re-installation [4.8-85.0] -> ['169.254.1.3'] [4.8-100.1] -> ['169.254.1.4', '169.254.1.2', '169.254.1.1']
On nodes 1 (preferably), 2 or 4, confirm that the xDoctor 4.8-100.1 package exists on /home/admin:
# ls -l /home/admin | grep xDoctor -rw-r--r-- 1 admin users 43793804 Mar 4 08:21 xDoctor4ECS-4.8-99.0.noarch.rpm -rw-r--r-- 1 root root 121382804 Sep 2 11:34 xDoctor4ECS-4.8-100.1.noarch.rpm
Reinstall xDoctor package:
sudo xdoctor --upgrade --local=/home/admin/xDoctor4ECS-4.8-100.1.noarch.rpm --reinstall 2025-10-09 12:24:19,996: xDoctor_4.8-100.1 - INFO : Local Upgrade (/home/admin/xDoctor4ECS-4.8-100.1.noarch.rpm) 2025-10-09 12:24:20,029: xDoctor_4.8-100.1 - INFO : Current Installed xDoctor version is 4.8-100.1 2025-10-09 12:24:20,046: xDoctor_4.8-100.1 - INFO : Requested package version is 4.8-100.1 2025-10-09 12:24:20,046: xDoctor_4.8-100.1 - WARNING : (Re)installing requested xDoctor package ... 2025-10-09 12:24:20,046: xDoctor_4.8-100.1 - INFO : Updating xDoctor RPM Package (RPM) 2025-10-09 12:24:20,176: xDoctor_4.8-100.1 - INFO : - Distribute package 2025-10-09 12:24:21,745: xDoctor_4.8-100.1 - INFO : - Install new rpm package 2025-10-09 12:24:37,095: xDoctor_4.8-100.1 - INFO : xDoctor successfully updated to version 4.8-100.1
Confirm xDoctor is now uniform on all nodes:
sudo -i xdoctor -s xDoctor Uniform on all nodes: 4.8-100.1