ECS: OBS: xDoctor: Manual Upgrade Procedure

Summary: Manually upgrade xDoctor from the command line.

This article applies to This article does not apply to This article is not tied to any specific product. Not all product versions are identified in this article.
Check out other resources

Instructions

xDoctor is a support and diagnostic tool that identifies and helps resolve known configuration, software, and hardware issues that may negatively impact ECS and OBS systems. Key features include:

  • Troubleshooting: Helps support teams and customers identify root causes of ECS issues.
  • Proactive Monitoring: Detects early signs of problems
  • Support Engagement: Some actions and resolutions require Dell support involvement.

Dell technical support recommends having the latest xDoctor health check on all VDC racks. As with each version, xDoctor is updated with improved health checks and auto healers. Auto healers were introduced in later xDoctor versions.  

Determine the ECS/OBS and xDoctor version installed:

admin@node1:~> sudo xdoctor -x
....................
ECS Version: 3.8.1.6
-----------------------
xDoctor Version: 4.8-104.0
-----------------------


Determine if all nodes have the same xDoctor version:

admin@node1:~> sudo xdoctor -s
xDoctor Uniform on all nodes: 4.8-104.0


Manual xDoctor Upgrade:

  1. Download the latest version of xDoctor depending on the ECS/OBS version as above. Note when downloading the xDoctor package, this depends on your code version in ECS and OBS - also take note of the file formats below:
    1. Sign in required to download the latest version:
      1. ECS (3.8 and below): xDoctor package starts with 4.x. (Example: 4.8-105.0 - ECS Product Support webpage
      2. OBS (3.9 and above): xDoctor package starts with 5.x (Example: 5.1-105.0) - OBS Product Support webpage
      3. The xDoctor release notes for the version downloaded have a dedicated section to upgrade xDoctor.
  2. Upload the latest xDoctor RPM file to /home/admin directory of a node on the rack - example below cover ECS, follow the same steps as with OBS changing the file name):
admin@node1~> ls -l | grep xDoctor
-rw-r--r-- 1 admin users   20057045 Jul  6  2025 xDoctor4ECS-4.8-104.0.noarch.rpm
-rw-r--r-- 1 admin users   31927626 Aug 26 15:11 xDoctor4ECS-4.8-105.0.noarch.rpm
  1. Run the xDoctor upgrade command. Upgrade can be performed on a per rack or per VDC basis.
Rack upgrade command - enable all auto healers by typing A: 
# sudo xdoctor --upgrade --local=/home/admin/xDoctor4ECS-4.8-105.0.noarch.rpm

admin@node1:~> sudo xdoctor --upgrade --local=/home/admin/xDoctor4ECS-4.8-105.0.noarch.rpm
This new xDoctor RPM has the following Auto Healers:

┌──────────────┐
│ Auto Healers │
└───┬──────────┘
    │
    │ time_zone                              = Enabled
    │ pmon_crontab_check                     = Disabled
    │ pmon_swapiness_check                   = Disabled
    │ rsyslogd_check                         = Disabled
    │ task_md_cleanup_status                 = Disabled
    │ ntpd_not_running                 (New) = Disabled
    │ cron_not_running                 (New) = Disabled
    │ machines_file_error              (New) = Disabled
    │ non_uniform_psnt                 (New) = Disabled
    │ racadm_stale_pid                 (New) = Disabled
    │ obj_control_svc_check            (New) = Disabled

In order to have them active, they need to be enabled ...
You can do this during this upgrade or later post upgrade via `xdoctor --config`

Would you like to enable (A)ll, only the (N)ew ones or (I)gnore them [I]: A

2025-10-09 11:36:15,663: xDoctor_4.8-104.0 - INFO    : User selected to enable all Auto Healers ...
2025-10-09 11:36:15,663: xDoctor_4.8-104.0 - INFO    : Local Upgrade (/home/admin/xDoctor4ECS-4.8-105.0.noarch.rpm)
2025-10-09 11:36:15,696: xDoctor_4.8-104.0 - INFO    : Current Installed xDoctor version is 4.8-104.0
2025-10-09 11:36:15,712: xDoctor_4.8-104.0 - INFO    : Requested package version is 4.8-105.0
2025-10-09 11:36:15,713: xDoctor_4.8-104.0 - INFO    : Updating xDoctor RPM Package (RPM)
2025-10-09 11:36:15,935: xDoctor_4.8-104.0 - INFO    :  - Distribute package
2025-10-09 11:36:17,402: xDoctor_4.8-104.0 - INFO    :  - Install new rpm package
2025-10-09 11:36:33,562: xDoctor_4.8-104.0 - INFO    : Enabling Auto Healer: pmon_crontab_check ...
2025-10-09 11:36:33,563: xDoctor_4.8-104.0 - INFO    : Enabling Auto Healer: pmon_swapiness_check ...
2025-10-09 11:36:33,563: xDoctor_4.8-104.0 - INFO    : Enabling Auto Healer: rsyslogd_check ...
2025-10-09 11:36:33,563: xDoctor_4.8-104.0 - INFO    : Enabling Auto Healer: task_md_cleanup_status ...
2025-10-09 11:36:33,563: xDoctor_4.8-104.0 - INFO    : Enabling Auto Healer: ntpd_not_running ...
2025-10-09 11:36:33,563: xDoctor_4.8-104.0 - INFO    : Enabling Auto Healer: cron_not_running ...
2025-10-09 11:36:33,563: xDoctor_4.8-104.0 - INFO    : Enabling Auto Healer: machines_file_error ...
2025-10-09 11:36:33,563: xDoctor_4.8-104.0 - INFO    : Enabling Auto Healer: non_uniform_psnt ...
2025-10-09 11:36:33,563: xDoctor_4.8-104.0 - INFO    : Enabling Auto Healer: racadm_stale_pid ...
2025-10-09 11:36:33,563: xDoctor_4.8-104.0 - INFO    : Enabling Auto Healer: obj_control_svc_check ...
2025-10-09 11:36:33,828: xDoctor_4.8-104.0 - INFO    : Auto Healer Settings saved and distributed ...

┌──────────────────────┐
│ Updated Auto Healers │
└───┬──────────────────┘
    │
    │ time_zone                              = Enabled
    │ pmon_crontab_check                     = Enabled
    │ pmon_swapiness_check                   = Enabled
    │ rsyslogd_check                         = Enabled
    │ task_md_cleanup_status                 = Enabled
    │ ntpd_not_running                 (New) = Enabled
    │ cron_not_running                 (New) = Enabled
    │ machines_file_error              (New) = Enabled
    │ non_uniform_psnt                 (New) = Enabled
    │ racadm_stale_pid                 (New) = Enabled
    │ obj_control_svc_check            (New) = Enabled

2025-10-09 11:36:33,829: xDoctor_4.8-104.0 - INFO    : xDoctor successfully updated to version 4.8-105.0
VDC upgrade command - enable all auto healers by typing A: 
# sudo xdoctor --upgrade --vdc-upgrade --local=/home/admin/xDoctor4ECS-4.8.105.0.noarch.rpm
 

xDoctor is not uniform across all nodes:

If a node is reinstalled or was offline during xDoctor updates, it may end up with a different version than the rest of the cluster: 
admin@node1:~> sudo xdoctor -s
xDoctor not uniform across all nodes ...
Trying xDoctor Resync ...
Resync failed: No xDoctor package found for re-installation
[4.8-104.0] -> ['169.254.1.2']
[4.8-105.0] -> ['169.254.1.6', '169.254.1.5', '169.254.1.4', '169.254.1.3', '169.254.1.1']
xDoctor output alerting that it is not uniform on all nodes of the rack: 
admin@node1:~> sudo xdoctor --report --archive=2022-06-26_101004 -WEC

xDoctor 4.8.105.0
CKM00xxxxxxxx - ECS 3.8.1.2

Displaying xDoctor Report (2022-06-26_101004) Filter:['CRITICAL', 'ERROR', 'WARNING'] ...

--------------------------------------------
ERROR - xDoctor not uniform across all nodes
--------------------------------------------
Extra     = Not allowed to use the SYSTEM scope, use LOCAL scope instead or reinstall xDoctor -> xdr_versions={'4.8-104.0': ['169.254.1.2'], '4.8-105.0': ['169.254.1.1', '169.254.1.6', '169.254.1.4', '169.254.1.3', '169.254.1.5']}
RAP       = RAP099
Solution  = KB 91703
Timestamp = 2022-06-26_101004
PSNT      = CKM00xxxxxxxx @ 4.8.105.0
Attempting a xDoctor upgrade would fail as xDoctor detects that the other nodes have the version that you are attempting to install: 
admin@node1:~> sudo xdoctor --upgrade --local=/home/admin/xDoctor4ECS-4.8.105.0.noarch.rpm
2022-06-28 15:11:40,101: xDoctor_4.8-105.0 - INFO    : xDoctor Upgrader Instance (2:FTP_SFTP)
2022-06-28 15:11:40,101: xDoctor_4.8-105.0 - INFO    : Local Upgrade (/home/admin/xDoctor4ECS-4.8-105.0.noarch.rpm)
2022-06-28 15:11:40,134: xDoctor_4.8-105.0 - INFO    : Current Installed xDoctor version is 4.8-105.0
2022-06-28 15:11:40,174: xDoctor_4.8-105.0 - INFO    : Requested package version is 4.8-105.0
2022-06-28 15:11:40,174: xDoctor_4.8-105.0 - INFO    : xDoctor is up-to-date, only newer versions allowed ...
If there are nodes on the latest version but you want to reapply the same xDoctor version on the rack, use the xDoctor reinstall option:  
admin@node1:~> sudo xdoctor --upgrade --local=/home/admin/xDoctor4ECS-4.8-105.0.noarch.rpm --reinstall
2022-06-28 15:12:53,079: xDoctor_4.8-105.0 - INFO    : xDoctor Upgrader Instance (2:FTP_SFTP)
2022-06-28 15:12:53,079: xDoctor_4.8-105.0 - INFO    : Local Upgrade (/home/admin/xDoctor4ECS-4.8-105.0.noarch.rpm)
2022-06-28 15:12:53,112: xDoctor_4.8-105.0 - INFO    : Current Installed xDoctor version is 4.8-105.0
2022-06-28 15:12:53,147: xDoctor_4.8-105.0 - INFO    : Requested package version is 4.8-105.0
2022-06-28 15:12:53,148: xDoctor_4.8-105.0 - WARNING : (Re)installing requested xDoctor package ...
2022-06-28 15:12:53,148: xDoctor_4.8-105.0 - INFO    : Updating xDoctor RPM Package (RPM)
2022-06-28 15:12:53,244: xDoctor_4.8-105.0 - INFO    :  - Distribute package
2022-06-28 15:12:54,115: xDoctor_4.8-105.0 - INFO    :  - Install new rpm package
2022-06-28 15:13:08,544: xDoctor_4.8-105.0 - INFO    : xDoctor successfully updated to version 4.8-105.0
2020-09-01 09:04:30,184: xDoctor_4.8-105.0 - INFO    : xDoctor Activation skipped. Only tested on Rack Master
Determine if all nodes now have the same xDoctor version: 
admin@node1:~> sudo xdoctor -s
xDoctor Uniform on all nodes: 4.8-105.0.
Recheck the xDoctor version: 
admin@node1:~> sudo xdoctor -x
........
ECS Version: 3.8.1.6
-----------------------
xDoctor Version: 4.8-105.0
-----------------------
Determine if all nodes have the same xDoctor version: 
admin@node1:~> sudo xdoctor -s
xDoctor Uniform on all nodes: 4.8-105.0
 

Run xDoctor Health Check:

To run a xDoctor health check, clear the cache first, then run xDoctor: 
admin@node1:~> sudo xdoctor --clear
You are about to clear all xDoctor Cache files. Are you sure you want to proceed?  [No]: yes
2020-09-07 14:03:34,263: xDoctor_4.8-105.0- INFO    : xDoctor Cache files successfully cleared.
To run the xDoctor health check: 
admin@node1:~> sudo xdoctor --hr
The --hr option attempts to send ECS support the latest xDoctor health check report as the connection may be blocked.

xDoctor runs from a single node as the primary, and pulls the other nodes' xDoctor reports together onto that node. Therefore, run xDoctor on that node, default is the first node on the rack. If the node is not responding to the VDC, the next node becomes the "primary node" that is, node 2. See "xDoctor Activation skipped" in the above xDoctor upgrade, only active on the primary node.

At the end of the xDoctor run, xDoctor generates a session report, the WARNING, ERROR, or CRITICAL are the items to check for in xDoctor reports: 
admin@node1:~> sudo xdoctor --hr
....................
2020-09-07 13:58:33,520: xDoctor_4.8-105.0 - INFO    : PSNT: CKM00xxxxxxxxxx
2020-09-07 13:58:33,520: xDoctor_4.8-105.0 - INFO    : --------------------
2020-09-07 13:58:33,520: xDoctor_4.8-105.0 - INFO    : Number of CRITICAL:    0
2020-09-07 13:58:33,520: xDoctor_4.8-105.0 - INFO    : Number of FIXED:         0
2020-09-07 13:58:33,520: xDoctor_4.8-105.0 - INFO    : Number of ERROR:       0
2020-09-07 13:58:33,520: xDoctor_4.8-105.0 - INFO    : Number of WARNING:     1
....................
....................
2020-09-07 13:58:42,910: xDoctor_4.8-105.0 - INFO    : --------------------------
2020-09-07 13:58:42,910: xDoctor_4.8-105.0 - INFO    : Session Report                - xdoctor --report --archive=2020-09-07_135109
To check the xDoctor report: 
sudo xdoctor --report --archive=2022-xxxxxxxxxxxxx -WEC
Example: 
admin@node1:~> sudo xdoctor --report --archive=2XXX-0X-01_0XXXX9 -WEC

xDoctor 4.8-105.0
CKM00xxxxxxxxx - ECS 3.8 Patch 2 (??) - 3.8.0.2 

Displaying xDoctor Report (2XX0-0X-01_0XXXX9) Filter:['CRITICAL', 'ERROR', 'WARNING'] ...

Timestamp    = 2XX0-0X-01_0XXXX9
    Category = Health
    Source   = Disk
    Severity = ERROR
    Node     = 169.254.1.1
    Message  = Boot device is not accessible
    Extra    = {'1XX.2XX.X.5': ['/dev/sdl']}
    RAP      = RAP004
    Solution = 46306
The results may produce Recommended Action Plan (RAP) codes. These are useful to search for linked knowledge articles that can possibly be relevant to the alert code.

Affected Products

ECS Appliance

Products

ECS Appliance, ECS Appliance Hardware Gen1 U-Series, ECS Appliance Software with Encryption, ECS Appliance Software without Encryption
Article Properties
Article Number: 000021704
Article Type: How To
Last Modified: 07 نوفمبر 2025
Version:  14
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.