ECS: OBS: xDoctor: RAP073/208: Switch Connection Failure detected

Summary: This knowledge base explains how to handle alert Switch Connection Failure detected.

Affected Products

This article applies to This article does not apply to This article is not tied to any specific product. Not all product versions are identified in this article.

Check out other resources

Symptoms

Beginning with ECS xDoctor v4.8‑109.0 and ObjectScale xDoctor v5.1‑109.0, RAP208 (Switch Connection Failure Detected) is implemented as an auto‑healer. When switch connectivity issues exceed the configured Error or Critical severity threshold, xDoctor raises a RAP208 alert and automatically initiates its integrated repair orchestration workflow. This workflow performs the required remediation actions, if xDoctor auto‑healers are enabled.

NOTE: If your environment is running an xDoctor version earlier than ECS xDoctor v4.8‑109.0 or ObjectScale xDoctor v5.1‑109.0, the RAP208 auto‑healer functionality is not available. In these versions, remediation must be performed using the AutoPilot process described below or by following the manual remediation steps outlined in the Resolution section.

Alert That Triggers RAP208 Auto‑Healing

The RAP208 auto‑healing workflow is triggered when switch connectivity failures cross the configured Error or Critical severity threshold. Once this threshold is exceeded, xDoctor generates a RAP208 alert, which serves as the trigger for the automated repair process.

Alert Output Example

NOTE: In xDoctor versions prior to ECS xDoctor v4.8‑109.0 and ObjectScale xDoctor v5.1‑109.0, this condition results in an alert only. Automatic remediation is not performed.

--------------------------------------------------------
INFO - Auto Healer for dell_switch_connectivity disabled
--------------------------------------------------------
Extra     = Auto Healer for dell_switch_connectivity disabled
Timestamp = 2026-04-01_180132
PSNT      = CKMXXXXXXXXXXX @ 4.8-109.0

----------------------------------------------------
ERROR - (Cached) Switch Connection Failure detected.
----------------------------------------------------
Node      = 169.254.1.1
Extra     = {"169.254.1.1": ["hare"]}
RAP       = RAP208
Solution  = KB 39838
Timestamp = 2026-04-01_180132
PSNT      = CKMXXXXXXXXXXX @ 4.8-109.0

Auto Healer Remediation (Example)

When auto‑healers are enabled, xDoctor automatically initiates remediation for detected switch connectivity issues by applying common corrective actions outlined in this knowledge base.

--------------------------------------------------------
FIXED - Auto Healer fixed Dell switch connectivity issue
--------------------------------------------------------
Node      = Nodes
Extra     = {"Nodes": ["169.254.1.1"]}
Timestamp = 2026-04-01_180344
PSNT      = CKMXXXXXXXXXXX @ 4.8-109.0

Auto Healer Requirement

The xDoctor auto‑healer feature must be enabled for this remediation to occur. Auto‑healers can be enabled either during installation or post‑installation by following the steps outlined in:

KB: ECS: xDoctor: How to Enable xDoctor Auto Healer after tool installation

Cause

After a switch replacement, the SSH host keys used to authenticate to the switch may change, or the management interface connecting to the switch may be administratively shut down. Sometimes the password configured in xDoctor does not match the current password on the affected switch and must be updated accordingly.

The xDoctor automation and auto‑healer workflows do not perform switch password remediation. Instead, xDoctor detects authentication‑related failures and raises the appropriate alert, directing the user to the relevant knowledge base article that describes how to configure xDoctor to use the password set on the switches.

Resolution

xDoctor Auto Healer: ObjectScale xDoctor v5.1‑109.0 / ECS xDoctor v4.8‑109.0 or later

To manually trigger the Enabled auto healer, run the following command on the master.rack node. This initiates the rack analyzers, which will validate and auto‑heal the nodes one at a time.

Command:

# sudo xdoctor --rap=RAP208

Example:

admin@ecsnode1:~> sudo xdoctor --rap=RAP208
2026-04-01 18:03:45,441: xDoctor_4.8-109.0 - INFO    : Initializing xDoctor v4.8-109.0 ...
[... Truncated Output ...]
2026-04-01 18:05:01,725: xDoctor_4.8-109.0 - INFO    : ANALYZER [ac_dell_switch_connectivity]
2026-04-01 18:05:02,063: xDoctor_4.8-109.0 - INFO    : Autohealing switch_connectivity on node 169.254.1.1 ...
2026-04-01 18:08:57,494: xDoctor_4.8-109.0 - INFO    : All data analyzed in 0:03:55
2026-04-01 18:08:58,529: xDoctor_4.8-109.0 - INFO    : --------------------
2026-04-01 18:08:58,529: xDoctor_4.8-109.0 - INFO    : Diagnosis Summary
2026-04-01 18:08:58,529: xDoctor_4.8-109.0 - INFO    : --------------------
2026-04-01 18:08:58,529: xDoctor_4.8-109.0 - INFO    : PSNT: CKMXXXXXXXXXXX
2026-04-01 18:08:58,529: xDoctor_4.8-109.0 - INFO    : --------------------
2026-04-01 18:08:58,529: xDoctor_4.8-109.0 - INFO    : FIXED             =  1
2026-04-01 18:08:58,530: xDoctor_4.8-109.0 - INFO    : CRITICAL          =  0
2026-04-01 18:08:58,530: xDoctor_4.8-109.0 - INFO    : CRITICAL (CACHED) =  0
2026-04-01 18:08:58,530: xDoctor_4.8-109.0 - INFO    : ERROR             =  0
2026-04-01 18:08:58,530: xDoctor_4.8-109.0 - INFO    : ERROR (CACHED)    =  0
2026-04-01 18:08:58,530: xDoctor_4.8-109.0 - INFO    : WARNING           =  0
2026-04-01 18:08:58,530: xDoctor_4.8-109.0 - INFO    : INFO              =  0
2026-04-01 18:08:58,530: xDoctor_4.8-109.0 - INFO    : VERBOSE           =  0
2026-04-01 18:08:58,531: xDoctor_4.8-109.0 - INFO    : REPORT            =  0
2026-04-01 18:08:58,646: xDoctor_4.8-109.0 - INFO    : ---------------------
2026-04-01 18:08:58,646: xDoctor_4.8-109.0 - INFO    : xDoctor Post Features
2026-04-01 18:08:58,646: xDoctor_4.8-109.0 - INFO    : ----------------
2026-04-01 18:08:58,646: xDoctor_4.8-109.0 - INFO    : Data Combiner
2026-04-01 18:08:58,646: xDoctor_4.8-109.0 - INFO    : -------------
2026-04-01 18:08:58,647: xDoctor_4.8-109.0 - INFO    : Created a Data Collection Report (data.xml)
2026-04-01 18:08:58,648: xDoctor_4.8-109.0 - INFO    : ------
2026-04-01 18:08:58,648: xDoctor_4.8-109.0 - INFO    : SysLog
2026-04-01 18:08:58,648: xDoctor_4.8-109.0 - INFO    : ------
2026-04-01 18:08:58,648: xDoctor_4.8-109.0 - INFO    : Using Fabric as Syslog Server
2026-04-01 18:08:58,648: xDoctor_4.8-109.0 - INFO    : Not triggered ... no WARNING, ERROR, nor CRITICAL
2026-04-01 18:08:58,648: xDoctor_4.8-109.0 - INFO    : ----
2026-04-01 18:08:58,648: xDoctor_4.8-109.0 - INFO    : SNMP
2026-04-01 18:08:58,648: xDoctor_4.8-109.0 - INFO    : ----
2026-04-01 18:08:58,649: xDoctor_4.8-109.0 - INFO    : Using 10.118.165.48:162 as SNMP server
2026-04-01 18:08:58,649: xDoctor_4.8-109.0 - INFO    : Not triggered .. no WARNING, ERROR nor CRITICAL
2026-04-01 18:08:58,649: xDoctor_4.8-109.0 - INFO    : ------------
2026-04-01 18:08:58,649: xDoctor_4.8-109.0 - INFO    : ProcComplete
2026-04-01 18:08:58,649: xDoctor_4.8-109.0 - INFO    : ------------
2026-04-01 18:08:58,649: xDoctor_4.8-109.0 - WARNING : ProcComplete is disabled, please re-enable it (xdoctor --config)
2026-04-01 18:08:58,767: xDoctor_4.8-109.0 - INFO    : ----------------
2026-04-01 18:08:58,767: xDoctor_4.8-109.0 - INFO    : Session Archiver
2026-04-01 18:08:58,768: xDoctor_4.8-109.0 - INFO    : ----------------
2026-04-01 18:08:58,777: xDoctor_4.8-109.0 - INFO    : Session Stored in folder      - /usr/local/xdoctor/archive/other/2026-04-01_180344
2026-04-01 18:08:58,777: xDoctor_4.8-109.0 - INFO    : Session Archived as tar       - /usr/local/xdoctor/archive/other/xDoctor-CKMXXXXXXXXXXX-2026-04-01_180344.tgz
2026-04-01 18:08:58,777: xDoctor_4.8-109.0 - INFO    : --------------------------
2026-04-01 18:08:58,777: xDoctor_4.8-109.0 - INFO    : Session Report                - sudo xdoctor --report --archive=2026-04-01_180344
2026-04-01 18:08:58,777: xDoctor_4.8-109.0 - INFO    : ---------------
2026-04-01 18:08:58,777: xDoctor_4.8-109.0 - INFO    : Session Cleaner
2026-04-01 18:08:58,777: xDoctor_4.8-109.0 - INFO    : ---------------
2026-04-01 18:08:58,789: xDoctor_4.8-109.0 - INFO    : Removing folder  (count limit) - /usr/local/xdoctor/archive/other/2026-04-01_170120
2026-04-01 18:08:58,790: xDoctor_4.8-109.0 - INFO    : Removing archive (count limit) - /usr/local/xdoctor/archive/other/xDoctor-CKMXXXXXXXXXXX-2026-04-01_170120.tgz
2026-04-01 18:08:58,793: xDoctor_4.8-109.0 - INFO    : Cleaned 2 archived session(s)
2026-04-01 18:08:58,793: xDoctor_4.8-109.0 - INFO    : -------
2026-04-01 18:08:58,794: xDoctor_4.8-109.0 - INFO    : Emailer
2026-04-01 18:08:58,794: xDoctor_4.8-109.0 - INFO    : -------
2026-04-01 18:08:58,794: xDoctor_4.8-109.0 - INFO    : Using Dedicated Server (25:25) as SMTP Server ...
2026-04-01 18:08:58,794: xDoctor_4.8-109.0 - INFO    : Email Type = Individual Events
2026-04-01 18:08:58,795: xDoctor_4.8-109.0 - INFO    : ------------------------------
2026-04-01 18:08:58,795: xDoctor_4.8-109.0 - INFO    : xDoctor session_1775066624.943 finished in 0:05:13
2026-04-01 18:08:58,813: xDoctor_4.8-109.0 - INFO    : Successful Job:1775066624 Exit Code:192

Run the session report to review the results of the manual auto‑heal execution.

Command:

Note: Use the Session Report command provided in the xDoctor example above to review the results of the manual auto‑heal execution.

# sudo xdoctor --report --archive=<session report>

Example:

admin@ecsnode1:~> sudo xdoctor --report --archive=2026-04-01_180344

xDoctor 4.8-109.0
CKMXXXXXXXXXXX - ECS 3.8.1.4

Displaying xDoctor Report (2026-04-01_180344) Filter:[] ...

--------------------------------------------------------
FIXED - Auto Healer fixed Dell switch connectivity issue
--------------------------------------------------------
Node      = Nodes
Extra     = {"Nodes": ["169.254.1.1"]}
Timestamp = 2026-04-01_180344
PSNT      = CKMXXXXXXXXXXX @ 4.8-109.0

In the case you have a failure, open an SRn for investigation.
Example of failure:

----------------------------------------------------
ERROR - (Cached) Auto fix failed - Switch Connection Failure detected.
----------------------------------------------------
Node      = 169.254.1.1
Extra     = {"169.254.1.1": ["hare"]}
RAP       = RAP208
Solution  = KB 39838
Timestamp = 2026-04-01_180132
PSNT      = CKMXXXXXXXXXXX @ 4.8-109.0

xDoctor Auto Pilot:

This knowledge base (KB) is now automated with xDoctor Auto Pilot addressing most issues without the need for support involvement.

This feature is native to xDoctor 4-8.104.0 and above, for syntax and usage issues reference ECS: ObjectScale: How to run KB Automation Scripts (Auto Pilot).

To find the master node of the rack:

Command:

ssh master.rack

To find the NAN IP, you can use the IP identified in the alert or from getrackinfo:

Command:

admin@ecsnode1:~> getrackinfo
Node private      Node              Public                                BMC
Ip Address        Id       Status   Mac                 Ip Address        Mac                 Ip Address        Private.4(NAN)    Node Name
===============   ======   ======   =================   ===============   =================   ===============   ===============   =========
192.168.219.1     1        MA       00:00:00:00:00      0.0.0.0           00:00:00:00:00      192.168.219.101   169.254.1.1       provo-red
192.168.219.2     2        SA       00:00:00:00:00      0.0.0.0           00:00:00:00:00      192.168.219.102   169.254.1.2       sandy-red
192.168.219.3     3        SA       00:00:00:00:00      0.0.0.0           00:00:00:00:00      192.168.219.103   169.254.1.3       orem-red
192.168.219.4     4        SA       00:00:00:00:00      0.0.0.0           00:00:00:00:00      192.168.219.104   169.254.1.4       ogden-red
192.168.219.5     5        SA       00:00:00:00:00      0.0.0.0           00:00:00:00:00      192.168.219.105   169.254.1.5       layton-red
192.168.219.6     6        SA       00:00:00:00:00      0.0.0.0           00:00:00:00:00      192.168.219.106   169.254.1.6       logan-red
192.168.219.7     7        SA       00:00:00:00:00      0.0.0.0           00:00:00:00:00      192.168.219.107   169.254.1.7       lehi-red
192.168.219.8     8        SA       00:00:00:00:00      0.0.0.0           00:00:00:00:00      192.168.219.108   169.254.1.8       murray-red

Run the automation command from the master node with xDoctor 4-8.104.0 and above.

Command:
Note: --target-rack is supported for this action.

# sudo xdoctor autopilot --kb 39838 --target-rack <rack_colour>

Example:

admin@ecsnode1:~>  sudo xdoctor autopilot --kb 39838 --target-rack red
Checking for existing screen sessions...
Starting screen session 'autopilot_kb_39838_20250626_112318'...
Screen session 'autopilot_kb_39838_20250626_112318' started successfully.
Attaching to screen session 'autopilot_kb_39838_20250626_112318'...

Using /etc/ansible/ansible.cfg as config file
VERSION: 3.0
Playbook tasks: 47
Role tasks: 97
Total tasks: 144 across 1 host(s)

PLAY [red] ******************************************************************************************************************************************************************
Detected 8 hosts for this play.

TASK [target_check : set_fact] **********************************************************************************************************************************************
ok: [169.254.1.1 -> localhost] => {"ansible_facts": {"allowed_targets": "Please use: --target-rack", "target_node_check": false, "target_rack_check": true, "target_vdc_check": false}, "changed": false}

TASK [target_check : context] ***********************************************************************************************************************************************
skipping: [169.254.1.1] => {"changed": false, "false_condition": "node_script == false and target_node_check == true or rack_script == false and target_rack_check == true or vdc_script == false and target_vdc_check == true", "skip_reason": "Conditional result was False"}

...truncated

Review summary:

Example:

TASK [Print all summaries] **************************************************************************************************************************************************
ok: [169.254.1.1] => {
    "msg": [
        "*******************************************************************************",
        "Switch xDoctor 'RAP073' password and SSH summary:",
        "*******************************************************************************",
        "Validated Frontend switch(es): FAIL: The passwords for the Dell managed switch(es) are incorrect and need to be configured in the xDoctor settings according to KB 39838.",
        "Validated Backend switch(es): FAIL: The passwords for the Dell managed switch(es) are incorrect and need to be configured in the xDoctor settings according to KB 39838.",
        "Validated Backend management connections: PASS: Management connections are up and connected to the frontend switches.",
        "*******************************************************************************",
        "Validated ssh keys to switch(es): PASS: All ssh keys are valid and nothing was corrected.",
        "Validated xDoctor alert: PASS: Alert RAP073 was not present in xDoctor.",
        "*******************************************************************************"
    ]
}

TASK [Set fact for context] *************************************************************************************************************************************************
ok: [169.254.1.1 -> localhost] => {"ansible_facts": {"context": " Validated Frontend switch(es): FAIL: The passwords for the Dell managed switch(es) are incorrect and need to be configured in the xDoctor settings according to KB 39838., Validated Backend switch(es): FAIL: The passwords for the Dell managed switch(es) are incorrect and need to be configured in the xDoctor settings according to KB 39838."}, "changed": false}

TASK [Fail if validation fails] *********************************************************************************************************************************************
fatal: [169.254.1.1]: FAILED! => {"changed": false, "msg": "Review the summary above for recommendations."}

NO MORE HOSTS LEFT **********************************************************************************************************************************************************

PLAY RECAP ******************************************************************************************************************************************************************
169.254.1.1                : ok=65   changed=13   unreachable=0    failed=1    skipped=73   rescued=0    ignored=1
169.254.1.2                : ok=4    changed=0    unreachable=0    failed=0    skipped=1    rescued=0    ignored=0
169.254.1.3                : ok=4    changed=0    unreachable=0    failed=0    skipped=1    rescued=0    ignored=0
169.254.1.4                : ok=4    changed=0    unreachable=0    failed=0    skipped=1    rescued=0    ignored=0
169.254.1.5                : ok=4    changed=0    unreachable=0    failed=0    skipped=1    rescued=0    ignored=0
169.254.1.6                : ok=4    changed=0    unreachable=0    failed=0    skipped=1    rescued=0    ignored=0
169.254.1.7                : ok=4    changed=0    unreachable=0    failed=0    skipped=1    rescued=0    ignored=0
169.254.1.8                : ok=4    changed=0    unreachable=0    failed=0    skipped=1    rescued=0    ignored=0

=============================================================================================================================================================================
Status: FAIL
Time Elapsed: 0h 1m 25s
Debug log: /tmp/autopilot/log/autopilot_39838_20250626_113201.log
Message:  Validated Frontend switch(es): FAIL: The passwords for the Dell managed switch(es) are incorrect and need to be configured in the xDoctor settings according to KB 39838., Validated Backend switch(es): FAIL: The passwords for the Dell managed switch(es) are incorrect and need to be configured in the xDoctor settings according to KB 39838.
=============================================================================================================================================================================

Update the xDoctor password:

admin@ecsnode7:~> sudo xdoctor -c --expert

xDoctor Configuration Menu
--------------------------
[Expert Mode Active]

(1)  Overview
(2)  Scheduling
(3)  Archiving
(5)  Repository



(9)  Miscellaneous

(0)  Exit

Please make a choice: 9

xDoctor Miscellaneous
---------------------


(3)  Switches
(4)  Remove Hardware Alerting Timestamp

(0)  Main menu

Please make a choice: 3

xDoctor Switch Settings
---------------------
Enable Switch Analysis?  [Yes]:
Switches [hare,rabbit,fox,hound]:
Username [admin]:
Password [*****]:

[New Switch Settings]
Enabled = Yes
Switches = hare,rabbit,fox,hound
Username = admin
Password = *****

> Issue new settings?  [No]: yes
2024-11-20 16:03:53,702: xDoctor_4.8-100.0 - INFO    : Settings saved and distributed ...

xDoctor Miscellaneous
---------------------


(3)  Switches
(4)  Remove Hardware Alerting Timestamp

(0)  Main menu

Base KB Automation:
ECS: xDoctor: RAP073: Switch Connection Failure detected

Additional KB consolidation into this automation:
ECS: xDoctor reports Switch Connection Failure due to RSA key in known_hosts

Affected Products

Elastic Cloud Storage

Products

Elastic Cloud Storage

Article Number: 000039838

Article Type: Solution

Last Modified: 02 أبريل 2026

Version: 10

Check if your device is covered by Support Services.

ECS: OBS: xDoctor: RAP073/208: Switch Connection Failure detected

Summary: This knowledge base explains how to handle alert Switch Connection Failure detected.

Symptoms

Cause

Resolution

Affected Products

Symptoms

Alert That Triggers RAP208 Auto‑Healing

Alert Output Example

Auto Healer Remediation (Example)

Auto Healer Requirement

Cause

Resolution

xDoctor Auto Pilot:

Affected Products

Products

Article Properties

Find answers to your questions from other Dell users

Support Services

Article Properties

Find answers to your questions from other Dell users

Support Services

ECS: OBS: xDoctor: RAP073/208: Switch Connection Failure detected

Summary: This knowledge base explains how to handle alert Switch Connection Failure detected.

Detailed Article

Symptoms

Cause

Resolution

Affected Products

Symptoms

Alert That Triggers RAP208 Auto‑Healing

Alert Output Example

Auto Healer Remediation (Example)

Auto Healer Requirement

Cause

Resolution

xDoctor Auto Pilot:

Affected Products

Products

Article Properties

Find answers to your questions from other Dell users

Support Services

Article Properties

Find answers to your questions from other Dell users

Support Services