ECS: REST is not reachable on any node on VDC: Unable to read a valid object-main_network.json after new install of 3.8

Summary: After a new install of ECS version of 3.8, ssh keys must be exchanged between VDCs. Results may be incomplete and report error 'Unable to read a valid object-main_network.json'

This article applies to This article does not apply to This article is not tied to any specific product. Not all product versions are identified in this article.
Check out other resources

Symptoms

Command svc_vdc list -d displays the following errors and warnings:

admin@ecsnode1:~> svc_vdc list -d
svc_vdc v1.0.28 (svc_tools v2.10.0)                 Started 2023-02-28 08:06:28

WARNING   Fabric node discovery failed.  Results may be incomplete. 
          Error was: Unable to read a valid object-main_network.json
Name: VDC1
  Local VDC:                  True

  Serial #:                   CKM01224123456
  ECS Release:                3.8.0.1 General Patch 1 
  Object Version:             3.8.0.1-138599.c85be50ba8f.1
  VDC ID:                     urn:storageos:VirtualDataCenterData:75bdc6b5-8d47-4e16-8fcf-c81565123456
  Short form VDC ID:          75bdc6b5-8d47-4e16-8fcf-c81565123456
  Secret Key:                 ***
  TSO Status:                 Healthy
  Permanently failed:         False
  Encryption Enabled:         True

  Storage pools (cos):
  SP02                       urn:storageos:VirtualArray:31f01234-7e8e-4c7e-873c-69b012345678

    Node                 managementEndPoints       interVdcCmdEndPoints      interVdcEndPoints        

    r1n1                 10.x.x.x              	   10.x.x.x                  10.x.x.x             
    r1n2                 10.x.x.x                  10.x.x.x                  10.x.x.x             
    r1n3                 10.x.x.x                  10.x.x.x                  10.x.x.x             
    r1n4                 10.x.x.x              	   10.x.x.x                  10.x.x.x             
    r1n5                 10.x.x.x                  10.x.x.x                  10.x.x.x             

ERROR     REST is not reachable on any node on vdc 'VDC2'.
ERROR     Unexpected fatal error while running REST command.

Name: VDC2
  Local VDC:                  False

  Serial #:                   Unknown
  Object Version:             None
  VDC ID:                     urn:storageos:VirtualDataCenterData:4a798bac-f41a-4896-846f-3b1234567890
  Short form VDC ID:          4a798bac-f41a-4896-846f-3b1234567890
  Secret Key:                 ***
  TSO Status:                 Healthy
  Permanently failed:         False
  Encryption Enabled:         True

  [Could not determine storage pool info]

    Node                 managementEndPoints       interVdcCmdEndPoints      interVdcEndPoints        


    <Unknown>*           10.x.x.x                  10.x.x.x                  10.x.x.x             
    <Unknown>*           10.x.x.x                  10.x.x.x                  10.x.x.x             
    <Unknown>*           10.x.x.x                  10.x.x.x                  10.x.x.x             
    <Unknown>*           10.x.x.x                  10.x.x.x                  10.x.x.x             
    <Unknown>*           10.x.x.x                  10.x.x.x                  10.x.x.x             

* - Unknown nodes indicate either node IPs couldn't be discovered, or endpoints are misconfigured

NOTE:  For a full list of network IPs/interfaces, use 'svc_network list' (local) or 'svc_network list -r' (all vdcs)

Also when using SSH to remote VDC nodes, a password is requested:

admin@ecsnode1:~> ssh 10.x.x.x
Password:

Cause

After a new installation of ECS 3.8, the SSH keys must be synchronized between VDCs.

Resolution

If the above symptoms are seen, open a Support case with Dell and reference this knowledge article.

Affected Products

ECS

Products

ECS Appliance, Elastic Cloud Storage
Article Properties
Article Number: 000210252
Article Type: Solution
Last Modified: 25 Nov 2024
Version:  4
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.