Dell NativeEdge: Creating HA Group Fails With 500 Internal Server Error

Summary: When creating an HA Group in a two‑node NativeEdge cluster with a Witness node, the operation may fail with a 500 Internal Server Error. Investigation shows that the failure occurs because Endpoint and Orchestrator system clocks are not synchronized, causing the HA configuration request sent over NATS to expire before the ECE agent processes it. ...

This article applies to This article does not apply to This article is not tied to any specific product. Not all product versions are identified in this article.
Check out other resources

Symptoms

  • Attempting to create an HA Group fails with: 500 (Internal Server Error) during POST to /api/v2/clusters/<cluster_id>/ha-groups
  • NATS request for leader update times out with error: "leader update fails with nats: Timeout" (observed in HA configuration handling)
  • In ece-agent.log, the HA configurations request immediately returns "request expired" when it reaches the Endpoint. Example from logs:
{"level":"error","component":"hzp-ece-agent","marker":"ha-configuration-handler",
"message":"request expired"}
  • Environment logs reveal:
    • Endpoint time is ~20 seconds ahead of Orchestrator,
    • Orchestrator is using systemd-timesyncd but no NTP server configured,
    • Endpoint is using chrony syncing to ntp.dell.com.

Cause

The issue is caused by time drift/NTP misconfiguration between Orchestrator and Endpoint.

  • Endpoint and Orchestrator must be configured with the same NTP server or maintain clock skew < 9 seconds.
  • The HA configuration create event sets a 10‑second NATS request expiry, but because Endpoint time was ahead of Orchestrator, the request was already considered expired upon arrival.
  • As a result, Endpoint drops the request without replying, causing the Orchestrator to interpret the missing response as a NATS timeout.

Resolution

To resolve the issue, ensure that Orchestrator and Endpoint are synchronized using the same NTP source, or have a clock skew less than 9 seconds.

Recommended configuration

Engineering guidance from the defect:

  • Configure Orchestrator as an NTP server.
  • Point Endpoint to the Orchestrator NTP service so both remain synchronized.
  • (Optional but recommended) Configure Orchestrator itself to sync to a public or corporate NTP server to maintain accurate time.

Environment‑specific workaround

If Endpoint syncs using chrony to ntp.dell.com and Orchestrator has no NTP server configured (default systemd-timesyncd), choose one of the following:

  1. Configure Orchestrator to use the same external NTP server as Endpoint, OR
  2. Configure Orchestrator as a local NTP server and point Endpoint nodes to it.

After time is corrected

Retry creating the HA Group — the operation should succeed without error.

Affected Products

NativeEdge Solutions
Article Properties
Article Number: 000443520
Article Type: Solution
Last Modified: 06 ذو القعدة 1447
Version:  1
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.