ECS: GUI Error 6500 (http: 500): Internal Server Error. An internal server error occurred.

When logging into the UI on a specific node, the following error is seen on the dashboard or end user license agreement notification:

There are three ECS services responsible for UI login, stat, "ecsportalsvc," and "obcontrolsvc" - these have stale entries which are not allowing the UI to log in.

On the concerning node, restart the stat, ecsportalsvc, and obcontrolsvc. There is no impact on restarting these services but it should be noted that any alerts generated on the UI on another node are now linked to these restarts:

Using service tools:

admin@node1:~> svc_node services restart -nw 10 -srv stat ecsportalsvc objcontrolsvc -n self
Starting utility inside a screen session

Confirm on the concerning node (rack 1 node 1) that the services have restarted and ensure that there are no services that are not running:

admin@node1:~> svc_node services -status
svc_node v1.2.1 (svc_tools v2.1.0)                 Started 2022-02-25 11:40:49

Node r1n1:
    Services running:
        fabric-agent

        blobsvc        (102925) Started: 2021-12-20 17:07:17      cm             (102859) Started: 2021-12-20 17:07:17
        coordinatorsvc (102843) Started: 2021-12-20 17:07:17      dataheadsvc    (102935) Started: 2021-12-20 17:07:17
        dtquery        (102951) Started: 2021-12-20 17:07:17      ecsportalsvc   (66505)  Started: 2022-02-25 11:31:08
        eventsvc       (102932) Started: 2021-12-20 17:07:17      georeceiver    (102950) Started: 2021-12-20 17:07:17
        metering       (102908) Started: 2021-12-20 17:07:17      objcontrolsvc  (69329)  Started: 2022-02-25 11:31:32
        resourcesvc    (102939) Started: 2021-12-20 17:07:17      rm             (102851) Started: 2021-12-20 17:07:17
        sr             (102905) Started: 2021-12-20 17:07:17      ssm            (102899) Started: 2021-12-20 17:07:17
        stat           (65535)  Started: 2022-02-25 11:30:55      storageserver  (102896) Started: 2021-12-20 17:07:17
        vnest          (102847) Started: 2021-12-20 17:07:17

    Services not running:
        <none>

Return to the ECS UI and you can log in without issue.