Dell NativeEdge: NativeEdge Orchestrator Upgrade Fails To Start - Error Code 0x019020004/0x01903000B

During an upgrade attempt for the NativeEdge Orchestrator (NEO) the following is observed:

Verify bundle, this could take some time...
Uploaded the eo bundle successfully
Checking eo bundle upload status for upload ID: (random ID number)
EO bundle upload and validations have failed. Error code: 0x019020004

After this point, the upgrade fails and will not continue.
Similar issues have been observed when upgrading from NEO 3.0 to NEO 3.1.

Waiting for EO image loader wrapper to start reading updates
Image loader wrapper has failed to start
login to registry...
Error: authenticating creds for "registryexampleFQDN.local.edge/dell": pinging container registry registryexampleFQDN.local.edge: Get "https://registryexampleFQDN.local.edge/v2/": dial tcp: lookup registryexampleFQDN.local.edge: no such host
failed to login to registry...
Getting image loader wrapper logs
EO Image Loader wrapper has failed. Please check the issue and fix the same. Error code: 0x01903000B

This issue is an environmental issue, which is common on systems where Domain Name System (DNS) is not used.

During this phase of the upgrade the NEO is attempting to login into the registry that is already defined, with the credentials that are already saved in the NEO namespace.

Step 1: 
Validate that the user name and password is correct to login to the registry.

docker login -u uservariable registryexampleFQDN.local.edge 

Step 2:

Check what status the eo-images-loader-wrapper-job container is in and retrieve its logs.

kubectl logs -f eo-images-loader-wrapper-job-k7qxq -n hzp
login to registry...
Error: authenticating creds for "registryexampleFQDN.local.edge/dell": pinging container registry registryexampleFQDN.local.edge: Get "https://registryexampleFQDN.local.edge/v2/": dial tcp: lookup registryexampleFQDN.local.edge: no such host
failed to login to registry...

If step 1 from the cause section validates correctly, this indicates that your system can resolve the registry and login to it.
If the step fails, then:

• Validate network connectivity to the registryfqdn.
• Correct username/password secret for registry in NativeEdge Kubernetes secrets.
• Validate that your certificate for your registry has not expired.

If step 2 from the cause section fails. this indicates that there is a DNS issue within your Kubernetes cluster.

• If your DNS is external to the cluster, validate your DNS values for the registry.
• If the user is not using DNS but using static entries in your /etc/hosts to validate the Fully Qualified Domain Name (FQDN) then the user must review the Kubernetes settings of the cluster.

Here we are using Alpine to troubleshoot the issue, but any similar pod like busybox may be used as an alternative.

kubectl run -it --rm alpine --image=public.ecr.aws/docker/library/alpine:3.18 --restart=Never -- /bin/sh
##(modify this command as required)
If you don't see a command prompt, try pressing enter.
/ # nslookup registryexampleFQDN.local.edge
Server:         192.168.0.100
Address:        192.168.0.100:53
 
** server can't find registryexampleFQDN.local.edge: NXDOMAIN

• Examine the configmap of the coredns pod

kubectl -n kube-system edit configmap/coredns

• The user may need to add a static host entry of the registry to the coredns configmap. 
• In the example below we have added a host in the coredns configmap of 192.168.10.50 for the host registryexampleFQDN.local.edge

.:53 {
    errors
    health
    ready
    kubernetes cluster.local in-addr.arpa ip6.arpa {
        pods insecure
        fallthrough in-addr.arpa ip6.arpa
        ttl 30
    }
    prometheus :9153
    hosts {
        192.168.10.50 registryexampleFQDN.local.edge
        fallthrough
    }
    forward . /etc/resolv.conf {
       max_concurrent 1000
    }
    cache 30
    loop
    reload
    loadbalance
}

Revalidate the DNS configuration by running a lookup using the pod.