Dell NativeEdge: Troubleshooting Dell NativeEdge Installation: Pod Startup Issues

During the installation of Dell NativeEdge, you may encounter a stall while the Helm chart rolls out. The latest message of the installation script execution is looking like:

Operation Type: INSTALL Operation Status: IN_PROGRESS ChartKey: NEO
Operation Type: INSTALL Operation Status: FAILED ChartKey: NEO
native-edge-orchestrator has NOT deployed and has status: failed
....
Saving logs to files
Cleaning up the execution log file

In some environments, the container POD blocks external DNS requests. To verify this issue, check the logs of the pod stuck during deployment:

kubectl get po -A
...
nativeedge        hzp-product-svc-595b7b888d-tpz88                          2/3     Running            8 (54m ago)      80m
...
nativeedge        fusion-stage-backend-845dfff4d7-9zqww                     1/2     CrashLoopBackOff   20 (4m34s ago)   79m
...
kubectl logs fusion-stage-backend-845dfff4d7-9zqww -c hzp-iam-proxy -n hzp | grep error -B17
...
runtime/debug.Stack()
	/usr/local/go/src/runtime/debug/stack.go:26 +0x5e
eos2git.cec.lab.emc.com/ISG-Edge/hzp-iam-proxy/server/commons/utils.LogRecover(...)
	/app/server/commons/utils/utils.go:37
eos2git.cec.lab.emc.com/ISG-Edge/hzp-iam-proxy/server/services/proxysvc.(*proxyService).pipeConnections.func1()
	/app/server/services/proxysvc/service.go:236 +0x174
panic({0xdbb580?, 0x177f950?})
	/usr/local/go/src/runtime/panic.go:785 +0x132
io.copyBuffer({0x0, 0x0}, {0x0, 0x0}, {0xc00057e000, 0x2000, 0x2000})
	/usr/local/go/src/io/io.go:429 +0x17e
io.CopyBuffer({0x0?, 0x0?}, {0x0?, 0x0?}, {0xc00057e000?, 0x1?, 0x1?})
	/usr/local/go/src/io/io.go:402 +0x36
eos2git.cec.lab.emc.com/ISG-Edge/hzp-iam-proxy/server/services/proxysvc.(*proxyService).pipeConnections(0xc0001b0b00, {0x10a17a8, 0xc0005a7200}, 0xc0003f4b00, 0xc0003f4b10, {0xecbefe, 0x2}, 0xc000483f20?)
	/app/server/services/proxysvc/service.go:254 +0x2ae
created by eos2git.cec.lab.emc.com/ISG-Edge/hzp-iam-proxy/server/services/proxysvc.(*proxyService).proxyConnections in goroutine 7423
	/app/server/services/proxysvc/service.go:224 +0x1db

{"level":"error","error":"no such file or directory","time":"2025-05-09T13:16:16Z","message":"Error getting socket from operating system"}
{"level":"error","error":"no such file or directory","time":"2025-05-09T13:16:16Z","message":"failed to get destination address from outgoing connection"}
...
## Note: if during the installation you specified your own namespace, then you will have to provide your namespace after -n argument key.

kubectl log fusion-stage-backend-845dfff4d7-9zqww -c hzp-iam-proxy -n hzp | grep "Forwarding connection id"
...
{"level":"info","time":"2025-05-09T13:16:16Z","message":"Forwarding connection id: 37da2adc-9f56-4e93-8875-83dd597c1aab, source: 161.200.192.67:43732, destination: , direction: Outbound, encryption: NotDetermined"}
...

## Note: the destination is empty in this part: destination: , direction: Outbound, encryption: NotDetermined"}
## Also note the "error getting socket from operating system, this is another symptom of this particular issue

Note, in this installation fusion-stage-backend and hzp-product-svc pods are not coming up. Check the logs of the pod stuck during deployment to verify this issue.

For this kind of errors in the hzp-iam-proxy container, there is a known workaround for this type of issue.

To fix this issue, do the following steps.

1. Restart the installation script to pick up the latest state and wait until the same installation step starts.

Operation Type: INSTALL Operation Status: IN_PROGRESS ChartKey: NEO

2. Start monitoring the pods are coming up with kubectl get po -A command.
3. When you see that some pod is not coming up, check that the logs from the 'Cause' section appear in the logs.
4. When you identified this issue on step 3, do these steps for three deployments: fusion-stage-backend, hzp-catalog-svc, hzp-product-svc

If you see the error only within one of the pods, ensure that you add the following annotations to all three pods.

kubectl edit deployment name_of_deployment -nhzp
# For example:
kubectl edit deployment fusion-stage-backend -nhzp
# Find the spec.template.metadata.annotations section:
# For example, you should see something like:
....
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: stage-backend
  strategy:
    ....
    type: RollingUpdate
  template:
    metadata:
      annotations:
        hzp.iam.webhook/authentication-enabled: "false"
        hzp.iam.webhook/authorization-enabled: "false"

....

# Add annotation 'hzp.iam.webhook/exclude-outbound-ports: "443"' to this section as below:
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: stage-backend
  strategy:
    ...
    type: RollingUpdate
  template:
    metadata:
      annotations:
        hzp.iam.webhook/authentication-enabled: "false"
        hzp.iam.webhook/authorization-enabled: "false"
        hzp.iam.webhook/exclude-outbound-ports: "443"
# check the deployment is restarting. For example:
kubectl get po -A | grep fusion-stage
hzp               fusion-stage-backend-5887976b57-4shlp               2/2     Running       0               23s
hzp               fusion-stage-backend-745ff99c8f-dkl7t               2/2     Terminating   4 (7d21h ago)   7d21h

This unblocks external DNS lookup queries and lets your installation continue.

After the pod restarts, the installation will continue.

Additional notes:

There is evidence that the problem happens after the installation completes. This means that this problem may affect other pods. In case you see same footprint in the logs of the pod, you must perform same steps for the deployment. Example of error appearing in the other pod below:

{"level":"debug","time":"2025-06-04T12:37:33Z","message":"Accepted connection with flow id: c565d0f3-d568-42e0-a0a6-21cd0080b6ee, remote address: 161.200.27.4:41880, local address: 127.0.0.1:15001, direction: Outbound"}
recovered while piping connections, flow id: %sa6178189-f353-46d2-8c07-20e67a8c72d9(0xdbb580,0x177f950)
{"level":"debug","time":"2025-06-04T12:37:33Z","message":"Open connections: 17"}goroutine 513252 [running]:runtime/debug.Stack()
/usr/local/go/src/runtime/debug/stack.go:26 +0x5eeos2git.cec.lab.emc.com/ISG-Edge/hzp-iam-proxy/server/commons/utils.LogRecover(...)
/app/server/commons/utils/utils.go:37eos2git.cec.lab.emc.com/ISG-Edge/hzp-iam-proxy/server/services/proxysvc.(*proxyService).pipeConnections.func1()
/app/server/services/proxysvc/service.go:236 +0x174
panic({0xdbb580?, 0x177f950?})
/usr/local/go/src/runtime/panic.go:785 +0x132io.copyBuffer({0x0, 0x0}, {0x0, 0x0}, {0xc00064a000, 0x2000, 0x2000})
/usr/local/go/src/io/io.go:429 +0x17eio.CopyBuffer({0x0?, 0x0?}, {0x0?, 0x0?}, {0xc00064a000?, 0x1?, 0x1?})
/usr/local/go/src/io/io.go:402 +0x36eos2git.cec.lab.emc.com/ISG-Edge/hzp-iam-proxy/server/services/proxysvc.(*proxyService).pipeConnections(0xc0001e6580, {0x10a17a8, 0xc0004811a0}, 0xc00050c930, 0xc00050c8f0, {0xecbefc, 0x2}, 0xc0001f4b70?)
/app/server/services/proxysvc/service.go:254 +0x2aecreated by eos2git.cec.lab.emc.com/ISG-Edge/hzp-iam-proxy/server/services/proxysvc.(*proxyService).proxyConnections in goroutine 513234
/app/server/services/proxysvc/service.go:225 +0x271
{"level":"debug","time":"2025-06-04T12:37:33Z","message":"Open connections: 16"}
{"level":"info","time":"2025-06-04T12:37:33Z","message":"Redirect Address before lookup 161.210.184.19:80"}
recovered while piping connections, flow id: %sa6178189-f353-46d2-8c07-20e67a8c72d9(0xdbb580,0x177f950)goroutine 513251 [running]:runtime/debug.Stack()
/usr/local/go/src/runtime/debug/stack.go:26 +0x5eeos2git.cec.lab.emc.com/ISG-Edge/hzp-iam-proxy/server/commons/utils.LogRecover(...)
/app/server/commons/utils/utils.go:37eos2git.cec.lab.emc.com/ISG-Edge/hzp-iam-proxy/server/services/proxysvc.(*proxyService).pipeConnections.func1()
/app/server/services/proxysvc/service.go:236 +0x174
panic({0xdbb580?, 0x177f950?})
/usr/local/go/src/runtime/panic.go:785 +0x132io.copyBuffer({0x0, 0x0}, {0x0, 0x0}, {0xc00071c000, 0x2000, 0x2000})
/usr/local/go/src/io/io.go:429 +0x17eio.CopyBuffer({0x0?, 0x0?}, {0x0?, 0x0?}, {0xc00071c000?, 0x1?, 0x1?})
/usr/local/go/src/io/io.go:402 +0x36eos2git.cec.lab.emc.com/ISG-Edge/hzp-iam-proxy/server/services/proxysvc.(*proxyService).pipeConnections(0xc0001e6580, {0x10a17a8, 0xc0004811a0}, 0xc00050c8f0, 0xc00050c930, {0xecbefe, 0x2}, 0xc0001f47e0?)
/app/server/services/proxysvc/service.go:254 +0x2aecreated by eos2git.cec.lab.emc.com/ISG-Edge/hzp-iam-proxy/server/services/proxysvc.(*proxyService).proxyConnections in goroutine 513234
/app/server/services/proxysvc/service.go:224 +0x1db
{"level":"error","error":"no such file or directory","time":"2025-06-04T12:37:33Z","message":"Error getting socket from operating system"}

## Note the error "getting socket from operation system" again

Preventive troubleshooting

Instead of waiting for failure during the installation, you may start doing steps 2-4 during your first installation.

When this message shown on the screen for the 10 minutes

Operation Type: INSTALL Operation Status: IN_PROGRESS ChartKey: NEO

Start monitoring the pods. If you see the footprint of the issue in the fusion-stage-backend-svc and product-svc pods, apply the annotations for both of the pods. Also apply the annotation to the catalog-svc deployment, which may encounter this issue later.