Dell Orchestrator: Repository TLS Fails to Verify Certificate

Certificate errors are seen in Kubernetes or pod events:

tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2025-09-28T14:55:57+01:00 is after 2025-09-27T14:35:33Z

Issue is commonly seen when attempting to restart pods, and the pod image is being pulled from the repository:

root@orchestrator:/tmp# kubectl get pod -n hzp | grep catalog
hzp-catalog-svc-66b58cb94c-44mr5                  0/2     Init:ErrImagePull       0             2s
hzp-catalog-svc-fb946bbfd-qgrmp                   2/2     Running                 0             11d

root@orchestrator:/tmp# kubectl describe pod hzp-catalog-svc-66b58cb94c-44mr5 -n hzp

>>LINES REMOVED

Events:
  Type     Reason     Age                    From               Message
  ----     ------     ----                   ----               -------
  Warning  Failed     43m (x6 over 44m)      kubelet            Error: ImagePullBackOff
  Normal   Pulling    43m (x4 over 44m)      kubelet            Pulling image "repository.local.edge/orchestrator/busybox:1.37.0"
  Warning  Failed     43m (x4 over 44m)      kubelet            Error: ErrImagePull
  Warning  Failed     43m                    kubelet            Failed to pull image "repository.local.edge/orchestrator/busybox:1.37.0": failed to pull and unpack image "repository.local.edge/orchestrator/busybox:1.37.0": failed to resolve reference "repository.local.edge/orchestrator/busybox:1.37.0": failed to do request: Head "https://repository.local.edge/v2/orchestrator/busybox/manifests/1.37.0": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2025-09-28T14:55:57+01:00 is after 2025-09-27T14:35:33Z

• Orchestrator communication with a repository depends on certificate trust.
• If a self-signed certificate or a certificate signed by Certificate Authority has expired, the orchestrator cannot pull images from the repository.

Prerequisites:

• A valid self-signed, or Certificate Authority signed certificate.
• The certificate should be added to the repository host or application.
• The updated certificate should be present in the web browser when visiting the repository's UI.

Attempt to pull the container again, and confirm that the error message has changed.

root@orchestrator:~# kubectl describe pod hzp-catalog-svc-66b58cb94c-9kwtr -n hzp

>>LINES REMOVED

Events:
  Type     Reason     Age                     From               Message
  ----     ------     ----                    ----               -------
  Warning  Failed     7m31s (x6 over 8m44s)   kubelet            Error: ImagePullBackOff
  Normal   Pulling    7m20s (x4 over 8m45s)   kubelet            Pulling image "repository.local.edge/orchestrator/busybox:1.37.0"
  Warning  Failed     7m20s (x4 over 8m44s)   kubelet            Failed to pull image "repository.local.edge/orchestrator/busybox:1.37.0": failed to pull and unpack image "repository.local.edge/orchestrator/busybox:1.37.0": failed to resolve reference "repository.local.edge/orchestrator/busybox:1.37.0": failed to do request: Head "https://repository.local.edge/v2/orchestrator/busybox/manifests/1.37.0": tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "ca.local.edge")
  Warning  Failed     7m20s (x4 over 8m44s)   kubelet            Error: ErrImagePull
  Normal   BackOff    3m32s (x23 over 8m44s)  kubelet            Back-off pulling image "repository.local.edge/orchestrator/busybox:1.37.0"

Ensure the orchestrator host trusts the repository's new certificate by pulling it from the remote host, and updating the CA certificates.

root@orchestrator:~# echo | openssl s_client -showcerts -connect repository.local.edge:443 2>/dev/null | openssl x509 > /usr/local/share/ca-certificates/repository.crt
root@orchestrator:~# sudo update-ca-certificates

The certificate is stored as a generic secret. The secret is stored in the orchestrator namespace, as registry.crt in the secret eo-registry-secret. In this example, the orchestrator namespace is hzp.

root@orchestrator:~# kubectl describe secret eo-registry-secret -n hzp
Name:         eo-registry-secret
Namespace:    hzp
Labels:       <none>
Annotations:  <none>

Type:  data

Data
====
password:        11 bytes
registry.crt:    2573 bytes
repo.secretRef:  13 bytes
repo.url:        34 bytes
username:        6 bytes

Pull the repository's certificate locally, and format it in Base64 for Kubernetes. Create a patch file to update registry.crt in the eo-registry-secret under the hzp namespace.

root@orchestrator:~# base64 -w0 /usr/local/share/ca-certificates/repository.crt > registry.crt.b64
root@orchestrator:~#
root@orchestrator:~# echo "data:" > secret-patch.yaml
root@orchestrator:~# echo "  registry.crt: $(cat registry.crt.b64)" >> secret-patch.yaml
root@orchestrator:~#
root@orchestrator:~# kubectl patch secret eo-registry-secret -n hzp --patch-file secret-patch.yaml

Last, restart the container runtime, to ensure that the new certificate is used. 

If containerd is used:

root@orchestrator:~# systemctl restart containerd

If docker is used:

root@orchestrator:~# systemctl restart docker

Confirm that images can be pulled from the repository.

root@orchestrator:~# ctr images pull registory.local.edge/orchestrator/busybox:1.37.0
registory.local.edge/orchestrator/busybox:1.37.0:                                 resolved       |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:5bec42d526660d3e926af429a337f0a968df8059bd902434e0e4670697c9cf83: exists         |++++++++++++++++++++++++++++++++++++++|
layer-sha256:05b9fdbb044a7ecdb5ec3cb0fac492d667ef57df990d3b0254c8c800d5c31eb0:    done           |++++++++++++++++++++++++++++++++++++++|
config-sha256:6d3e4188a38af91b0c1577b9e88c53368926b2fe0e1fb985d6e8a70040520c4d:   done           |++++++++++++++++++++++++++++++++++++++|
elapsed: 0.2 s                                                                    total:   0.0 B (0.0 B/s)
unpacking linux/amd64 sha256:5bec42d526660d3e926af429a337f0a968df8059bd902434e0e4670697c9cf83...
done: 12.069718ms

Check if the pod has now come up. 

root@orchestrator:~# kubectl get po -n hzp | grep hzp-catalog-svc
hzp-catalog-svc-66b58cb94c-9kwtr                  2/2     Running     0             14s