Dell Automation Platform: MTLS certificates Expiration Issue

• NEO user interface Inaccessible - Users cannot access the NativeEdge Orchestrator web interface.
• Pod Startup Failures - Many pods fail to start due to certificate validation errors.
• Service Communication Breakdown - Services cannot establish secure communication with each other.
• Vault Service Unresponsive - The vault service becomes unresponsive due to mTLS handshake failures.
• Specific Error Messages:
  • x509: certificate has expired or is not yet valid
  • remote error: tls: bad certificate
  • tls: failed to verify certificate
  • TLS Handshake Failed
• Widespread Impact - All services using the hzp-iam-proxy sidecar for mTLS encryption are affected.
• Connection Refused Errors - Services attempting to communicate with vault receive connection refused errors.
• Misleading Rotation Success - The cron job may appear to run successfully (weekly mTLS certificate renewal), but certificates still expire due to CA certificate limitation.

The root cause is the expiration of mTLS certificates used by the hzp-iam-proxy sidecar for secure service-to-service communication. These certificates expire approximately one year after initial deployment, and the automated rotation mechanism has a critical limitation that prevents proper certificate renewal.

Technical Details

• Certificate Expiration - mTLS certificates expire after one year from initial deployment (for example, 2026-05-19T10:57:15Z)
• CA Certificate Limitation - The critical issue is that newly created mTLS certificates cannot be valid beyond the period the CA certificate is valid to. This means, mTLS certificates get renewed every time the cron job runs (weekly), but CA certificate only gets renewed if it meets the DAYS_TO_EXPIRATION condition. Even when mTLS certificates are renewed weekly, their expiry date does not extend beyond the CA certificate's expiry. This creates a situation where certificates appear to be renewed but still expire when the CA certificate expires.
• Configuration Issue - The internal-mtls-rotation cron job has problematic default settings:
• DAYS_TO_EXPIRATION: "7" - Only rotates CA certificates with 7 days or less to expiry
• Schedule: 0 3 * * 0 - Runs once a week at 3 AM on Sunday
• Single Point of Failure - If the CA certificate rotation job fails during the critical 7-day window before expiration, all certificates expire regardless of weekly mTLS certificate renewal.
• Cascade Failure - Expired certificates prevent the hzp-iam-proxy sidecar from: Verifying incoming connections from other services, Establishing outbound connections to other services, and Properly terminating TLS handshakes
• Vault Cluster Impact - Vault cluster nodes cannot communicate with each other, making the entire vault service unresponsive. 
• Rotation Job Failure - The internal-mtls-rotation jobs themselves fail with context deadline exceeded errors when trying to communicate with vault, creating a deadlock situation.

Affected Components

• All three vault pods (edgevault-0, edgevault-1, edgevault-2)
• All services with hzp-iam-proxy sidecars (fusion-secrets, fusion-rest-service, hzp-* services, and so forth)
• Both inbound and outbound mTLS connections
• Internal service-to-service communication

Workaround:

The following steps were successfully used to resolve the issue by manually bypassing the certificate rotation limitations:

1. Remove Vault Certificate timestamp

kubectl patch secret eo-vault-imtls-secret --type=json -p='{"op": "remove", "path": "/data/lastUpdated"}' -n [orchestrator-namespace]

2. Scale down Vault Pods

kubectl scale statefulset edgevault --replicas=0 -n [orchestrator-namespace]

3. Disable IAM Webhook temporarily.

kubectl patch statefulset edgevault -p '{"spec":{"template":{"metadata":{"labels":{"hzp.iam.webhook/active":"false"}}}}}' -n [orchestrator-namespace]

4. Scale up Vault Pods

kubectl scale statefulset edgevault --replicas=3 -n [orchestrator-namespace]

5. Manually trigger Certificate Rotation

kubectl create job --from=cronjob/internal-mtls-rotation internal-mtls-rotation-01 -n [orchestrator-namespace]

6. Restore Vault to default settings.

kubectl create job --from=cronjob/internal-mtls-rotation internal-mtls-rotation-01 -n [orchestrator-namespace]

7. Cleanup temporary job

kubectl create job --from=cronjob/internal-mtls-rotation internal-mtls-rotation-01 -n [orchestrator-namespace]

After applying the workaround:

1. Verify that the vault pods are running and healthy.
2. Check that services can establish mTLS connections.
3. Confirm that the NEO user interface is accessible.
4. Monitor logs for any remaining certificate-related errors.
5. Verify that the internal-mtls-rotation job completes successfully.

Fix:

This issue is addressed in Dell Automation Platform 2.2.

Also:

• For OpenSearch certificate issues, use the specific OpenSearch certificate renewal routine as provided separately.
• If you want to avoid this situation before upgrading to Dell Automation Platform 2.2, contact Dell Support.