Multiple problems with SSL Certificates with iDRAC firmware 2.81.81.81

Question

Following the (largely unannounced) change in iDRAC 2.81.81.81 firmware to require `ManualDNSEntry ` be set in order to access the web-interface from hostnames other than the one set by "iDRAC Settings -> Network -> Common Settings" which (bug #1) doesn't actually appear to correctly discover the device's DNS Domain Name even when "Auto Config Domain Name" is set - I started looking at uploading a custom SSL certificate to the controller.

… and I've encountered a myriad of issues with this, and indeed the functionality appears broken in several key ways.

The summary is that Let's Encrypt certificates of the form `*.dellr330.domain.tld` (e.g. wildcard) and `idrac.dellr330.domain.tld` don't work with iDRAC (bug #2), without any validation errors and leave the web interface in an unusable state (bug #3), can't always be deleted depending on the format in which they were uploaded(!) (bug #4), and don't seem to be able to be worked around by disabling HTTPS redirects (bug #5).

Given a valid SSL certificate from any provider, the iDRAC system should accept and correctly use the provided certificate. The uploads interface (either via RACADM or the web interface) accepts as the provided certificate file either the stand-alone certificate or the certificate chain without reporting any warning or error, but doesn't work in either case. Once either form of the certificate has been uploaded by either method (web or RACADM) as two files in PEM format for as a single password-protected PKCS#12 and the iDRAC controller is restarted, then the iDRAC system fails to respond correctly to SSL requests:

$ openssl s_client -connect idrac.dellr330.domain.tld:443
CONNECTED(00000003)
write:errno=0
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 331 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---

… noting that the output is the same if attempting to access by IP.

At this point, the valid-but-not-correctly-handled certificate can only be removed byresetting the iDRAC settings to factory defaults from pre-boot EFI, or through direct RACADM (as remote RACADM access is also broken at this point).

If the certificate was uploaded in PKCS#12 format, then the RACADM `sslcertdelete -f 3` command can be used to remove it. However, if the certificate was uploaded as two files in PEM format, then this command will fail saying that no custom certificate is present - in which case, the only option for recovery (other than a factory reset) is to re-upload the (incorrectly handled) PKCS#12 form of the same certificate, and then to again issue the same deletion command, which the succeeds.

Finally, as a mitigation there appeared to be the option to set `iDRAC.Webserver.HttpsRedirection` to disabled - but when this is enabled - it is only possible to access the login page, and attempting a login with correct credentials simply returns back to that same login page - so this setting no longer appears to have any utility (or not in isolation, at least).

To summarise:

Bug #1: Auto Config Domain Name doesn't appear to detect the system's assigned DNS Domain Name (for the purposes of configuring valid host headers in firmware 2.81.81.81);
Bug #2: iDRAC doesn't work with valid SSL certificates generated by Let's Encrypt;
Bug #3: Due to insufficient validation, SSL certificates which iDRAC doesn't correctly handle break web-interface and Remote RACADM access;
Bug #4: Only PKCS#12 certificates seem to be able to be cleared via RACADM, PEM certificate-pairs cause the `sslcertdelete` to report that no certificates are present;
Bug #5: Disabling HTTPS redirect appears to have no current function, as it's not possible to pass the login page.

DELL-Shine K · Answer

Sorry to hear that you faced multiple issues with iDRAC. Are these issues seen after updating to 2.81.81.81 or seen with 2.75.75.75 firmware also.

I also captured more details to each of your issue. Please check and revert

Bug #1: Auto Config Domain Name doesn't appear to detect the system's assigned DNS Domain Name (for the purposes of configuring valid host headers in firmware 2.81.81.81);

Can you give more details on this failure? Are you not getting "DNS Domain Name" from DHCP server on iDRAC after enabling "Auto Config Domain Name"

Bug #2: iDRAC doesn't work with valid SSL certificates generated by Let's Encrypt;

How are you creating SSL certificate from Let's Encrypt. Are you creating CSR from iDRAC and use it to create SSL certificate from Let's Encrypt or you are creating keypair and SSL certificate from Let's Encrypt

Bug #3: Due to insufficient validation, SSL certificates which iDRAC doesn't correctly handle break web-interface and Remote RACADM access;
Bug #4: Only PKCS#12 certificates seem to be able to be cleared via RACADM, PEM certificate-pairs cause the `sslcertdelete` to report that no certificates are present;

You can SSH to iDRAC and run "racadm sslresetcfg" command to reset iDRAC SSL certificate to default

Bug #5: Disabling HTTPS redirect appears to have no current function, as it's not possible to pass the login page.

Once httpredirection is disabled, are you launching iDRAC using http or https, Can you try this after running sslresetcfg command

stuart.shelton · Answer

> Are these issues seen after updating to 2.81.81.81 or seen with 2.75.75.75 firmware also

I've only tried using a custom SSL certificate with 2.81.81.81, and haven't rolled-back to confirm whether this is a regression or a longer-term issue.

> Can you give more details on this failure? Are you not getting "DNS Domain Name" from DHCP server on iDRAC after enabling "Auto Config Domain Name"

With this option enabled, the iDRAC did not pick-up a hostname from DNS (by reserve-lookup of its IP)/DHCP, but kept the default of 'idrac- '. It also didn't appear to find a domain name, as the UI field for this was not populated, and with 2.81.81.81 the Host header rules didn't allow access on 'idrac- .mydomain.tld' as it does when the auto-detection is disabled and the domain is manually specified.

> How are you creating SSL certificate from Let's Encrypt. Are you creating CSR from iDRAC and use it to create SSL certificate from Let's Encrypt or you are creating keypair and SSL certificate from Let's Encrypt

This wasn't using the iDRAC's CSR - the certificate for `idrac.dellr330.mydomain.tld` was requested via a Let's Encrypt ACME client, and private key, certificate, certificate chain, and full certificate chain PEM files were generated as a result. I then tried converting both the leaf certificate and the full chain to PKCS#12 format for upload to the iDRAC web-interface: in both cases, the (password-encrypted) certificates were accepted without warning, but SSL was non-functional after an iDRAC restart. I also tried these certificates via RACADM by uploading the certificate and private key individually, with the same result.

> You can SSH to iDRAC and run "racadm sslresetcfg" command to reset iDRAC SSL certificate to default

Ah, excellent, this is good to know! However, a certificate deletion command which can only remove certificates uploaded by one of the two supported methods can't be the intention, surely?

> Once httpredirection is disabled, are you launching iDRAC using http or https, Can you try this after running sslresetcfg command

After disabling HTTP redirection (but I think still with SSL not working because of the certificate not being correctly handled, I think) I was trying to access the web-interface by HTTP, and whilst I got to the login page I wasn't able to proceed past this point. Possibly this was related to SSL being non-functional at this point… but then having an option to bypass SSL (for testing/emergency access) which relies upon SSL being operational feels like a somewhat circular dependency?

(My intent in enabling this was purely to access the UI to clear certificates or upload a new one, rather than anything I'd keep enabled permanently - but with the apparent fragility of the SSL subsystem it feels even more important that this feature be decoupled as a method to restore access in an emergency)

DELL-Shine K · Answer

'Auto Config Domain Name' option can only configure 'DNS Domain Name' on iDRAC provided DHCP server have the option 15 configured. Enabling 'Auto Config Domain Name' will not populate hostname from DHCP to iDRAC DNS name. iDRAC DNS name need to be manually configured.  If you are using a different name to launch iDRAC than one configured in iDRAC then you can add the hostname/FQDN used for launching as an exception by using below racadm command to make it work To add hostname/FQDN as an exception racadm set idrac.webserver.ManualDNSEntry test.domain.com You can also disable host header check on iDRAC by running below command. This command will disable security fix of host header check (Link) racadm set idrac.webserver.HostHeaderCheck Disabled   PKCS#12 format in iDRAC 8 is supported to upload a signing certificate. It is not supported to upload a SSL certificate to private key. If you need to upload a private key and a chain certificate to iDRAC you need to first upload the private key using 'racadm sslkeyupload' command then upload chain certificate in base64 format using 'racadm sslcertupload' command using -t 1 option.

stuart.shelton · Answer

> If you need to upload a private key and a chain certificate to iDRAC you need to first upload the private key using "racadm sslkeyupload" command then upload chain certificate in base64 format using "racadm sslcertupload" command using -t 1 option.

This is what I've been trying to achieve: there appears to be no means to do this via the Web UI (which allows the upload of signed CSRs and PKCS#12-format files), and this upload via RACADM succeeds without error.

However, on iDRAC restart, the HTTPS Web UI is not accessible and the protocol-level output is invalid, as above in the original message.

What I'd like to understand now is whether there's something fundamentally incompatible between the (entirely valid, but adherent to the latest standards) Let's Encrypt certificates and the Dell iDRAC web-server - and, if not, if there's a way to get this working correctly.

DELL-Shine K · Answer

Can you share the steps you followed to upload a SSL certificate to iDRAC in detail with commands used?

stuart.shelton · Answer

Literally as simple as:/opt/dell/srvadmin/sbin/racadm sslkeyupload -t 1 -f /etc/ssl/letsencrypt/certs/idrac.dell330.domain.tld/privkey.pem /opt/dell/srvadmin/sbin/racadm sslcertupload -t 1 -f /etc/ssl/letsencrypt/certs/idrac.dell330.domain.tld/chain.pem /opt/dell/srvadmin/sbin/racadm racreset … which I've tried with the 'cert.pem', 'chain.pem', and 'fullchain.pem' files, all of which result in iDRAC SSL connections being broken on restart. I've also tried both wildcard and host-specific certificates.The certificates validate correctly and indeed (in case of the wildcard variants) are used elsewhere with 100% success, so it very much appears to be the issue that the iDRAC web-server can't handle the certificate format (perhaps the default key length is too long?) rather than any issue with the certificate itself.And, of course, if the iDRAC system can't handle the certificate then it should gracefully reject it, not silently accept it and then be broken on reboot.

DELL-Shine K · Answer

Can you share how are you creating certificate chain "chain.pem". Below post have details on creating certificate chain which is supported in iDRAC. Can you try that?

https://www.dell.com/community/Systems-Management-General/Uploading-a-private-CA-root-intermediate-certificate-to-iDRAC/m-p/7425309#M28299

pmarini · Answer

Exactly the same issue here with IDRAC 2.84.84.84. Is there a way to use Let's Encrypt certificates with IDRAC?

pmarini · Answer

It seems that a full reset is not needed. For that you just need to connect via ssh to your idrac system and run:/admin1- racadm racadmsslresetcfg racadmracreset

thegood85 · Answer

Just posting because I run in the same issue (or maybe I should say list of issues). I found out that the default certificate from certbot is created using a 256-bit elliptical curve key.

You can check by running :

openssl x509 -in fullchain.pem -text -noout

Since the dell guide mentions 2048bit rsa, I added the following to the certbot command line:

--rsa-key-size 2048 --key-type rsa

This seems to fix the problem. If I upload the rsa-based privkey.pem and fullcert.pem using racadm and restart the idrac, I can finally open the https page and login in idrac with my new certificate

pmarini · Answer

@thegood85​, thanks for sharing. I can confirm that it fixed the issue.

Systems Management General

Multiple problems with SSL Certificates with iDRAC firmware 2.81.81.81

Was this post helpful?