Dell Networking OS10 Certificate Expiration and Solution

Specific versions of OS10 (see Affected Software Releases) contain a default certificate that is used for VLT peer establishment and SFS cluster formation. This default certificate expires on July 27, 2021. Post expiry, traffic reachability issues occur when one of these switches has a subsequent switch reboot, link flap, operator-triggered configuration change, vMotion, and other network events. Certificate expiration does not have any corresponding messages in syslog, traps, or on the user interface.

Affected Software Releases, by model:
 

OS10 Version	Model
10.3.2ER3P1	S5148F only
10.4.0E.R3SP2 through 10.4.0E.R4SP2	MX9116 and MX5108 only
10.4.1.4 through 10.4.3.6P5	All OS10 Models (Incl. MX9116, MX5108 and S5148F)
10.5.0.0 through 10.5.0.7P3	All OS10 Models (Incl. MX9116 and MX5108)

Solution

NOTE: See the attached Tech Sheet for a PDF version of this resolution that you can provide to the engineer who will perform the upgrade.

NOTE: Isilon Backend Ethernet users should not follow the procedures that are outlined in this article. For Isilon, users see KB article 185548: Dell switch Certificate expiration on Dell Z9100-ON switches used for Isilon Backend Ethernet for details about the issue. The article requires a customer login.

NOTE:  ECS Gen3 Appliance users should not follow the procedures that are outlined in this article. For ECS, users see KB article 185691: DTA 185691: ECS: Dell OS10 switch X.509v3 security certificate expiration on ECS Front-End and Back-End Switches may lead to data unavailability for details about the issue. The article requires a customer login.

NOTE: Users of affected versions from 10.3.2.x through 10.4.2.x, certificate update is not available. Users must upgrade to 10.4.3.0 or later (10.5.0.x or later for MX) and must follow the table below.   

NOTE:  For PowerEdge MX users on 10.4.0E(R3SP2)- 10.4.0E(R4SP2), If upgrading to the latest baseline is not feasible by July 27, 2021, Contact Dell Technical Support for assistance with updating the default certificate.

NOTE: The following OS10 release versions comes with the new default certificate that address this certificate expiry issue:
10.4.3.7
10.5.0.9
10.5.1.9
10.5.2.6

ALERT: For users who choose to upgrade to the latest firmware you MUST follow the upgrade path that is stated in the Preparing for an upgrade section of the Dell EMC SmartFabric OS10 Installation, Upgrade, and Downgrade Guide document on our support site.

For releases 10.4.3.0 through 10.5.0.7P3, users must follow the resolution as per this table to prevent network issues due to the default certificate expiration:       
 

Deployment Category	Recommended Resolution	Alternatively, if an upgrade is possible.
Non-MX switches in non-VLT non-SFS mode	No action required.	No action required.
MX-SFS mode	Update the default certificate to a new default certificate using the Dell provided scripts on all nodes. To make the new default certificate in effect, it is a MUST to: - Reboot SFS primary - Also, in multicluster deployment, reboot one of the VLT peers in every VLT pair.	Upgrade to >=10.5.1.7 before July 27, 2021 Follow the upgrade path found in the  Dell EMC OpenManage Enterprise-Modular Edition for PowerEdge MX7000 Chassis User's Guide MX7000 Solution Baselines - Page 15-17 shows the compatible component firmware baselines. OS10 Firmware Update Matrix - Page 22 details the upgrade path for OS10 switches.
MX-Full-switch mode
Switches in VLT non-SFS mode	Update the default certificate to a new default certificate using the Dell provided scripts on all nodes. To make the new default certificate in effect, it is a MUST to do "shut" and "no shut" on VLT Primary switch's VLTi interface/link.	Upgrade to version >=10.5.1.0, before July 27, 2021
VxRail-SFS-Single Rack	Update the default certificate to a new default certificate using the Dell provided scripts on all nodes. To make the new default certificate in effect, it is a MUST: - Reboot SFS primary - Also, in multicluster deployment, reboot one of the VLT peers in every VLT pair.	Upgrade to version >=10.5.2.2, before July 27, 2021
VxRail-SFS-Multi-Rack
S5148	Upgrade to 10.4.3.x and then update the default certificate to a new default certificate using the Dell provided scripts on all nodes. Update the default certificate to a new default certificate using the Dell provided scripts on all nodes. To make the new default certificate in effect, it is a MUST to do "shut" and "no shut" on VLT Primary switch's VLTi interface/link.	Same as Recommended Resolution.

NOTE: A maintenance window is required for VLTi link flap or switch reboot as these can potentially disrupt network traffic flow. When calculating your maintenance window:       

• For script, allow 3 to 5 minutes per device. While the script is running, traffic is not impacted.
• For upgrading OS10, estimate 30 minutes per node when going from one release to the next.

“Dell Networking Default X.509 Certificate Update” script package is available on Dell Digital Locker with your OS10 entitlement. The Script package filename is "cert_upgrade_script," and it includes a README file that has detailed instructions on how to run the scripts. Click a switch in your DDL account, then go to available downloads to see the script package.

For more details on logging in and downloading the script see Dell Networking How to Download OS10 Cert_Upgrade_Script from Dell Digital Locker

Further Step by Step details on how to complete the certificate update are available in these articles:

Choose one

Dell Networking OS10 How to Run Certificate Update from Linux

Dell Networking OS10 How to Run Certificate Update Directly from the OS10 Switch

Dell Networking OS10 How to Run Certificate Update from Windows with Python

Here is a video showing the process to update the certificate using a python script.

NOTE: Depending on where you have your script file saved you may have a different file path then what is used in this video.





CAUTION: All switches in a cluster or VLT must have the same certificate that is installed, for cluster or VLT communication. It is mandatory to run the script on all the nodes in the cluster and VLT during the same maintenance window.

After updating the default certificate on the switches, note the following:      

• If you downgrade another software version that has the old default certificate, you may experience the issue again.
• If you boot on another partition that still has an old default certificate, you may experience the issue again.
• If you replace a switch with a version using the old default certificate, you may experience the issue again.

Should any of these scenarios occur, you should:     

•   Use the script to update the default certificate again.

ALERT: The certificate is not used when all systems in a cluster are using 10.5.1.0 or later. If the cluster is running with mixed versions of OS10 with some nodes running 10.5.0.x and below. Then systems running 10.5.1.x or above must run the script to install the new certificate for the nodes to form a cluster.

Some new switches shipping with 10.4.3 or 10.5.0 already have a new default certificate installed. In addition, some service replacement switches have a new default certificate installed. These units are identified by a sticker on the unit that reads “Cert Updated.”

To use such a switch in VLT or SFS cluster  

• All switches in the cluster must have the new default certificate installed.