Liquid Cooled PowerEdge servers with AMD 9005 series CPUs may report THRM0019 event

When Liquid Cooled PowerEdge servers with AMD EPYC 9005 series processors runs workloads targeting limited cores or Core Complex Die (CCD), the system may experience:

THRM0019 The thermal state of the server is Critical. If configured to shut down, the Shutdown operation will be initiated in 5 seconds. during system is in utilize.

Eventually, the system may also shut down and log the following events:

STS001 A critical thermal state is reached resulting in an immediate server shutdown operation. For more information, see the Detailed Description.

RAC0704 Requested system powerdown.

The affected liquid cooled systems affected, but not limited to:

• PowerEdge XE9785L
• PowerEdge XE9685L
• PowerEdge R7725
• PowerEdge R7715
• PowerEdge R6725
• PowerEdge R6715
• PowerEdge M7725

Dell is aware of thermal shutdowns occurring in systems running AMD EPYC 9005 series processors, especially the EPYC 9575F. The cause is a conflict between hardware protection put in place for liquid cooling infrastructure and Tj-Limited performance in Power Deterministic mode seen with certain AMD CPUs running workloads targeting limited cores/CCDs.

Options to Resolve this:

1. Disable system shutdown based on CPU temperature. This does not endanger the CPU but adds risk to system cooling if there is facility water or Coolant Distribution Unit (CDU) related failures. To disable system shutdown, change the default setting from ‘Power off’ to ‘No Action’ for the ‘Thermal Shutdown’ sensor in the iDRAC alert configuration. If selecting this option, CDU alarms MUST be set and monitored and react promptly if there is a CDU anomaly and shut down the system. 
   • Facility water anomaly Examples:
     • Temperature is out of specification.
     • Primary control valve is stuck closed.
   • Technology Cooling Systems (TCS) Examples:
     • TCS supply temperature is out of spec (Dell design point varies between 40-45C max depending on server)
     • TCS return temperature is more than CDU limits

2. Change BIOS System Profile from Max Performance to Custom (CPU Power management must be Maximum Performance) and set CPU Determinism mode from Power to Performance.

3. Lower coolant temperature to bring CPU temperature below 87C while running workloads. The exact combination of flowrate and temperature to achieve this may vary by workload, but the coolant in the range of 24C is expected. Recommend to lower the coolant inlet temperature to 24-25C at 22 PSI/1.5 LPM per CPU loop.