ECS: RAP015: Temperature Failure; Symptom Code: 2010
Summary: A temperature sensor on the node is reporting at a critical level has been reached.
This article applies to
This article does not apply to
This article is not tied to any specific product.
Not all product versions are identified in this article.
Symptoms
A temperature sensor has detected a temperature that is above a critical threshold.
A component may not be operating correctly that is causing a temperature sensor to report that a critical level has been reached.
A temperature sensor on the node is reporting that a critical level has been reached.
A component may not be operating correctly that is causing a temperature sensor to report that a critical level has been reached.
A temperature sensor on the node is reporting that a critical level has been reached.
Cause
An issue has occurred that caused a temperature sensor to go above a critical level.
Resolution
For Gen2, scroll to the bottom.
Gen3 Hardware:
1. Check the status of the temperature sensors using cs_hal on the reported node.
Command:
Gen3 Hardware:
1. Check the status of the temperature sensors using cs_hal on the reported node.
Command:
#cs_hal sensors temp
Example: For Gen3, there are only three Temp sensors as follows.
admin@n1-mgmt:~> cs_hal sensors temp Entity Type Label Status Info ----- ----- ----- ----- ----- Processor Temperature Temp OK 53 Degrees Celsius Processor Temperature Temp OK 54 Degrees Celsius System Board Temperature Inlet Temp CRIT 40 Degrees Celsius; above critical threshold System Board Temperature Exhaust Temp OK 50 Degrees Celsius NOTE: on Axum and EX-series, use "sudo -i racadm getsensorinfo" to obtain sensor information. admin@n1-mgmt:~>
2.Check all nodes in the rack and see if other nodes are reporting temp sensor not "OK"
Command:
Example: In this example, multiple nodes in the upper half of the rack are reporting inlet temp too high.
3. Possible Scenarios:
4. Important : Use https://central.dell.com/case-lookup/ and lookup the PSNT (Product Serial Number Tag) to check the history. Check how many occurrences happened over the last 3-6 months. Check if the issue was persistent and affecting several nodes or if a whole rack is affected with inlet temp being higher that normal, then this indicates a persisting environmental problems that to beesolve. Do not close the case as duplicate unless there is a clear action plan and conclusions to resolve the temperature issue.
5. If the PE team does not find an issue or if the history contains many occurrences from the same alert (for 3 months or more) then consult with an L2 over Swarm and prepare to work order a CE to review the environmental conditions of the rack and nodes affected.
Command:
viprexec -i cs_hal sensors temp
Example: In this example, multiple nodes in the upper half of the rack are reporting inlet temp too high.
admin@n1-mgmt:~> viprexec -i cs_hal sensors temp Output from host : 192.168.219.1 Entity Type Label Status Info ----- ----- ----- ----- ----- Processor Temperature Temp OK 53 Degrees Celsius Processor Temperature Temp OK 53 Degrees Celsius System Board Temperature Inlet Temp CRIT 40 Degrees Celsius; above critical threshold System Board Temperature Exhaust Temp OK 50 Degrees Celsius NOTE: on Axum and EX-series, use "sudo -i racadm getsensorinfo" to obtain sensor information. Output from host : 192.168.219.2 Entity Type Label Status Info ----- ----- ----- ----- ----- Processor Temperature Temp OK 47 Degrees Celsius Processor Temperature Temp OK 49 Degrees Celsius System Board Temperature Inlet Temp CRIT 39 Degrees Celsius; above critical threshold System Board Temperature Exhaust Temp OK 50 Degrees Celsius NOTE: on Axum and EX-series, use "sudo -i racadm getsensorinfo" to obtain sensor information. Output from host : 192.168.219.3 Entity Type Label Status Info ----- ----- ----- ----- ----- Processor Temperature Temp OK 46 Degrees Celsius Processor Temperature Temp OK 46 Degrees Celsius System Board Temperature Inlet Temp OK 35 Degrees Celsius System Board Temperature Exhaust Temp OK 47 Degrees Celsius NOTE: on Axum and EX-series, use "sudo -i racadm getsensorinfo" to obtain sensor information. Output from host : 192.168.219.4 Entity Type Label Status Info ----- ----- ----- ----- ----- Processor Temperature Temp OK 48 Degrees Celsius Processor Temperature Temp OK 50 Degrees Celsius System Board Temperature Inlet Temp OK 35 Degrees Celsius System Board Temperature Exhaust Temp OK 47 Degrees Celsius NOTE: on Axum and EX-series, use "sudo -i racadm getsensorinfo" to obtain sensor information. Output from host : 192.168.219.5 Entity Type Label Status Info ----- ----- ----- ----- ----- Processor Temperature Temp OK 48 Degrees Celsius Processor Temperature Temp OK 50 Degrees Celsius System Board Temperature Inlet Temp WARN 38 Degrees Celsius; above non-critical threshold System Board Temperature Exhaust Temp OK 49 Degrees Celsius NOTE: on Axum and EX-series, use "sudo -i racadm getsensorinfo" to obtain sensor information. Output from host : 192.168.219.6 Entity Type Label Status Info ----- ----- ----- ----- ----- Processor Temperature Temp OK 50 Degrees Celsius Processor Temperature Temp OK 52 Degrees Celsius System Board Temperature Inlet Temp CRIT 39 Degrees Celsius; above critical threshold System Board Temperature Exhaust Temp OK 51 Degrees Celsius NOTE: on Axum and EX-series, use "sudo -i racadm getsensorinfo" to obtain sensor information. Output from host : 192.168.219.7 Entity Type Label Status Info ----- ----- ----- ----- ----- Processor Temperature Temp OK 45 Degrees Celsius Processor Temperature Temp OK 48 Degrees Celsius System Board Temperature Inlet Temp OK 36 Degrees Celsius System Board Temperature Exhaust Temp OK 47 Degrees Celsius NOTE: on Axum and EX-series, use "sudo -i racadm getsensorinfo" to obtain sensor information. Output from host : 192.168.219.8 Entity Type Label Status Info ----- ----- ----- ----- ----- Processor Temperature Temp OK 51 Degrees Celsius Processor Temperature Temp OK 49 Degrees Celsius System Board Temperature Inlet Temp OK 31 Degrees Celsius System Board Temperature Exhaust Temp OK 43 Degrees Celsius NOTE: on Axum and EX-series, use "sudo -i racadm getsensorinfo" to obtain sensor information. Output from host : 192.168.219.9 Entity Type Label Status Info ----- ----- ----- ----- ----- Processor Temperature Temp OK 52 Degrees Celsius Processor Temperature Temp OK 51 Degrees Celsius System Board Temperature Inlet Temp OK 30 Degrees Celsius System Board Temperature Exhaust Temp OK 42 Degrees Celsius NOTE: on Axum and EX-series, use "sudo -i racadm getsensorinfo" to obtain sensor information. Output from host : 192.168.219.10 Entity Type Label Status Info ----- ----- ----- ----- ----- Processor Temperature Temp OK 54 Degrees Celsius Processor Temperature Temp OK 51 Degrees Celsius System Board Temperature Inlet Temp OK 28 Degrees Celsius System Board Temperature Exhaust Temp OK 41 Degrees Celsius NOTE: on Axum and EX-series, use "sudo -i racadm getsensorinfo" to obtain sensor information. 192.168.219.7 Entity Type Label Status Info ----- ----- ----- ----- ----- Processor Temperature Temp OK 45 Degrees Celsius Processor Temperature Temp OK 48 Degrees Celsius System Board Temperature Inlet Temp OK 36 Degrees Celsius System Board Temperature Exhaust Temp OK 47 Degrees Celsius Output from host : 192.168.219.11 Entity Type Label Status Info ----- ----- ----- ----- ----- Processor Temperature Temp OK 56 Degrees Celsius Processor Temperature Temp OK 55 Degrees Celsius System Board Temperature Inlet Temp OK 27 Degrees Celsius System Board Temperature Exhaust Temp OK 40 Degrees Celsius NOTE: on Axum and EX-series, use "sudo -i racadm getsensorinfo" to obtain sensor information. Output from host : 192.168.219.12 Entity Type Label Status Info ----- ----- ----- ----- ----- Processor Temperature Temp OK 59 Degrees Celsius Processor Temperature Temp OK 59 Degrees Celsius System Board Temperature Inlet Temp OK 26 Degrees Celsius System Board Temperature Exhaust Temp OK 38 Degrees Celsius NOTE: on Axum and EX-series, use "sudo -i racadm getsensorinfo" to obtain sensor information. Output from host : 192.168.219.13 Entity Type Label Status Info ----- ----- ----- ----- ----- Processor Temperature Temp OK 51 Degrees Celsius Processor Temperature Temp OK 49 Degrees Celsius System Board Temperature Inlet Temp OK 26 Degrees Celsius System Board Temperature Exhaust Temp OK 36 Degrees Celsius NOTE: on Axum and EX-series, use "sudo -i racadm getsensorinfo" to obtain sensor information. Output from host : 192.168.219.14 Entity Type Label Status Info ----- ----- ----- ----- ----- Processor Temperature Temp OK 57 Degrees Celsius Processor Temperature Temp OK 60 Degrees Celsius System Board Temperature Inlet Temp OK 26 Degrees Celsius System Board Temperature Exhaust Temp OK 38 Degrees Celsius NOTE: on Axum and EX-series, use "sudo -i racadm getsensorinfo" to obtain sensor information. Output from host : 192.168.219.15 Entity Type Label Status Info ----- ----- ----- ----- ----- Processor Temperature Temp OK 59 Degrees Celsius Processor Temperature Temp OK 59 Degrees Celsius System Board Temperature Inlet Temp OK 26 Degrees Celsius System Board Temperature Exhaust Temp OK 39 Degrees Celsius NOTE: on Axum and EX-series, use "sudo -i racadm getsensorinfo" to obtain sensor information. Output from host : 192.168.219.16 Entity Type Label Status Info ----- ----- ----- ----- ----- Processor Temperature Temp OK 56 Degrees Celsius Processor Temperature Temp OK 56 Degrees Celsius System Board Temperature Inlet Temp OK 26 Degrees Celsius System Board Temperature Exhaust Temp OK 38 Degrees Celsius NOTE: on Axum and EX-series, use "sudo -i racadm getsensorinfo" to obtain sensor information. admin@n1-mgmt:~>
3. Possible Scenarios:
- One node only reporting a sensor or more: If the issue is seen on only one node where temperature is reporting not "OK," then this mostly likely indicates a part issue or the node not having good airflow due to more likely an internal issue than a rack issue.
- Multiple nodes are affected this more an environmental issue within the rack itself or possible the Data center
4. Check fans are running OK. If not then a fan may be need replacement.
Command:
#cs_hal sensors fanExample:
admin@ecs:~>cs_hal sensors fan Output from host : 192.168.219.1 Entity Type Label Status Info ----- ----- ----- ----- ----- System Board Fan Fan1 OK 12600 RPM System Board Fan Fan2 OK 12600 RPM System Board Fan Fan3 OK 16920 RPM System Board Fan Fan4 OK 16800 RPM System Board Fan Fan5 OK 17040 RPM System Board Fan Fan6 OK 16920 RPM System Board Fan Fan Redundancy OK fully redundant; NOTE: on Axum and EX-series, use "sudo -i racadm getsensorinfo" to obtain sensor information.3. If all fans are reporting OK, this means that there is no issues with the fan systems. Engage the Power Edge team to check if a part needs replacement. If any fans are reporting issue then follow ECS: Dial Home: Fan Failure; SymptomCode: 2008
4. Important : Use https://central.dell.com/case-lookup/ and lookup the PSNT (Product Serial Number Tag) to check the history. Check how many occurrences happened over the last 3-6 months. Check if the issue was persistent and affecting several nodes or if a whole rack is affected with inlet temp being higher that normal, then this indicates a persisting environmental problems that to beesolve. Do not close the case as duplicate unless there is a clear action plan and conclusions to resolve the temperature issue.
5. If the PE team does not find an issue or if the history contains many occurrences from the same alert (for 3 months or more) then consult with an L2 over Swarm and prepare to work order a CE to review the environmental conditions of the rack and nodes affected.
Gen 2:
1. Check the status of the temperature sensors using cs_hal.
Example:
# cs_hal sensors temp Entity Type Label Status Info ----- ----- ----- ----- ----- System Board Temperature SSB Therm Trip OK System Board Temperature BB Inlet Temp OK 32 Degrees Celsius CPU (DCMI Compat) Temperature HSBP Temp OK -222 Degrees Celsius System Board Temperature SSB Temp OK 60 Degrees Celsius System Board Temperature BB BMC Temp OK 51 Degrees Celsius System Board Temperature P1 VR Temp OK 38 Degrees Celsius System Board Temperature IB Temp OK 46 Degrees Celsius System Board Temperature Exit Air Temp OK 54 Degrees Celsius Front Panel Temperature IOM Temp OK 43 Degrees Celsius Drive Backplane Temperature HSBP PSOC OK 37 Degrees Celsius Front Panel Temperature LAN NIC Temp OK 67 Degrees Celsius Power Supply Temperature PS1 Temperature OK 34 Degrees Celsius Power Supply Temperature PS2 Temperature OK 34 Degrees Celsius Processor Temperature P1 Therm Margin OK 216 Degrees Celsius Processor Temperature P2 Therm Margin OK 206 Degrees Celsius Processor Temperature P1 Therm Ctrl % OK 0 Unspecified Processor Temperature P2 Therm Ctrl % OK 0 Unspecified Processor Temperature P1 DTS Therm Mgn OK 216 Degrees Celsius Processor Temperature P2 DTS Therm Mgn OK 206 Degrees Celsius Processor Temperature P1 VRD Hot OK Processor Temperature P2 VRD Hot OK System Board Temperature DIMM Thrm Mrgn 1 OK 201 Degrees Celsius System Board Temperature DIMM Thrm Mrgn 2 OK 200 Degrees Celsius System Board Temperature DIMM Thrm Mrgn 3 OK 198 Degrees Celsius System Board Temperature DIMM Thrm Mrgn 4 OK 197 Degrees Celsius System Board Temperature Agg Thrm Mgn 1 OK 233 Degrees Celsius
2. Follow the same steps are Gen 3 (but do not report to PowerEdge), more details will be updated for Gen2 in the future.
Affected Products
ECS ApplianceProducts
ECS ApplianceArticle Properties
Article Number: 000046763
Article Type: Solution
Last Modified: 30 Apr 2024
Version: 6
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.