ECS: System Detected High Temperature On A Node
Summary: What can I check if I receive an email alert informing me that the system has detected a high temperature sensor reading on a node.
Instructions
-
Confirm what hardware the alerted node is.
admin@node1:~> sudo xdoctor -x Telegraf Version: 3.8.0.2-1549.73c8abc2 Fabric Version: 3.8.0.2-4347.d30cd09 Fabric-Zookeeper Version: 3.8.0.2-120.b4a1c5c Utilities Version: 3.7.0.4-1166.b78f3fe Influxdb Version: 3.8.0.2-1549.73c8abc2 Grafana Version: 3.8.0.2-1549.73c8abc2 Syslog Version: 3.8.0.2-4347.d30cd09 Service Version: 9.0.0.0-22840.479b013c74 Os Version: 3.8.0.2-2113.3fa664c.3 Fluxd Version: 3.8.0.2-1549.73c8abc2 Throttler Version: 3.8.0.2-1549.73c8abc2 Object Image Version: 3.8.0.2-138636.7343cd5c2c3 -------------------- ECS Version: 3.8.0.2 -------------------- HW Gen : 2 HW Model: U-Series HW Code : S2600KP ------------------------- xDoctor Version: 4.8-98.0 -------------------------For Gen 1/2 nodes, respond to the form in the email that assistance is required. For Gen 3 nodes, follow the rest of this kb.
-
Check the current status of the temperature sensors. Below we see "CRIT" on two of the nodes indicating problems on those two nodes. If all nodes report as "OK" but this alert has been received this alert multiple times recently, then it could be a recurring issue. If so, respond to the form in the email that assistance is required with a recurring temperature alert.
admin@node1:~> viprexec -i cs_hal sensors temp Output from host : xxx.xxx.xxx.xxx Entity Type Label Status Info ----- ----- ----- ----- ----- Processor Temperature Temp OK 53 Degrees Celsius Processor Temperature Temp OK 53 Degrees Celsius System Board Temperature Inlet Temp CRIT 40 Degrees Celsius; above critical threshold System Board Temperature Exhaust Temp OK 50 Degrees Celsius NOTE: on Axum and EX-series, use "sudo -i racadm getsensorinfo" to obtain sensor information. Output from host : xxx.xxx.xxx.xxx Entity Type Label Status Info ----- ----- ----- ----- ----- Processor Temperature Temp OK 47 Degrees Celsius Processor Temperature Temp OK 49 Degrees Celsius System Board Temperature Inlet Temp CRIT 39 Degrees Celsius; above critical threshold System Board Temperature Exhaust Temp OK 50 Degrees Celsius NOTE: on Axum and EX-series, use "sudo -i racadm getsensorinfo" to obtain sensor information. Output from host : xxx.xxx.xxx.xxx Entity Type Label Status Info ----- ----- ----- ----- ----- Processor Temperature Temp OK 46 Degrees Celsius Processor Temperature Temp OK 46 Degrees Celsius System Board Temperature Inlet Temp OK 35 Degrees Celsius System Board Temperature Exhaust Temp OK 47 Degrees Celsius NOTE: on Axum and EX-series, use "sudo -i racadm getsensorinfo" to obtain sensor information. ... ... ...
If multiple nodes are reported as not "OK," then it is possibly an issue within the environment of your data center. Check that there are no issues in the area in which the ECS is located that might be increasing the ECS temperature.
-
Check the status of the ECS fans.
admin@ecs:~>cs_hal sensors fan Output from host : xxx.xxx.xxx.xxx Entity Type Label Status Info ----- ----- ----- ----- ----- System Board Fan Fan1 OK 12600 RPM System Board Fan Fan2 OK 12600 RPM System Board Fan Fan3 OK 16920 RPM System Board Fan Fan4 OK 16800 RPM System Board Fan Fan5 OK 17040 RPM System Board Fan Fan6 OK 16920 RPM System Board Fan Fan Redundancy OK fully redundant; NOTE: on Axum and EX-series, use "sudo -i racadm getsensorinfo" to obtain sensor information.
-
Respond to the form in the email that assistance is required, including both the temperature sensor outputs and the fan outputs.