Dell iDRAC Telemetry Reference Guide
GPU Metrics
GPU metrics offer comprehensive insights into the performance, health, and operational status of GPU components in server environments. These metrics include temperature readings, power consumption, memory usage, clock frequencies, PCIe throughput, NVLink activity, and various alert states. Each metric is captured at regular intervals and categorized as either numeric, discrete, or counter types, enabling precise monitoring and diagnostics for optimizing GPU reliability and efficiency.
| Metric Ids | Description | Metric Type | Units | Sensing Interval in seconds |
|---|---|---|---|---|
| GPUStatus | GPU Availability Status | Discrete (Available, Unavailable, Not Applicable) | Not Applicable | 5 |
| MemoryTemperature | Temperature (degrees C) on GPU memory, if supported | Numeric | Celsius (Cel) | 5 |
| GPUHealth | GPU health status | Discrete (Unknown, OK, Degraded, Critical) | Not Applicable | 5 |
| PowerSupplyStatus | Specifies Power supply status | Discrete (Disabled, Enabled, Not Applicable) | Not Applicable | 5 |
| BoardPowerSupplyStatus | GPU board power supply status | Discrete (UnderPowered, SufficientPower, Not Applicable) | Not Applicable | 5 |
| BoardTemperature | Temperature (degrees C) on GPU board, if supported | Numeric | Celsius (Cel) | 5 |
| PowerBrakeState | Power brake state | Discrete (Released, Set, Not Applicable) | Not Applicable | 5 |
| PowerConsumption | Total GPU board power consumption in mWatts (100 mW resolution) | Numeric | milliwatts (mW) | 5 |
| ThermalAlertState | Thermal alert state | Discrete (NotPending, Pending, Not Applicable) | Not Applicable | 5 |
| PrimaryTemperature | Temperature (degrees C) on primary GPU | Numeric | Celsius (Cel) | 5 |
| SecondaryTemperature | Temperature (degrees C) on secondary GPU in dual GPU boards | Numeric | Celsius (Cel) | 5 |
| GPUMemoryClockFrequency | GPU Memory Clock Frequency | Numeric | Mega-Hertz (MHz) | 5 |
| GPUClockFrequency | GPU Clock Frequency | Numeric | Mega-Hertz (MHz) | 5 |
| GPUMemoryUsage | GPU Memory Usage Percent | Numeric | Percentage (%) | 5 |
| GPUUsage | GPU Usage Percent | Numeric | Percentage (%) | 5 |
| GPUEnforcedPowerLimit | GPU Configured Power Cap Limit (measured in mW) | Numeric | millIWatts (mW) | 5 |
| GPUArbitratedPowerLimit | GPU Current Power Cap Limit (measured in mW) | Numeric | milliWatts (mW) | 5 |
| GPUSMOccupancy | The ratio of number of warps residents on an SM to the theoretical maximum number of warps per elapsed cycle as a percentage | Numeric | Percentage (%) | 5 |
| GPUPCIeLinkSpeedMax | Max PCIe Link Speed. 0 = Unknown, 1 = 2500 MTPS (PCIe Gen1.0), 2 = 5000 MTPS (PCIe Gen2.0), 3 = 8000 MTPS (PCIe Gen3.0), 4 = 16000 MTPS (PCIe Gen4.0), 5 = 32000 MTPS (PCIe Gen5.0), ... | Numeric | Not Applicable | 5 |
| GPUHmmaUsage | The ratio of cycles the tensor (HMMA) pipe is active (off the peak sustained elapsed cycles) as a percentage | Numeric | Percentage (%) | 5 |
| GPUMemBandwidthUsage | The memory Bandwidth utilization. DRAM usage as a percentage (relative to maximum theoretical bandwidth). | Numeric | Percentage (%) | 5 |
| GPUSMActivity | The ratio of cycles an SM has at least one warp assigned (computed from the number of cycles and elapsed cycles). | Numeric | Percentage (%) | 5 |
| GPUTotalSMUsageTime | Accumulated SM utilization time in millisecond. Clears on SBR or fundamental reset; no rollover protection | Counter | milliseconds (mS) | 5 |
| GPUTensorCoreUsage | The ratio of cycles any tensor pipe is active as a percentage. | Numeric | Percentage (%) | 5 |
| GPUPCIeTxThroughput | PCIe TX throughput. | Numeric | Gbps | 5 |
| GPUPCIeRxThroughput | PCIe RX throughput. | Numeric | Gbps | 5 |
| GPUClockEventReason | List of factors that are reducing clock frequency. Each bit represents one factor. The current defined bits (in order of bits from right to left) are - 1-ClockOptimizedForPower, 2-HWSlowdown, 3-HWThermalSlowdown, 4-HWPowerBrakeSlowdown, 5-SyncBoost, and 6-ClockOptimizedForThermal. Other bits might be defined in the future | Numeric | Not Applicable | 5 |
| GPUPCIeLinkSpeed | Current PCIe Link Speed. 0 = Unknown, 1 = 2500 MTPS (PCIe Gen1.0), 2 = 5000 MTPS (PCIe Gen2.0), 3 = 8000 MTPS (PCIe Gen3.0), 4 = 16000 MTPS (PCIe Gen4.0), 5 = 32000 MTPS (PCIe Gen5.0), ... | Numeric | Not Applicable | 5 |
| GPUPCIeCorrectableErrorCount | Correctable error count of the following errors - Receiver error, - Bad TLP, - Bad DLLP, - Replay Rollover, - Replay timeout, - Advisory nonfatal error | Numeric | Not Applicable | 5 |
| GPUResetRecommendedState | A flag that indicates if a GPU reset is recommended. 0 - reset not recommended, 1 - reset recommended | Numeric | Not Applicable | 5 |
| GPUNVLinksCount | Count of NVLinks for the current device. Useful for other GPUNVLink metrics. | Numeric | Not Applicable | 5 |
| GPUNVLinkStatusFlag | Each bit represents a status flag for NVLink (0 - NVLink is down and 1 - NVLink is up). GPUNVLinksCount gives the number of bits to be read. | Numeric | Not Applicable | 5 |
| GPUNVLinkRuntimeError | Each bit represents a runtime error for NVLink (0 - no error occurred and 1 - error occurred). GPUNVLinksCount gives the number of bits to be read. | Numeric | Not Applicable | 5 |
| GPUNVLinkTrainingError | Each bit represents a training error for NVLink (0 - no error occurred and 1 - error occurred). GPUNVLinksCount gives the number of bits to be read. | Numeric | Not Applicable | 5 |
| GPUNVLinkRxThroughput | NVLink Rx data throughput (measured in Gbps) | Numeric | Gbps | 1 |
| GPUNVLinkTxThroughput | NVLink Tx data throughput (measured in Gbps) | Numeric | Gbps | 1 |
| GPUSWViolationDuration | Total duration of throttling (in nanoseconds) caused by global software violation of the processor since reset. | Numeric | nanoseconds (nSec) | 1 |
| GPUPowerViolationDuration | Total duration of throttling (in nanoseconds) caused by a power limit of the processor since a reset. | Numeric | nanoseconds (nSec) | 1 |
| GPUThermalViolationDuration | Total duration of throttling (in nanoseconds) caused by a thermal limit of the processor since reset. | Numeric | nanoseconds (nSec) | 1 |
| GPUDMMAUsage | The ratio of cycles the tensor (DFMA) pipe is active (off the peak sustained elapsed cycles) | Numeric | Percentage (%) | 5 |
Please provide ratings (1-5 stars).
Please provide ratings (1-5 stars).
Please provide ratings (1-5 stars).
Please select whether the article was helpful or not.
Comments cannot contain these special characters: <>()\