PowerEdge: nvidia-smi shows "PCIe.Link.Gen.Max" is different from "PCIe.Link.Gen.Current"
Summary: Why the nvidia-smi output shows the PCIe link Gen current value different from the Max value.
Symptoms
A situation when the user complains that the nvidia-smi output shows that the GPU PCIe link is not connecting at the maximum available PCIe link and suspects a hardware issue with the board/riser or the GPU itself.
A sample output from NVSMI logs shows a similar situation for reference:

The output shows two similar GPUs connected to a PowerEdge system showing different values of the "PCIe.Link.Gen.Current" parameter. While the GPU in slot 2 it links up at Gen 1, the second GPU in slot 7 links up at the max supported PCIe Gen.
Cause
This behavior is by design. When the GPUs are not in use, they can reduce the link speed to free up resources and operates at lower clocks consuming less power. They can transition to MAX link as and when required.
In some instances when the GPU can support higher PCIe Gen but the system slot or riser is of lower Gen, then the Max link reported is of the corresponding slot or riser of the system.
Resolution
https://enterprise-support.nvidia.com/s/article/Useful-nvidia-smi-Queries-2