Basically, at boot, default power draw is capped at 25W and negotiated upwards to a maximum of 75W for high power x16 devices in presumably high power x16 PCIe slots. This usually means a x16 graphics card.
If the hardware can't or won't provide more than 25W via the PCIe slot, well it is capped at 25W unless the card itself can draw extra power from a PCIe auxilary power connector (6 pin and/or 8 pin).
Do note that a x16 graphics card will draw the full 75W from the slot (after negotiating up from 25W limit at boot) and then draw the remaining power needed from the auxilary power connectors. This can result in 75W from the PCIe slot with another 75W from the 2x3/6 pin connector plus a further 150W from the 4x2/8 pin connector for a total of 300W for the x16 graphics card. IIRC, some cards have two 2x4/8 pin auxilary power connectors meaning the can consume a total of 375W.
That is, it just works if hardware and BIOS (ACPI/DSDT tables within?) are correctly designed by the OEM manufacturer of the system.
So, to sum up, no user action, bios settings or magic fairy dust is needed :emotion-5:
Of course the PCie specs are applicable, otherwise nothing would work...
The card you have inserted, based on the output of your command, has negotiated 8 lanes and 25W from the PCIe slot.
I'm not sure what the PCIe spec says about negotiated power levels for x8 cards (spec not freely available) but either the card asked for more and firmware said 'NO' or card didn't ask for more...
After all it's the card that negotiates the power levels it needs with the firmware at boot time. So if the card isn't getting enough juice from the PCIe slot and needs an auxilary power connector, then plug one in may be enough. Alternatively there may be some card specific setting to tell it to negotiate up to the full 75W limit during boot (but why have such a setting when it should happen automagically) or the card needs to be put into a x16 graphics capable slot that the firmware is aware can feed the full 75W.
I don't know which one of the above it is so may be time for RTM for the card in question (which i don't own).
I think that I am going to leave everything as it. I did insert the card into the x16 "Slot 1", and yet the max power available is stuck at 25W. Go figure. What's odd is that this error message disappears after a while, on a repeat execution of "fio-status" utility.
"Power write governing activated, performance may be limited. If this condition persists, switch to a higher powered PCIe slot or attach external power cable."
The FusionIO manual also states that fio-status does not correctly report the wattage intake. Your card could actually be withdrawing 75W, but the utility fails to show that. When I have something that I can't prove, makes me nervous. I blew up so much money on these servers, and now I can't squeeze out the max performance out of it, which is really sad. Dell does not have a solution for it either :(
ThatServerGuy
3 Posts
0
September 7th, 2017 10:00
So, how do I enable high power draw mode?
I just recently acquired an R820, and installed a 3.2TB SX350-3.2TB card. How do I enable 75W? I installed the card in Slot 1.
What settings are needed to be configured in the BIOS and where/how to access them?
Thanks!
skylarking
2 Intern
•
548 Posts
0
September 11th, 2017 00:00
Firmware knows the capability of the hardware and so during boot process can negotiate power requirements with the PCIe cards themselves. It's part of the PCIe spec.
Basically, at boot, default power draw is capped at 25W and negotiated upwards to a maximum of 75W for high power x16 devices in presumably high power x16 PCIe slots. This usually means a x16 graphics card.
If the hardware can't or won't provide more than 25W via the PCIe slot, well it is capped at 25W unless the card itself can draw extra power from a PCIe auxilary power connector (6 pin and/or 8 pin).
Do note that a x16 graphics card will draw the full 75W from the slot (after negotiating up from 25W limit at boot) and then draw the remaining power needed from the auxilary power connectors. This can result in 75W from the PCIe slot with another 75W from the 2x3/6 pin connector plus a further 150W from the 4x2/8 pin connector for a total of 300W for the x16 graphics card. IIRC, some cards have two 2x4/8 pin auxilary power connectors meaning the can consume a total of 375W.
That is, it just works if hardware and BIOS (ACPI/DSDT tables within?) are correctly designed by the OEM manufacturer of the system.
So, to sum up, no user action, bios settings or magic fairy dust is needed :emotion-5:
ThatServerGuy
3 Posts
0
September 11th, 2017 00:00
I'm not sure if this is applicable, because I tried to run this fio-status -a utility, and here's the output:
I tried manually overriding the iomemory config file to draw 75W, but it doesn't seem like it's drawing 75W.
The "PCIe slot available power" still bugs me.
Adapter: ioMono (driver 4.3.1)
ioMemory SX350-3200, Product Number: , SN: , FIO SN:
ioMemory Adapter Controller, PN:
Product UUID:
External Power Threshold Override: 74.75W
PCIe Bus voltage: avg 12.04V
PCIe Bus current: avg 0.60A
PCIe Bus power: avg 7.19W
PCIe Power limit threshold: 74.75W
PCIe slot available power: 25.00W
PCIe negotiated link: 8 lanes at 5.0 Gt/sec each, 4000.00
MBytes/sec total
Connected ioMemory modules:
fct0: 05:00.0, Product Number:
skylarking
2 Intern
•
548 Posts
0
September 11th, 2017 00:00
Of course the PCie specs are applicable, otherwise nothing would work...
The card you have inserted, based on the output of your command, has negotiated 8 lanes and 25W from the PCIe slot.
I'm not sure what the PCIe spec says about negotiated power levels for x8 cards (spec not freely available) but either the card asked for more and firmware said 'NO' or card didn't ask for more...
After all it's the card that negotiates the power levels it needs with the firmware at boot time. So if the card isn't getting enough juice from the PCIe slot and needs an auxilary power connector, then plug one in may be enough. Alternatively there may be some card specific setting to tell it to negotiate up to the full 75W limit during boot (but why have such a setting when it should happen automagically) or the card needs to be put into a x16 graphics capable slot that the firmware is aware can feed the full 75W.
I don't know which one of the above it is so may be time for RTM for the card in question (which i don't own).
All i know is it's not a BIOS setting.
Cheers.
ThatServerGuy
3 Posts
0
September 11th, 2017 03:00
Thanks for your detailed explanation.
I think that I am going to leave everything as it. I did insert the card into the x16 "Slot 1", and yet the max power available is stuck at 25W. Go figure. What's odd is that this error message disappears after a while, on a repeat execution of "fio-status" utility.
"Power write governing activated, performance may be limited. If this condition persists, switch to a higher powered PCIe slot or attach external power cable."
The FusionIO manual also states that fio-status does not correctly report the wattage intake. Your card could actually be withdrawing 75W, but the utility fails to show that. When I have something that I can't prove, makes me nervous. I blew up so much money on these servers, and now I can't squeeze out the max performance out of it, which is really sad. Dell does not have a solution for it either :(