1 Rookie

 • 

56 Posts

14213

January 10th, 2021 21:00

The system board System Level current is greater than the upper critical threshold.

Hi all,

Yesterday I received warning mail from iDRAC of PowerEdge 610 server about the system board.

This is the message:

Event: The system board System Level current is greater than the upper critical threshold.
Date/Time: Mon Jan 11 2021 06:30:20
Severity: Critical
Model: PowerEdge R610
Service Tag: xxxxx
BIOS version: 6.6.0
Hostname: 
OS Name: 
iDrac version: 2.90

This PowerEdge server is located inside a server room with other PowerEdge Server. Other PowerEdge machine did not send me any alert mail.

I know that this Dell PowerEdge R610 has got the lastest BIOS and iDRAC firmware version.

How can I solve it?

Moderator

 • 

790 Posts

February 3rd, 2021 02:00

Heya Federico,

 

"I got the same issue with PSU1 in Slot 2" -> so, it seems that the issue is not related to the PSU itself, but to the connector on the mainboard. It could be that the power distribution fails.

If it is this, a replacement of the mainboard should be considered, as the power distribution unit is directly soldered to the motherboard. There is no separate part for this. If you need a part number, please let me know the systems ServiceTag, simply send it to me via a private message, as there are several to choose from.

 

You're always welcome!

 

BR
Stefan

Moderator

 • 

790 Posts

January 11th, 2021 02:00

Hey there.

 

This message is related to a power source issue.

 

I see that the iDRAC version is not current, so you might go for that before doing anything else.

 

The latest version is 2.92: https://www.dell.com/support/home/en-us/drivers/driversdetails?driverid=kpccc

 

The next step would be to check the systems event log/server event logs to see if there is any related issue with the PSU(s).

 

If you do not see any issues/messages related to this in the log files kindle export and clear the logs followed by an iDRAC restart.

 

This should remove the message. If it comes back again check the logs again - there have to be any related messages in order to drill down the issue.

 

Let me know your findings.

 

Best regards
Stefan

1 Rookie

 • 

56 Posts

January 11th, 2021 04:00

Dear @DELL-Stefan R 
Thanks so much for your help.

I will try soon as possible to upgrade iDRAC 6 firmware from  2.90 to 2.92 version.

 

I will keep you update!

Take care
Federico

1 Rookie

 • 

56 Posts

January 11th, 2021 06:00

Hi @DELL-Stefan R 

I have received warning again with new iDRAC Firmware (iDRAC6 2.92)

Message is:
"The system board System Level current is greater than the upper critical threshold"

I don't have any faulty part:

FedericoCoppola97_0-1610376874793.png

 

How can I solve it?

Best regards
Federico

 

Moderator

 • 

790 Posts

January 11th, 2021 06:00

Hi Federico,

 

That sounds good - if the issue does not come back right soon, it's solved

 

1 Rookie

 • 

56 Posts

January 11th, 2021 06:00

Hi,
I have upgraded iDRAC firmware from 2.90 version to 2.92 version.

FedericoCoppola97_0-1610375663056.png

After that I removed old logs and I restarted iDRAC.

At the moment I did not received any warning mail.

Thanks for your support
Federico

 

Moderator

 • 

790 Posts

January 11th, 2021 07:00

Welcome back

 

Could you please take a look at the iDRAC 6 manual here:
https://dell.to/3i3k5D5

 

Starting on page 287 we have the power management section, could you check the settings if there is something not normal in there?

 

Maybe the threshold is not set correctly there - or maybe the issue is that it is set, but it is not needed.

 

Please take a look and let me know.

 

BR
Stefan

1 Rookie

 • 

56 Posts

January 11th, 2021 08:00

Dear @DELL-Stefan R ,
Thanks for your help.

I have followed the documents and I collected the following data.

 

 

/admin1-> racadm getconfig -g cfgServerPower
# cfgServerPowerStatus=1
# cfgServerPowerAllocation=945 AC W | 3225 Btu/hr
# cfgServerActualPowerConsumption=64652 AC W | 220657 Btu/hr
# cfgServerMinPowerCapacity=75 AC W | 255 Btu/hr
# cfgServerMaxPowerCapacity=551 AC W | 1882 Btu/hr
# cfgServerPeakPowerConsumption=2184 AC W | 7454 Btu/hr
# cfgServerPeakPowerConsumptionTimestamp=Mon Jan 11 12:07:28 2021
# cfgServerPowerConsumptionClear=******** (Write-Only)
cfgServerPowerCapWatts=551 AC W
cfgServerPowerCapBtuhr=1881 btu/hr
cfgServerPowerCapPercent=100
cfgServerPowerCapEnable=0
# cfgServerPowerLastMinAvg=1296 AC W | 4423 Btu/hr
# cfgServerPowerLastHourAvg=-1371 AC W | -4678 Btu/hr
# cfgServerPowerLastDayAvg=-1563 AC W | -5334 Btu/hr
# cfgServerPowerLastWeekAvg=-1563 AC W | -5334 Btu/hr
# cfgServerPowerLastHourMinPower=1054 AC W | 3597 Btu/hr
# cfgServerPowerLastHourMinTime=Mon Jan 11 16:18:06 2021
# cfgServerPowerLastHourMaxPower=64780 AC W | 221094 Btu/hr
# cfgServerPowerLastHourMaxTime=Mon Jan 11 16:39:38 2021
# cfgServerPowerLastDayMinPower=1051 AC W | 3587 Btu/hr
# cfgServerPowerLastDayMinTime=Mon Jan 11 15:53:50 2021
# cfgServerPowerLastDayMaxPower=64780 AC W | 221094 Btu/hr
# cfgServerPowerLastDayMaxTime=Mon Jan 11 16:39:38 2021
# cfgServerPowerLastWeekMinPower=1051 AC W | 3587 Btu/hr
# cfgServerPowerLastWeekMinTime=Mon Jan 11 15:53:50 2021
# cfgServerPowerLastWeekMaxPower=64780 AC W | 221094 Btu/hr
# cfgServerPowerLastWeekMaxTime=Mon Jan 11 16:39:38 2021
# cfgServerPowerInstHeadroom=0 AC W | 0 Btu/hr
# cfgServerPowerPeakHeadroom=0 AC W | 0 Btu/hr
# cfgServerActualAmperageConsumption=0.3 Amps
# cfgServerPeakAmperage=5.3 Amps
# cfgServerPeakAmperageTimeStamp=Sat Oct 03 16:59:02 2020
# cfgServerCumulativePowerConsumption=713384 AC W | 2434780 Btu/hr
# cfgServerCumulativePowerConsumptionTimeStamp=Mon Jan 11 17:02:50 2021
# cfgServerCumulativePowerClear=******** (Write-Only)
# cfgServerPeakPowerClear=******** (Write-Only)
/admin1-> racadm getconfig -g cfgServerPower -o cfgServerPowerCapWatts
551 AC W

 

 

 

Power Redundancy is Ok and Voltage values are Ok from the Web GUI of iDRAC:

PowerSupply_Redundancy_OK.PNGVoltagesPowerSupply_OK.PNG

 

 

 

 

 

 

 

 

 

 

 

 

 

 

2021-01-11 17_21_18-Window.png

Finally Power Budget is not enable on this iDRAC

FedericoCoppola97_0-1610381901233.png

I did not found any wrong setting

Best regards
Federico

1 Rookie

 • 

56 Posts

January 11th, 2021 11:00

Dear @DELL-Stefan R 

I followed your document.

I have collected some information about iDRAC configuration from WebGUI:

  •  Power Supplies Redundancy Status is OK
  • Individual Power Supply Elements are OK
  • Power Budget is disabled
  • Voltages are OK

From the WebGui I have noted this difference about current but I don't know if it is a problem.
Amperage PSU1 is 0.3A@228V
Amperage PSU2 is 4.6A@230V

There is too much difference beetwen two PSU. I tried to disconnect PSU one at the time and this R610 Server continue to work without issue.

 

Finally I have collected this information from SSH:

/admin1-> racadm getconfig -g cfgServerPower
# cfgServerPowerStatus=1
# cfgServerPowerAllocation=945 AC W | 3225 Btu/hr
# cfgServerActualPowerConsumption=64601 AC W | 220483 Btu/hr
# cfgServerMinPowerCapacity=75 AC W | 255 Btu/hr
# cfgServerMaxPowerCapacity=551 AC W | 1882 Btu/hr
# cfgServerPeakPowerConsumption=2184 AC W | 7454 Btu/hr
# cfgServerPeakPowerConsumptionTimestamp=Mon Jan 11 12:07:28 2021
# cfgServerPowerConsumptionClear=******** (Write-Only)
cfgServerPowerCapWatts=551 AC W
cfgServerPowerCapBtuhr=1881 btu/hr
cfgServerPowerCapPercent=100
cfgServerPowerCapEnable=0
# cfgServerPowerLastMinAvg=1300 AC W | 4437 Btu/hr
# cfgServerPowerLastHourAvg=-1377 AC W | -4699 Btu/hr
# cfgServerPowerLastDayAvg=-1406 AC W | -4798 Btu/hr
# cfgServerPowerLastWeekAvg=-1406 AC W | -4798 Btu/hr
# cfgServerPowerLastHourMinPower=1054 AC W | 3597 Btu/hr
# cfgServerPowerLastHourMinTime=Mon Jan 11 19:29:04 2021
# cfgServerPowerLastHourMaxPower=64762 AC W | 221033 Btu/hr
# cfgServerPowerLastHourMaxTime=Mon Jan 11 19:39:28 2021
# cfgServerPowerLastDayMinPower=1045 AC W | 3567 Btu/hr
# cfgServerPowerLastDayMinTime=Mon Jan 11 17:11:08 2021
# cfgServerPowerLastDayMaxPower=64780 AC W | 221094 Btu/hr
# cfgServerPowerLastDayMaxTime=Mon Jan 11 16:39:38 2021
# cfgServerPowerLastWeekMinPower=1045 AC W | 3567 Btu/hr
# cfgServerPowerLastWeekMinTime=Mon Jan 11 17:11:08 2021
# cfgServerPowerLastWeekMaxPower=64780 AC W | 221094 Btu/hr
# cfgServerPowerLastWeekMaxTime=Mon Jan 11 16:39:38 2021
# cfgServerPowerInstHeadroom=0 AC W | 0 Btu/hr
# cfgServerPowerPeakHeadroom=0 AC W | 0 Btu/hr
# cfgServerActualAmperageConsumption=0.3 Amps
# cfgServerPeakAmperage=5.3 Amps
# cfgServerPeakAmperageTimeStamp=Sat Oct 03 16:59:02 2020
# cfgServerCumulativePowerConsumption=902134 AC W | 3078983 Btu/hr
# cfgServerCumulativePowerConsumptionTimeStamp=Mon Jan 11 19:59:39 2021
# cfgServerCumulativePowerClear=******** (Write-Only)
# cfgServerPeakPowerClear=******** (Write-Only)

Thanks for your help
Federico

7 Practitioner

 • 

9.7K Posts

 • 

48K Points

January 11th, 2021 13:00

Could you confirm if the server is fine and without error when either of the power supplies are installed individually, as well as if that is in both slots?

Also, are the power supplies matching part numbers as well?

 

Let us know.

 

 

1 Rookie

 • 

56 Posts

January 12th, 2021 07:00

Hi @DELL-Chris H ,
I tried to connect just one power supply at time and I see just Error Message about "PSU loss".

>>Also, are the power supplies matching part numbers as well?
I will check parts numbers soon as possibile

 

Best regards
Federico

1 Rookie

 • 

56 Posts

January 13th, 2021 13:00

@DELL-Chris H 

How can I check part number of two Power Supplies?

Thanks
Federico

Moderator

 • 

5.4K Posts

 • 

37 Points

January 13th, 2021 20:00

Hi, do you see something like this?

 

1 Attachment

1 Rookie

 • 

56 Posts

January 14th, 2021 01:00

Hi all,
I have collected Part Number from physical power supply.

I can not check part number from iDRAC, due to I get the following message:

FedericoCoppola97_0-1610615117832.png

I have already restarted iDRAC of this physical server, but I continue to get the same message from "Hardware Inventory" page.

I have the same error inside logs when I connected again second PSU after that I removed it to take part number.

FedericoCoppola97_0-1610615727072.png

This event did not happen when I reconnected PSU1.

Moderator

 • 

790 Posts

January 14th, 2021 02:00

Federico, 

 

I guess the part numbers even, but let me know them.

 

And you reseated both PSUs now (removing and reinstalling them)? Great. That's another troubleshooting step.

 

And you get the message when PSU2 is connected again. So, we can narrow down the issue to PSU2 or System Board/Power Distribution Board (PDB).

 

So, what happens when you power the system off, remove PSU1 and PSU2 and install them in the opposite order. With this, you do a so-called cross-test (x-test). And we are able to see if the issue comes from the PSU or the PDB.

 

If it follows the PSU2, we have a broken PSU and this should be replaced.

 

If the issue stays with slot 2 - we have a broken PDB (this one is on the system board, so in this case, you need to replace the motherboard (this one might be expensive).

 

Let us know your findings.

 

Best regards,
Stefan

No Events found!

Top