Hi thanks for choosing Dell, if your system is still in warranty you could call in for support if out for warranty,  you could also check the log, you can refer to this material:
https://dell.to/3eoUNz2
Should you have any further questions, please let us know. Have a good one!

Can you login to iDRAC, Server-> Power/Thermal page and see current power reading and threshold.

You can also login to CMC Power page and check power monitoring page to check total power available and consumed.

How many power supply you have on the chassis and what Redundancy Policy is configured for PSU on CMC.

Can you also check whether you have latest CMC FW installed on the chassis?

I checked the power cap and it is indeed not enabled on anything.

The BIOS settings for the nodes should be identical (as I've set them up over a year ago and didn't touch it since).

The power thresholds and budgets are slightly different between nodes but they have different NICs and things so it is sort of to be expected. All the budgets and thresholds are over 300W though, and the throttled nodes don't reach that.

At some point I'll cold reboot the servers and/or chassis. I expect it'll be fixed for a little while until it happens again. It would be nice to have an understanding as to the cause/permanent solution.

In the iDRAC, the current power reading and threshold is 250W to 300W (current)/350W-400W (warning threshold) for the nodes that are not throttling their CPU frequencies and a reading of 100 to 150W (current)/350W-400W (warning threshold) for the nodes that are throttling their CPU frequency.

System potential power is 7338W and the peak system power is 3948W. I used to run grid redundancy before notice this problem. I switched to no redundancy since. It didn't make a difference. I have 6x2700W PSUs.

I have CMC FW 6.21.200.201909170048 which I believe is the latest.

Hello,

i confirm the CMC is up to date, and which are the version of iDRAC and BIOS of the M620s?

Thanks

Marco

All BIOS on 2.9.0 and iDRAC versions are 2.65.65.65 (15)

Hello,

Are they throttling at present? If they are, I'd be happy to take a look at the chassis logs to see if there's anything of note there. Also, what linux distro are you running on the host? It may not be related, but it's something worth looking into. If sharing the chassis logs is something you'd be open to, you can get them through an SSH session to the chassis and running the commands 'racdump' and 'dumplogs.' They won't save to a file, but rather, output to the screen. You'll need to make sure to save the outputs. If you use something like PuTTY, you should have an option to save the shell output to a file. 

Once you have that log file, you can either upload it someplace for me to download, or I can send an email to you at whatever email you like for you to respond to with the log file attached. With the firmware being confirmed as up to date, I think it may make the most sense to give hardware logs a look, which should tell us if the OS even needs to be considered as an issue.

Our CRM software has an email function built into we can use for file transfers. If it works, we can email you using the email you used to register with the forum. If you'd rather we didn't use that email, another can be used in its place.

OK that sounds great. I'll get the logs sometime tomorrow.

Do you have an email I can send the logs to.