Multiple HDD failures after BIOS firmware upgrade - PowerEdge 730xd

Question

Hello,

Bit of a strange issue here;

We have two identical servers running Windows Server 2016 Datacenter Edition and we decided to re-purpose these servers and thought we could start by completing all the firmware and driver upgrades from Dell.

After checking, we noticed that the servers only needed the BIOS firmware upgraded from 2.15.0 to 2.16.0. We have downloaded and executed BIOS_7557V_WN64_2.16.0_02.exe file within windows. After the BIOS installation was completed we rebooted the servers. While connected to the iDRAC we could see the upgrade was successful and it rebooted back in to the windows again.

This is when we started seeing HDD failures.

First server had two HDD failures and the second one had four failures which I thought it was very odd.

Troubleshooting steps I have tried:

Drained the power from the server by removing both power cables and pressing the power button for 20 seconds.
Rolled back firmware back to 2.15.0.
Then I realized there was a newer bios firmware available, 2.17.0. Upgraded to this.

None of the steps above helped.

I have noticed the failed HDD models are all very similar (PX02SSF020 or PX02SSF020 with A4B3 firmware).

Is it possible that these have failed all at the same time? Is there anything else for me to try?

Latif · Accepted Answer

We will treat this as HDD failure and replace the disks. Just glad that we did not do any firmware upgrades while the server was in production.

DELL-Charles R · Answer

Hello Latif,

That is strange. I have not seen a BIOS update make hard drives fail.

Having that many drives failing at the same time is also strange. Not saying that couldn't happen but it is strange.

Was there any data loss? If there was important data and you do not have a backup, you may consider a Data Recovery company to see if it can be retrieved.

You stated all the failed ones were similar. Did you have any similar drives that did not fail?

Have you reseated the drives or try them in different slots? If the failure follows the drive then I would suspect it is failed.

Are these Dell Branded hard drives? Can you post picture of label showing DP/N number?

Could you post which PERC controller you have and list the firmware version?

Can you also post picture of where you see the drives shown as failed? Is that in the controller or the iDRAC?

Have you tried running the embedded diagnostics?

Boot to F11 on Dell Splash screen, selecting Boot Manager -> System Utilities -> Launch Dell Diagnostics. Note down any messages and continue testing.

Latif · Answer

Hi Charles,

Thank you for your response.

Lucky for us, we were in the process of re-purposing these servers so there was no data loss (even though we have lost one of the raid5 disks).

Servers are at a remote location and I will have to wait for someone to come in to re-seat the drives. I have however managed to run the embedded diagnostics which shows the drives as failed.

Below are some screenshots I captured which shows the failed HDD and the PERC controller information.

Let me know if you require any further information

Thanks.

DELL-Marco B · Answer

Yes this is a strange behaviour, what you can do as suggested is to try different slot to exclude some issue related to the backplane.

Unfortunately is hard to trobleshoot this issue as it is very odd to have 3 drives failed on the same thing.

Thanks

PowerEdge Hardware General

Multiple HDD failures after BIOS firmware upgrade - PowerEdge 730xd

Was this post helpful?