Start a Conversation

Unsolved

G

35 Posts

132

September 20th, 2023 14:59

Unrecoverable disk media error during the rebuild or recovery operation

Hello,
We have a server with 2 failed disks and 1 disk with a predicted failure. We have already tried to replace the 2 failed disks a few times, each time it ends in 'Failed' after rebuilding up to 60% (approximately).
There is a total of 8 disk slots in the server and there is a 3rd disk with predicted failure.

In /var/log/messages we see the following message about the 3rd disk 0:0:8 :

Sep 19 22:09:16 helium-2 Server_Administrator: 2900 2095 - Storage Service  Unexpected sense. SCSI sense data: Sense key:  6 Sense code:  0 Sense qualifier:  0:  Physical Disk 0:0:8 Controller 0, C
onnector 0
Sep 19 22:09:16 helium-2 Server_Administrator: 2900 2350 - Storage Service  There was an unrecoverable disk media error during the rebuild or recovery operation:  Physical Disk 0:0:8 Controller 0,
Connector 0
Do you think we have any hope ? Can we for exemple unmount the partition and run a fsck on it and retry to replace the dead disks ?
Thank you !

Moderator

 • 

3.1K Posts

September 20th, 2023 19:48

Hello Gilbert,

 

First consider if you have a valid backup. If you don't have one and can't get one, and this is critical data, you may consider contacting a Data Recovery company.

 

When rebuilding into an array that has an on online member that is Pred Fail will increase the chance the array will Puncture. It sounds like the array may be punctured, or may soon puncture, since every rebuild fails and you have Unrecoverable errors.

 

Check what the LifeCycle Controller log shows for any disk errors.

 

Try to complete these steps:

1. Get a backup of the data

2. Update the firmware of the PERC controller, Hard drives, iDRAC and BIOS

3. Run a Consistency Check on the virtual disk(s) via OpenManage Server Administrator (OMSA) or controller BIOS.

4. Run diagnostics: Boot to the Lifecycle Controller <F10> and run the built in diagnostics.

5. Try to see if it will complete a rebuild.

 

Any drive that is Pred Fail or fails diagnostics would need to be replaced.

 

 

Resources you may like to review:

Double Faults and Punctures in RAID Arrays

https://dell.to/3PpfM6m

How to Fix a RAID Puncture

https://dell.to/45Z5xNa

 

35 Posts

September 26th, 2023 15:59

Hey again, 

Thank you for your reply !

So we had to reboot the server and we had a message 'Theare are offline or missing virtual drives with preserved cache'.

And we can only access to the configuration utility.

When we tried to do a consistency Check, we couldn't navigate to it, we could only choose Delete VD and Properties.

Also, 'Virtual Disk: 1' is written with the red color.

Do we have to delete the virtual disk and create a new one ?

Thank you !

Moderator

 • 

3.1K Posts

26-09-2023 05:56 PM

Hello Gilbert,

 

If you see a message about Preserved Cache then you will need to clear it on the VD Mgmt tab and retry import.

 

Have you checked the LifeCycle Controller Log, <F10> at boot, and check for any storage related issues?

 

Check the status of your backup and review the links I posted.
You may need to delete VD, replace failed drives, re create VD and install/restore from backup.

 

DELL-Charles R

Social Media and Communities Professional
Dell Technologies | Enterprise Support Services
#IWork4Dell

Did I answer your query? Please click on ‘Accept as Solution’. ‘Thumbs up’ the posts you like!

1 Rookie

 • 

28 Posts

October 2nd, 2023 08:18

Stop with whatever rebuilding you are doing because you might have more than one drive busy failing and you are putting stress on that drive or drives, and they will fail before your rebuild is successful. Get all the drive out of the unit and test them one by one to see what their SMART report is and go from there. There is a good change you will have to make contact with a drive recovery company.

Where are you located ?

No Events found!

Top