Unsolved
35 Posts
0
367
Unrecoverable disk media error during the rebuild or recovery operation
Hello,
We have a server with 2 failed disks and 1 disk with a predicted failure. We have already tried to replace the 2 failed disks a few times, each time it ends in 'Failed' after rebuilding up to 60% (approximately).
There is a total of 8 disk slots in the server and there is a 3rd disk with predicted failure.
In /var/log/messages we see the following message about the 3rd disk 0:0:8 :
Sep 19 22:09:16 helium-2 Server_Administrator: 2900 2095 - Storage Service Unexpected sense. SCSI sense data: Sense key: 6 Sense code: 0 Sense qualifier: 0: Physical Disk 0:0:8 Controller 0, C
onnector 0
Sep 19 22:09:16 helium-2 Server_Administrator: 2900 2350 - Storage Service There was an unrecoverable disk media error during the rebuild or recovery operation: Physical Disk 0:0:8 Controller 0,
Connector 0
onnector 0
Sep 19 22:09:16 helium-2 Server_Administrator: 2900 2350 - Storage Service There was an unrecoverable disk media error during the rebuild or recovery operation: Physical Disk 0:0:8 Controller 0,
Connector 0
Do you think we have any hope ? Can we for exemple unmount the partition and run a fsck on it and retry to replace the dead disks ?
Thank you !
DELL-Charles R
Moderator
Moderator
•
3.5K Posts
0
September 20th, 2023 19:48
Hello Gilbert,
First consider if you have a valid backup. If you don't have one and can't get one, and this is critical data, you may consider contacting a Data Recovery company.
When rebuilding into an array that has an on online member that is Pred Fail will increase the chance the array will Puncture. It sounds like the array may be punctured, or may soon puncture, since every rebuild fails and you have Unrecoverable errors.
Check what the LifeCycle Controller log shows for any disk errors.
Try to complete these steps:
1. Get a backup of the data
2. Update the firmware of the PERC controller, Hard drives, iDRAC and BIOS
3. Run a Consistency Check on the virtual disk(s) via OpenManage Server Administrator (OMSA) or controller BIOS.
4. Run diagnostics: Boot to the Lifecycle Controller <F10> and run the built in diagnostics.
5. Try to see if it will complete a rebuild.
Any drive that is Pred Fail or fails diagnostics would need to be replaced.
Resources you may like to review:
Double Faults and Punctures in RAID Arrays
https://dell.to/3PpfM6m
How to Fix a RAID Puncture
https://dell.to/45Z5xNa
GSoucy
35 Posts
0
September 26th, 2023 15:59
Hey again,
Thank you for your reply !
So we had to reboot the server and we had a message 'Theare are offline or missing virtual drives with preserved cache'.
And we can only access to the configuration utility.
When we tried to do a consistency Check, we couldn't navigate to it, we could only choose Delete VD and Properties.
Also, 'Virtual Disk: 1' is written with the red color.
Do we have to delete the virtual disk and create a new one ?
Thank you !
Stellenbosch
1 Rookie
1 Rookie
•
28 Posts
0
October 2nd, 2023 08:18
Stop with whatever rebuilding you are doing because you might have more than one drive busy failing and you are putting stress on that drive or drives, and they will fail before your rebuild is successful. Get all the drive out of the unit and test them one by one to see what their SMART report is and go from there. There is a good change you will have to make contact with a drive recovery company.
Where are you located ?