All 3 drives failed RAID 5 - Perc S100

From the screenshot looks like you have one drive in offline state and one drive is failed. As RAID 5 can take only one drive failure and your system have 2 drive failed virtual disk is in failed state. You can not access the data when virtual disk is failed. When virtual disk failed adding hot spare will not help as it can not recreate the data from failed virtual disk.

If you can make one of the failed drive working then virtual disk will become degraded and you will be able to recover the data or add new disk to replace failed drive from degraded virtual disk.

Is the system have hot plug hard drive or cabled hard drive? If it have hot plug hard drive can you check LED status on each drive?

Thank you for the reply.

So the one "offline" physical disk (0-00) is the one that is ticking and not recognized at all.

The other 2 physical disks (1-01 and 2-02) say "online", but for some reason don't say "ready".

There's no indication of why or what the problem may be so I don't know what steps to take. Below is a more up to date screenshot of the PERC S100 now that I've installed the hot spare and set up a small SSD in non-RAID so I can run Open Manage Administrator. I'll Attach a screenshot of that shortly.

Also it's not hot swapable.

Here are the screenshots from the OpenManage Server Admin. It almost seems like the controller doesn't recognize these drives as even being part of the raid array anymore... Is that normal?

Hi, 
SSD and  SATA CAN'T co exsit in RAID.
"So the one "offline" physical disk (0-00) is the one that is ticking and not recognized at all.
The other 2 physical disks (1-01 and 2-02) say "online", but for some reason don't say "ready"."
->
dlrjdptj 0-00  is SSD 
and
1-01 , 2-02 is SATA.
Please see the attached.

Hmm it seems OpenManager is confusing things...

Disk 0-0.00 is not an SSD. I dind't even notice that myself. I do not have a 2TB SSD in this system. 

I have 3 2TB HDDs (2 original form the old RAID and one I replaced the failing disk with and it is a 3.5" WD Spinning HDD set as hot spare) . I later added one 120GB SSD in "non-RAID" to install Windows Server and run Open Manage. 

I don't know why it's reporting the first 2TB as an SSD but you can see in the BIOS screen that the disk is listed there as a HDD.

Thank you though... I wish that were the case. 

Hi, at this point I'd really like to ask you to back up your data first. It doesn't sound stable. And you could also tell us the LED status of the new HDD.

I would love to except I cannot access the data as the virtual drive is not mounted or accessible and this server does not have HDD LEDs. 

Hello Dan_Dan_84,

Were there any configuration changes or firmware updates made before issue?

Thanks,

No configuration changes or updates in the past 2 years. 

Hello Dan_Dan_84,

I would advise if you need the data and do not have a backup you may consider sending all original array member drives to a data recovery company. Example is DriveSavers, there are many others.

The screen shot with the Virtual Disks list is showing the size of the arrays as 0.02GB.  There may be meta data corruption.

The S100 is a software driven controller and it logs events in the Windows System Event log, which in inaccessible if the array  is failed.

Can you check the System Event Log (may also be SEL, ESM log) in OpenManage server administrator to check for drive failures?

Can you get screen shot of each physical in the S100 interface? View Physical Disk Details for each drive

Try shutting down and disconnect all drives. Boot to the S100 controller interface to see no configuration is present, exit and shut down. Seat all original member drives to see if it will import. If not try, shut down, unseating drive 00 then see if the array will import in a degraded state.

S100 user guide linked here. You may be down to recreate array, install and restore from backup.

WARNING: Found virtual disks that are Failed

Determine if two or more physical disks in a redundant virtual

disk have failed. If "yes," data is lost. Recover the lost data

from a backup storage source.

Page 66 : https://dell.to/36CWuHN

Thank you for the reply.

No there is no backup unfortunately. So we're thinking of trying a data recover firm. 

It's weird that there are such inconsistencies in reporting. The BIOS screen of the controller shows the correct the correct array sizes. 

I checked the system event logs in OpenManager and there is no events related to disk or array health. The server was functioning normally until one day it suddenly shut down. It was on battery backjup and no power loss. Then when attempted to turn on, it was in this state. 

I'll get those screenshots soon. You mention "physical in the S100 interface". I think you mean p[re-boot and not Open Manager correct? I'll have a look shortly.

As for unseating the 00 drive, I did try this before inserting the new drive. I'll try the unseating of all drives and checking configuration. 

I ran an express hardware diagnostics and everything passed.

I tried extended and found error code 0F00:1332 on disk  01 which seems to be bad blocks/sectors. I continued the test and it stopped again so I stopped the testing. 

Not a good outlook...

Hello Dan_Dan_84,

You are correct, I am talking about preboot in the S100 controller interface.

It's not looking good with PD00 failed and PD01 bad blocks.  You may like to review the following article:

How to Handle Puncturing (Bad Blocks) on Virtual Disks for PowerEdge servers

https://dell.to/3C4bmdV

Here are the photos of the controller screen for each physical disk.

I've dropped off the physically damaged disk for an evaluation from a data recovery firm. I'll find out tomorrow what the prognosis is for that drive. 

Hello Dan_Dan_84,

Thank you for the images. I thought it may give more detail about each physical disk but it doesn't.

We'll have to see what the data recovery firm finds out. Did you only drop off one hard drive? To my knowledge if a data recovery company is attempting to retrieve data of a virtual disk, they need all member drives of the array. But I have never worked at a data recovery company. You may want to check with them to make sure they know you want the data from your virtual disk retrieved if possible.

Hello Dan_Dan_84,

If you still have the configuration shown in your latest images, try a Rescan Disk

I don't know if it will work. It should require all original member disk but drive 0 was mechanically failed and wouldn't work anyway.