Is it total backplane failure or all disks have failed

hmm so its array, yes the server is up & running, its a PE 2800 box with server 2003 installed, should I run the diagnostics from OMSA or array manager?

Do you have other disks in this system that are running the OS?  If not, then none of the drives have actually failed, since your OS is still up and running and you are presumably seeing the Array Manager screen above from the booted/running OS :).  

1) Your array is corrupt and it has marked them all with a Predicted Failure (one or more may still be bad, having likely caused the corrupt array).

2) Possible that something happened to the Array Manager software or the PERC's driver, causing erroneous messages to occur.  (could be out of date firmware as the culprit too)

Which system is this?  (From the PERC 4e/Di in the screenshot, it would have to be a 2800, 2850, 6800, or 6850.)

First, make sure you have the important data backed up, while it is accessible, then run diagnostics on the drives to see which ones (if any) are actually bad.

Run Online Diagnostics (years ago, diagnostics were split off from OMSA/AM - all latest versions are standalone now):

ftp.dell.com/.../dell-onlinediags-win32-2.16.0.165.exe

sort of strange that the array disk 0:0, 0:1 & 0:2 are listed healthy in Dell online diagnostics but after a replacement since my last post, I now see this;

 I ran LSI array disk test but it failed for all 3 which baffles me even more. Why is this happening? I have updated the drivers & firmware but of no use. On comparing this image with the 1st, you'll see that now 2 drives have exclamation mark next to them whereas initially all 3 had a mark. Below I have pasted part of log from array manager. PERC & array disk 0:0 seems fine but 0:1 & 0:2 are reporting predictive failure. Does that mean replacing both drives will fix the issue once & for all.

**Release information**
Version = 3.7.0
Build = 575.6
Install Directory = C:\Program Files\Dell\SysMgt\Array Manager

Controller info:
Name: PERC 4e/DI Controller 0
Vendor: DELL
Firmware version: 5.B2D
Driver version: 6.46.2.32
Status: WARNING, READY
Memory size: 256.00 MB
Rebuild Rate: 30
Battery State: AUTO
Charge Count: 0


Array Disk info:
Name: Array Disk 0:0
Vendor: SEAGATE 
Status: ONLINE
Capacity: 33.87 GB
Product ID: ST336754LC 
Serial Number: 3KQ0PFVY
Revision: D402
Free Space: 0.00 Bytes

Array Disk info:
Name: Array Disk 0:1
Vendor: SEAGATE 
Status: PREDICTIVE WARNING, ONLINE
Capacity: 33.87 GB
Product ID: ST336754LC 
Serial Number: 3KQ0L2DH
Revision: D402
Free Space: 0.00 Bytes

Array Disk info:
Name: Array Disk 0:2
Vendor: SEAGATE 
Status: PREDICTIVE WARNING, ONLINE
Capacity: 33.91 GB
Product ID: ST336754LC 
Serial Number: 3KQ27GCV
Revision: D404
Free Space: 48.00 MB


Virtual Disk info:
Name: Virtual Disk 0
Status: READY
Layout: RAID 5
Capacity: 67.73 GB
Stripe size: 64 KB
Write Policy: Write-Back
Cache Policy: Direct I/O
Array Disk Members:
Array Disk 0:0
Array Disk 0:1
Array Disk 0:2

"I ran LSI array disk test but it failed for all 3 which baffles me even more"

Use Dell's diagnostics.  Dell drives have Dell firmware on them that third-party  utilities may not be able to navigate properly, making the results of anyone else's diagnostics unreliable.

Run the diagnostics I linked to above.

"I have updated the drivers & firmware but of no use"

That is only half the story (if it were a software mismatch causing this, see below) ... you should also uninstall Array Manager and install OpenManage Server Administrator now that the firmware/drivers are up to date.

"0:1 & 0:2 are reporting predictive failure"

This could still mean that either one or more disk is actual bad and/or that the array is damaged.

"Does that mean replacing both drives will fix the issue once & for all"

No.  If the array is damaged/corrupt, then replacing disks will not help; you'd be looking at a backup/restore after replacing the disks and recreating the RAID.  If the disks are bad, then there is also a good chance that rebuilding will fail, as the bad disk cannot be read or has corrupted the data.

You need to run reliable diagnostics to know what  you are dealing with, but either way, odds are you will need to rebuild/restore from backup and replace two disks.

hi flash, I used the Dell's diagnostic from the link you pasted earlier, LSI array disk test is one of the tests in that console, the other being blink test so I am not using any 3rd party utility...I have both array manager & OMSA installed on the server, are you suggesting to uninstall both & then re-install from scratch? Second option of replacing both the disks & the restoring server from backup sounds good but after 2 disks replacement & different results (each time 1 or more disks showing predictive failure) I am suspecting that its more to do with array but I am not sure. The Dell online diagnostics has very basic functionality, in main console it tells that the hard drive are healthy but nothing about array.

hi flash, the rescan once showed disk 0:0 & 0:1 with exclamations but a second rescan simply put exclamation next to all of them, 0:0, 0:1 & 0:2 all of them have predictive failure. Server is not happy at all which is making me nervous. Disk replacement is an easy job but rebuilding array will destroy everything on the disks. I am so hating it now!!!

If it were me (and feel free to blaze your own trail), at this point, I would simply backup my data (bit-level and file-level), test the drives, throw away any that fail, then create a new array with the good disks, then restore from backup.

just a thought which needs verification, if I replace disk 0:0 & disk 0:1, will I have to re-install the OS? Disk 0:2 has already been replaced once.

"if I replace disk 0:0 & disk 0:1, will I have to re-install the OS?"

Yes, if you replace them at the same time.  

No, if you replace them one at a time, allowing each to rebuild before replacing the other.

any update on this.  i presume that you ran full backups on the drives before replaceing/rebuilding them?

??

Thanks for all your help Flash