UNSOLVED
Strange Failures
I have a T310 with a PERC S300 and 4 drives configured in a RAID 5 array. I noticed in the logs that drive 1 had failed and the Virtual disk was now degraded. So I obtained a replacement disk put it in, set it as hot spare and began the rebuilding process. About 10 minutes into the rebuild a log entry was made stating that drive 0 now a had a medium error and had failed.
2416 Thu Oct 02 17:10:22 2014 Storage Service Disk medium error detected: Physical Disk 0:0:0 Controller 0, Connector 0
2048 Thu Oct 02 17:10:22 2014 Storage Service Device failed: Physical Disk 0:0:0 Controller 0, Connector 0
2416 Thu Oct 02 17:10:22 2014 Storage Service Disk medium error detected: Physical Disk 0:0:0 Controller 0, Connector 0
2098 Thu Oct 02 16:59:46 2014 Storage Service Global hot spare assigned: Physical Disk 0:0:1 Controller 0, Connector 0
2064 Thu Oct 02 16:59:46 2014 Storage Service Virtual disk Rebuild started: Virtual Disk 1 (MAIN) Controller 0 (PERC S300)
The rebuild continued and after a few hours showed that it was successful:
2091 Thu Oct 02 20:41:56 2014 Storage Service Virtual disk Rebuild completed: Virtual Disk 1 (MAIN) Controller 0 (PERC S300)
2124 Thu Oct 02 20:41:56 2014 Storage Service Redundancy normal: Virtual Disk 1 (MAIN) Controller 0 (PERC S300)
The virtual disk still shows degraded as you would expect with a drive failure but the last entries in the log are the two above about success.
I'm not onsite any longer so I don't know what the drive lights are doing.
This seems like an amazing coincidence that as soon the new drive starts building another failed. And if it failed during the build would the build really have been successful?
Should I get another drive and start the process again? I can't help but feel like this behavior will continue like whack-a-mole. Or do you think something else is going on here?
Additional info: I updated the firmware and driver just before I added the new drive. Thanks!
In OpenManage all the drives show as normal and online (which is what it showed before I took the bad drive out). If OpenManage is not reporting the drive status properly then close out of OMSA and restart the services. They all start with DSM.
The S series controllers are driver based controllers. They do not have advanced error reporting like our hardware RAID controllers. Because of this it can be difficult to troubleshoot issues with these controllers. Errors reported in the operating system and OpenManage come from the driver. These errors cannot be trusted, and if possible you should reference the controller log to find out what the real problem is, if any.
Since that is not an option with this controller you have to make educated guesses as to what is going on. If OMSA still reports that all drives are online after restarting the services then you need to find a secondary method for checking drive status before proceeding. Checking the lights on the drives would be a secondary method. If the drive is online then the failure reported during the rebuild could have simply been a bad block that was found on the drive during the rebuild process. If the drive is offline then I would suggest reseating it and initiating a rebuild.
Thanks
Daniel My
12 Elder
•
6205 Posts
1025
0
Posted October 3rd, 2014 18:00
Hello
You should check OpenManage to find out what the status of the drives are. You should also run diagnostics.
Do you have a sliced array on the drives? If a failure occurred then the rebuild should not have completed. The only way I can think that a failure would occur and a rebuild would still complete is if there were multiple arrays on the same drive.
Thanks