How to verify I'm running the right version of Array Manager?

Uninstall the Array Manager, and download OpenManage v5.5 for this server.  When you install, choose  the custom installation option, and select the storage management option. This will let you monitor your arrays.  You can download from this link:  ftp.us.dell.com/.../OM_5.5.0_ManNode_A00.exe.

To check the consistency of your data, restart the server and enter the Ctrl-M bios during POST.  Once inside the bios, on the initial screen, arrow down to Check Consistency. Select the logical drives you want to check with the space bar, it will turn yellow. Hit F10 to run, select yes to check consistency.   This should take about 4-5 minutes per gigabyte to run.

"How to verify I'm running the right version of Array Manager?"

If you are running any version of Array Manager, it is out of date.  Array Manager has been replaced by OpenManage Server Administrator (OMSA), which now monitors the system itself AND RAID.

http://www.dell.com/support/drivers/us/en/04/DriverDetails?DriverId=R183698&FileId=2731119587&DriverName=Dell%20OpenManage%20Server%20Administrator%20Managed%20Node%2C%20v.5.4.0%2C%20A01

Download and run to extract files, then run C:\Openmanage\windows\setup.exe; choose Custom to make sure that Storage Management is installed.

"Maybe I've got a bad driver? Maybe the scsi controller is flaking out?"

Could be either, but your driver is up-to-date and these are less-likely scenarios ... it could also be a firmware bug or some uncorrectable corruption in Windows that just happens over the course of your OS's life span.  Or, more likely, it could also be a bad drive and/or a corrupt array.

What I would do is this:

First, make sure you have a good backup, then ...

Boot to 32-bit Diagnostics (it is downloadable from the Drivers and Downloads page of support.dell.com - or it might be available via F10 Utility Partition during POST) and test your system (memory and hard drives in particular) using the full diagnostics (not Quick Tests).

http://www.dell.com/support/drivers/us/en/04/DriverDetails?DriverId=5WXFW&FileId=2731106733&DriverName=Dell%2032%20Bit%20Diagnostics%2C%20v.5114A0%2C%205114.2

If anything fails, replace it immediately (although if drives, there may be corruption that cannot be corrected at this point).

If everything passes, boot to your OS, Array Manager/OMSA, and do a Consistency Check on the RAID 5.  After the Consistency Check, run a CHKDSK /R (you may even consider running it twice).  After that, I would update the system firmware (BIOS, ESM/BMC, RAID, then HDD).

"I wish Dell had a 'let me scan your system and tell you what versions of drivers and software utilities you should have' tool."

They do.  It is called SUU (Server Update Utility):
http://www.dell.com/support/drivers/us/en/04/DriverDetails/DriverFileFormats?DriverId=GHC9V&FileId=2731109500

You may need to run this first if very old:
http://www.dell.com/support/drivers/us/en/04/DriverDetails/DriverFileFormats?DriverId=R99082&FileId=2731131089

What are you looking for?  OMSA?

http://ftp.dell.com/sysman/OM_5.4.0_ManNode_A01.exe

oh man, clearly I've not been keeping up with this stuff....  :emotion-10:

This is really good info, and I hope to spend the day working on this.  I don't think I ever would have gleaned this info from the Driver downloads pages (and clearly I didn't!) 

Can I load and run Open Manage Server Administrator from either a volume other than "C:" or another system and do it remotely?  I have very little room on the C: boot volume to load it up with any more programs.

You can extract the 150MB of install files to another drive (say D:), but the program must actually be installed on C:.  I think it ends up using around 200MB once installed, but I believe it requires around 400-500MB free to install at all.

If you need help freeing up that amount of space to get it installed, take a look at this for places to trim:

www.lwcomputing.com/.../bootdrivesize.asp

Having a hard time with the link...

Ftp.us.dell.com/

and then what?

The fact that your CC didn't complete is a bad sign ... means that there chunks of data missing that cannot be reconstructed or recovered.  Too broad of damage can cause the controller to time out while trying to read data, causing drives to go offline, the OS to crash, etc.  You might check the controller log (OMSA, Storage, PERC, Information/Configuration (link at top of page), Export Log) for messages regarding the CC (and/or post it here).

Good luck, and yes, let us know how it goes.

"this machine was to be decommissioned after migrating it to another machine"

I never hold conversations like that around my computers/servers, for that very reason.

Forgot to mention, I did run consistency checks on both C: and E: (my two volumes) last night.  C: went very well.  E: took about 10 hours, which is about right for 136Gb.  But, it then shut itself down with a BSOD.  Event logs do show that the consistency check completed fully.  I didn't note that there were any errors in the log.  Still, things are buggy.

and yes, I am looking for OMSA, that link should do it, thanks!

I'm not out of the woods, but I'll have lots of work to do over the weekend.  What I find most ironic is that this machine was to be decommissioned after migrating it to another machine, but it's decided it would rather be troublesome than be shut down!  It's a struggle getting data restored from backups (the data corruption crashes the backups), but I'm doing that with redirection to a more stable machine.  Downtime is hard to come by...

I'll check back on Monday, if you don't mind, with status update.

THANK YOU for your help!

the CC DID complete at about 10:15 pm last night.  I stupidly then thought to run a defrag Analyze only on that volume, to see how large the Master File Table is, as well as how many files are there, and that crashed it. I had read that sometimes a huge MFT, > 4Gb, can cause problems, so I was curious to see how big it was.  Chkdsk /f was in the queue on re-boot, so that ran on that volume for a long time, but the system did restart.

okay, time to burn some iso's...

I was going to update BIOS first, not last, as @theflash1932 suggests.  I'm at A010, and need to be at A014.  I'll do what you all think is best, though.  :-)

update:  

I updated the Diags partition plus I created a bootable diags CD.

I had to update both the PERC 4/DI driver and firmware, before I could install OpenManage storage manager.

Installed OM, but is there an actual "OpenManage" application I should see, or how do I access it's functions from the OS?  I did not install the Web Admin option, or any of the others, just Storage Manager.  

I restarted, and am now running consistency checks from the BIOS.  I'll check back on it in the a.m.

I suggested updating the firmware first, as it isn't usually a good idea unless things are running well and are healthy

You can't second-guess what is required to run OMSA ... all OMSA data is served through a web interface, so all default portions selected for install are required.  Run through the install again, installing the web server portion.

thanks for that info.  I have made good progress.  Consistency checks ran overnight and came back absolutely good.  Got the BIOS  updated.  Ran chkdsk /f on C: 2 times and it came back with no errors.  Running chkdsk /f on E, which is where the data lies....  it's currently verifying indexes.  I'll reinstall OMSA w/ Typical settings...  Then, I'm not screwing with it anymore!  Next step will be to pull a "good" copy of the data off the drive, ASAP.  I have backups but the latest ones may be incomplete.  If I can pull data while no users are also accessing it, I'll be okay, I think.