linux on Poweredge 2650: procedure to check and hotswap failed disk

Question

Hi.
I have 5 drives in a raid5 config. One drive has gone bad- amber led is permanently lit, and message about failed drive appear in admin mail:
---
WARNING: Kernel Errors Present
aacraid:ID(0:04:0); Error Event [command:0...: 2Time(s)
aacraid:ID(0:04:0); Medium Error [k:0x3,c:0x3,q:0...: 1Time(s)
aacraid:ID(0:04:0); Medium Error [k:0x3,c:0xc,q:0...: 1Time(s)
aacraid:ID(0:04:0); Write Error - Auto Reallocat...: 1Time(s)
---
WARNING: Kernel Errors Present
aacraid:ID(0:04:0); Error Event [command:0...: 11Time(s)
aacraid:ID(0:04:0); Hardware Error [k:0x4,c:0x15,q:...: 4Time(s)
aacraid:ID(0:04:0); Mechanical Positioning Error...: 4Time(s)
aacraid:ID(0:04:0); Medium Error [k:0x3,c:0x3,q:0...: 7Time(s)
---
I found some info on the PERC CLI for linux, but it seems quite old (2001), and it has an error when I install the rpm (missed it - could not determine controller...). Running afacli runs, but I can't get it to select the controller with: controller list.
----
1. Is the orange LED on the drive, and the kernel errors mean the drive is dead ?

2. What tools/commands can I use to view the drive serial numbers ?

3. Can I hot swap, or do I need to pause i/o on the particular drive first (how) ?

Thanks, Dave.

kaprino · Answer

Have you check the Reference->userguide->storage->controller Interfaces cards-> PERC I think you can set things up from firmware.

ratherthan · Answer

Thanks, I know I can shut the server down to swap the drive, and use the raid bios to rebuild the array!

But how do I do this while the server stays online - this is the promise of hot swap raid drive bays ? (If you know a web link, please post, I haven't been able to find anything). Thanks, Dave.

Message Edited by ratherthan on 02-04-2004 09:05 AM

bwonderlich · Answer

You need the manager software. It is on the support site, however I don't have a link for you off hand.
If Redhat was factory installed it might already be on your system.
Try running, as root
#dellmgr
If it is installed you will get a text interface similar to the BIOS one.
You will then be able to hot swap and rebuild the failed drive as usual.
HTH.

Hairyanimal · Answer

Can you not just pull out the failed disk and then replace it with a new one and it automatically starts to rebuild??

I am running RedHat 9.0 and I have yet to have a HD fail but when one does I would think thatyou just jank it and replace it??

Please correct me if I am wrong, I want to know what to do.

Thanks,

Danny

bwonderlich · Answer

Yes the drive should rebuild automatically. I would still recommend getting the tools installed ahead of time...if for no other reason than to have the ability to silence the alarm beeper. It can be very distracting to have to listen to it. :)
Also you would then be able to see the progress of the rebuild.

bwonderlich · Answer

Your Server Assistant CD will have the tools on it.
They show up under linux7.3 section on the CD I have, however I think they are statically linked, so they should run fine on RH9 also.
I think the actual file name is "lin_apps_a02.tar.gz"

HTH
Brian

Hairyanimal · Answer

I did not have my linux distro installed by Dell so how could I get dellmgr or other scsi utils installed??

I looked under the support->Downloads->SCSI PERC 3Di & RH9-> and only get :

Adaptec PERC3/Di, v. 2.7.1.3170, A05

File Date: 8/20/2002

PERC 3/Di Firmware for PowerEdge 2650 and PowerVault 775N, Multi Language, V2.7.1.3170, A05

Hairyanimal · Answer

I am afraid that those utils might be out of date for RH9. Is there a place where I can get src or does dell have some updates or does dell themselves approve that those tools will work on RH9?

Thanks!

ratherthan · Answer

I think that you should tell the drive to go offline - someone on the web hinted that it is possible to crash the server if this is not done. This would not be good. Further comment welcome...

Some info somewhere pointed me to a newer version of cli tools:
afaapps-2.7-1.i386.rpm [inside afa-linux-app-A01.tar.gz]
located : ftp://ftp.dell.com/scsi-raid/
This version did not have a problem during rpm install, and indeed can properly connect to the raid controller.

Basic usage: (any other tips on usage welcome).
:afacli
fastcmd>controller list

fastcmd>open afa0 [or whatever the controller has been named]
afa0>controller show channels
afa0>controller details
afa0>diagnostic show history
shows the errors.
afa0>diagnostic dump text
afa0>container list
afa0>disk list /full
afa0>disk show smart
afa0>disk show partition
afa0>disk show defects
afa0>disk show space
afa0>disk verify (0,4,0) [useful to confirm drive references to actual drive bays]
afa0>disk blink (0,4,0) 0 [off!]
afa0>task list [list controller tasks eg verify, rebuild]
afa0>enclosure show slot
afa0>enclosure prepare slot 0 4 [enclosure=0, slot=4, toggles activation], then show slot. At this point the drive is turned off (ie. power led off), ready for removal. We then installed the new drive, and typed this again to powerup the drive (ie power led on).

at this point the drives went spastic- the rebuild had begun automatically. Typing task list showed this active task, and it's completion percentage. Also, the bottom line of the terminal screen shows the rebuild task status %. It took some time (as ready next day).

afa0> quit (or was it exit) [to exit afacli].

Before the drive had been replaced, the following defect stats for each of the 5 drives in the array was noted:

id primary grown
0 290 0
1 923 1
2 1148 0
3 774 0
4 2612 278 had failed.

Can anyone enlighten me as to the exact meaning of primary (perhaps: detected at drive manufacture, and remapped to the "spare" area ?) and grown (perhaps: defects that have occured since installation, that would also have been mapped "spare" area until spare area was full ?) defects ?

Thanks to bwonderlich & Hairyanimal for some tips.

PowerEdge OS Forum

linux on Poweredge 2650: procedure to check and hotswap failed disk

Was this post helpful?