UNSOLVED

darkog

updated

6 years ago

D

darkog

2 Intern

56 Posts

0

4802

October 2nd, 2020 08:00

PERC H710P Reporting all disks failed - 2013 model controller

Hello, I am working on a situation where I have a server that is powering up but the controller is reporting all the disks are failed. The controller and the PERC battery are reporting working. There was a power outage and the server did crash due to loss of power. The disks are a mix on TOSHIBA MK2001GRZB and SEAGATE ST9600205SS  SAS disks. .They should be all certified Dell disks as we didn't replace them since in went into production.

The BOIS POST indicated that all there is no configuration and all disks are failed and there is no foreign configuration state to import. I tried both the PERC interface , iDRAC  as the Dell Life Cycle manager interface. In all interfaces the foreign config is greyed out and all the disks are reporting failed state and zero size .

Server: PE R620 circa 2012 Q4.

Model: PERC H710P Mini (Embedded)

FW: 21.3.5-0002

Has anyone run into this issue and is there any troubleshooting advice on how to further troubleshoot this.

I am looking into re-seating the controller and disks.

  • DELL-Chris H

    7 Practitioner

    9682 Posts

    48046 Points

    2404

    0

    Posted October 2nd, 2020 13:00

    Darkog,

     

    Do you happen to have another R620 that we can use known good parts from to test the server? I would start by testing the controller, backplane, and drives. If unable to verify the issue as those, then I would look at the motherboard. You can also try running diagnostics to see if it spots an issue. 

     

    Let me know what you see. 

     

     

    DELL-Chris H

    Social Media and Communities Professional

    Dell Technologies | Enterprise Support Services

    #IWork4Dell

    Did I answer your query? Please click on ‘Mark as Accepted Answer’. ‘Thumbs up’ the posts you like!

  • darkog

    2 Intern

    56 Posts

    2402

    0

    Posted October 2nd, 2020 14:00

    I am afraid not.

    But all the disks are listed as failed.

    I find it hard to believe all drives failed from one crash.

    /admin1-> racadm raid get controllers -o
    RAID.Integrated.1-1
    Status = Ok
    DeviceDescription = Integrated RAID Controller 1
    RollupStatus = Warning
    Name = PERC H710P Mini (Embedded)
    FirmwareVersion = 21.3.5-0002
    DriverVersion = Information Not Available
    RebuildRate = 30
    BgiRate = 30
    CheckConsistencyRate = 30
    ReconstructRate = 30
    PatrolReadRate = 30
    PatrolReadMode = Automatic
    PatrolReadState = Stopped
    CheckConsistencyMode = Normal
    LoadBalanceSetting = Auto
    CopybackMode = ON
    PreservedCache = Not Present
    CacheMemorySize = 1024 MB
    PersistHotspare = Disabled
    KeyID = null
    SpindownUnconfiguredDrives = Disabled
    SpindownHotspare = Disabled
    Timeintervalforspindown = 30 (Minutes)
    SecurityStatus = Encryption Capable
    EncryptionMode = None
    SasAddress = 0x5D4AE520B2523200
    PciDeviceId = 0x5b
    PciSubdeviceId = 0x1f34
    PciVendorId = 0x1000
    PciSubvendorId = 0x1028
    PciBus = 0x3
    PciDevice = 0x0
    PciFunction = 0x0
    BusWidth = Unknown
    SlotLength = Unknown
    SlotType = Unknown
    MaxCapableSpeed = 6.0 Gb/s
    LearnMode = Not supported
    T10PICapability = Not Capable
    SupportRAID10UnevenSpans = Not Supported
    SupportEnhancedAutoForeignImport = Supported
    EnhancedAutoImportForeignConfig = Disabled
    SupportControllerBootMode = Supported
    ControllerBootMode = User Mode
    RealtimeConfigurationCapability = Incapable
    bootVD = None
    CurrentControllerMode = Not Supported

    /admin1-> racadm raid get enclosures -o
    Enclosure.Internal.0-1:RAID.Integrated.1-1
    State = Ready
    Status = Ok
    DeviceDescription = Backplane 1 on Connector 0 of Integrated RAID Controller 1
    RollupStatus = Failed
    Name = BP12G+ 0:1
    BayId = 1
    FirmwareVersion = 1.00
    SasAddress = 0x5E4AE020B2523200
    SlotCount = 8
    /admin1->

     

    /admin1-> racadm raid get pdisks -o -p Manufacturer,ProductId,State,Status
    Disk.Bay.0:Enclosure.Internal.0-1:RAID.Integrated.1-1
    Manufacturer = TOSHIBA
    ProductId = MK2001GRZB
    State = Failed
    Status = Failed
    Disk.Bay.1:Enclosure.Internal.0-1:RAID.Integrated.1-1
    Manufacturer = TOSHIBA
    ProductId = MK2001GRZB
    State = Failed
    Status = Failed
    Disk.Bay.2:Enclosure.Internal.0-1:RAID.Integrated.1-1
    Manufacturer = TOSHIBA
    ProductId = MK2001GRZB
    State = Failed
    Status = Failed
    Disk.Bay.3:Enclosure.Internal.0-1:RAID.Integrated.1-1
    Manufacturer = TOSHIBA
    ProductId = MK2001GRZB
    State = Failed
    Status = Failed
    Disk.Bay.4:Enclosure.Internal.0-1:RAID.Integrated.1-1
    Manufacturer = TOSHIBA
    ProductId = MK2001GRZB
    State = Failed
    Status = Failed
    Disk.Bay.5:Enclosure.Internal.0-1:RAID.Integrated.1-1
    Manufacturer = SEAGATE
    ProductId = ST9600205SS
    State = Failed
    Status = Failed
    Disk.Bay.6:Enclosure.Internal.0-1:RAID.Integrated.1-1
    Manufacturer = SEAGATE
    ProductId = ST9600205SS
    State = Failed
    Status = Failed
    Disk.Bay.7:Enclosure.Internal.0-1:RAID.Integrated.1-1
    Manufacturer = SEAGATE
    ProductId = ST9600205SS
    State = Failed
    Status = Failed

     

  • Dell-DylanJ

    6 Operator

    2924 Posts

    2398

    0

    Posted October 2nd, 2020 15:00

    The hardware reseat doesn't seem like a bad idea to me, I'd go ahead and include the backplane cabling, as well. I'd have to suggest exporting the RAID log, if reseating doesn't help. This will at least help to give some indication as to what the issue may be. You can view the log in a text editor. I'd search for the term "puncturing." A puncture is an event that could cause multiple drive failures, so it may be worth checking for.

     

    Punctures and Double Faults: https://dell.to/3iypilj

    Exporting RAID log: https://www.dell.com/support/article/en-us/sln249450/how-to-export-the-perc-controller-log-via-openmanage-server-administrator-omsa?lang=en 

  • darkog

    2 Intern

    56 Posts

    2394

    0

    Posted October 2nd, 2020 15:00

    Thanks.  I suspect that as well. I will try the re-seat.

  • darkog

    2 Intern

    56 Posts

    2349

    0

    Posted October 5th, 2020 09:00

    I re-seated the drives and the PERC controller. Same issue.

    I ran system level diagnostics and when I pull out the drives, all components seem to pass.

    What I am was able to do and see.

    the system consists of toshiba and seagate drives.

    When I re-insert each drive, one by one, the toshiba drives report as blocked by the PERC controller software.

    The seagate drives report as foreign. I am then able to import the foreign drives and volumes for the data on seagate drives. But not on Toshiba drives.

    What is odd is all the drives are original Dell drives that were shipped by Dell and should not be non-certified drives.

    I am attempting to downgrade the PERC controller firmware to see if the Toshiba drives will become unblocked. 

    The firmware is at 21.3.5-0002, A09

    The Toshiba drives have never hard their firmware updated.

     

  • DELL-Chris H

    7 Practitioner

    9682 Posts

    48046 Points

    2341

    0

    Posted October 5th, 2020 11:00

    Using the directions found here, would you export a TSR and upload it to https://dell.to/3lkcFvB. Afterwards would you Private Message the svc tag used to upload it. 

    Thank you. 

     

     

    DELL-Chris H

    Social Media and Communities Professional

    Dell Technologies | Enterprise Support Services

    #IWork4Dell

    Did I answer your query? Please click on ‘Mark as Accepted Answer’. ‘Thumbs up’ the posts you like!

  • darkog

    2 Intern

    56 Posts

    2335

    0

    Posted October 5th, 2020 12:00

    Hello, I just uploaded the zip file to that location. Unfortunately, I am not able to PM you.
  • darkog

    2 Intern

    56 Posts

    2333

    0

    Posted October 5th, 2020 12:00

    The PERC firmware is older now because I downgraded it to see if the drives will be unblocked. I will update the PERC firmware again.
  • darkog

    2 Intern

    56 Posts

    1307

    0

    Posted October 6th, 2020 05:00

    thanks. that finally worked.

  • darkog

    2 Intern

    56 Posts

    2295

    0

    Posted October 6th, 2020 05:00

    The PM feature still seems to not work.