Solved!
Go to SolutionDell R510 PERC 6/i - Read found an uncorrectable medium error
Dear Support,
I have a Linux server with a hardware raid behind a PERC 6/i controller (as per the lsscsi command)
[2:0:0:0] cd/dvd TEAC DVD-ROM DV-28SW R.2A /dev/sr0
[4:0:32:0] enclosu DP BACKPLANE 1.10 -
[4:2:0:0] disk DELL PERC 6/i 1.22 /dev/sda
[4:2:1:0] disk DELL PERC 6/i 1.22 /dev/sdb
[4:2:2:0] disk DELL PERC 6/i 1.22 /dev/sdc
Also, I'm getting this very often:
megaraid_sas 0000:02:00.0: 202961 (608338067s/0x0002/FATAL) - P-a-t-r-ol Read found an uncorrectable medium error on PD 05(e0x20/s5) at e900c8b7
That is coming from from the megaraid_sas linux kernel module. In the output it mentions PD 5, which is physical disk 5. The problem happens with /dev/sdc, which is the linux block device representing the mirror of two 2 TiB drives on Virtual Disk 2 through the perc 6/i. From time to time that mount point get slower and slower falling below 5M/s when writing.
And, when I execute: megacli -LdPdInfo -a0| sed -n '/^Virtual Drive: 2/,$p'
----------------------------------------------------------------------------------------------------------------
Virtual Drive: 2 (Target Id: 2)
Name :texasd1
RAID Level : Primary-1, Secondary-0, RAID Level Qualifier-0
Size : 1.999 TB
Mirror Data : 1.999 TB
State : Optimal
Strip Size : 64 KB
Number Of Drives : 2
Span Depth : 1
Default Cache Policy: WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU
Current Cache Policy: WriteThrough, ReadAheadNone, Direct, No Write Cache if Bad BBU
Default Access Policy: Read/Write
Current Access Policy: Read/Write
Disk Cache Policy : Disk's Default
Encryption Type : None
Is VD Cached: No
Number of Spans: 1
Span: 0 - Number of PDs: 2
PD: 0 Information
Enclosure Device ID: 32
Slot Number: 5
Drive's postion: DiskGroup: 1, Span: 0, Arm: 0
Enclosure position: N/A
Device Id: 5
WWN:
Sequence Number: 2
Media Error Count: 189
Other Error Count: 31
Predictive Failure Count: 0
Last Predictive Failure Event Seq Number: 0
PD Type: SATA
Raw Size: 1.999 TB [0xfffffffe Sectors]
Non Coerced Size: 1.999 TB [0xffeffffe Sectors]
Coerced Size: 1.999 TB [0xffec0000 Sectors]
Firmware state: Online, Spun Up
Device Firmware Level: CE02
Shield Counter: 0
Successful diagnostics completion on : N/A
SAS Address(0): 0x1221000005000000
Connected Port Number: 5(path0)
Inquiry Data: Z1F35954ST3000NC000 CE02
FDE Capable: Not Capable
FDE Enable: Disable
Secured: Unsecured
Locked: Unlocked
Needs EKM Attention: No
Foreign State: None
Device Speed: Unknown
Link Speed: Unknown
Media Type: Hard Disk Device
Drive Temperature : N/A
PI Eligibility: No
Drive is formatted for PI information: No
PI: No PI
Port-0 :
Port status: Active
Port's Linkspeed: Unknown
Drive has flagged a S.M.A.R.T alert : No
PD: 1 Information
Enclosure Device ID: 32
Slot Number: 6
Drive's postion: DiskGroup: 1, Span: 0, Arm: 1
Enclosure position: N/A
Device Id: 6
WWN:
Sequence Number: 2
Media Error Count: 0
Other Error Count: 25
Predictive Failure Count: 0
Last Predictive Failure Event Seq Number: 0
PD Type: SATA
Raw Size: 1.999 TB [0xfffffffe Sectors]
Non Coerced Size: 1.999 TB [0xffeffffe Sectors]
Coerced Size: 1.999 TB [0xffec0000 Sectors]
Firmware state: Online, Spun Up
Device Firmware Level: CE02
Shield Counter: 0
Successful diagnostics completion on : N/A
SAS Address(0): 0x1221000006000000
Connected Port Number: 6(path0)
Inquiry Data: Z1F3593LST3000NC000 CE02
FDE Capable: Not Capable
FDE Enable: Disable
Secured: Unsecured
Locked: Unlocked
Needs EKM Attention: No
Foreign State: None
Device Speed: Unknown
Link Speed: Unknown
Media Type: Hard Disk Device
Drive Temperature : N/A
PI Eligibility: No
Drive is formatted for PI information: No
PI: No PI
Port-0 :
Port status: Active
Port's Linkspeed: Unknown
Drive has flagged a S.M.A.R.T alert : No
---------------------------------------------------------------------------------------------------------------
I have not checked the BIOS to see if the drive is in Predictive Failure ONLINE or OFFLINE because this is a production server. I already have the replacement drive. I also see in the previous output "nquiry Data: Z1F35954ST3000NC000 CE02" so that is a 3 TiB drive used on a 2 TiB mirror correct?
I see that Z1F35954 is the serial number that match when I run smartctl (which also shows 3 TiB instead of 2). And also the other drive in this 2TiB mirror seems to be of 3 TiB. The new drive I have is a 2 TiB. Is this information correct from megacli? I'm planning to replace one of the disks (the one with serial number: Z1F35954). Also that disk have many errors ("Media Error Count: 189"). I assume that physically it has bad sectors that are being reallocated.
I want to shutdown the server, replace the disk (by using the serial number to identify it) and then it should rebuild by itself correct? (Do I have to force it offline first before removal?).
Edit: I found out that the PERC 6/i have a 2 TiB limit. The replacement drive I have is 2 TiB. Will I run into issues? Because the drives on the mirror are both 3 TiB. What worries me is that for a 2 TiB one that will go down to 1.8x or something.
Hello
Disk replacement in a virtual disk requires that the replacement be equal to or greater than the smallest drive in the virtual disk. That is in regard to used space on the disk, so if the virtual disk is only using 2TB of the 4TB drive it should be fine to replace it with a 2TB drive.
The best practice to replace an online drive would be:
- Backup important data
- Run a consistency check (if supported)
- Offline the drive (if supported)
- Hot-swap drive (if supported)
- Shut down the system to replace cabled drives
- A rebuild should initiate automatically, set drive as hot spare if automatic rebuild does not start
- It can take a few minutes for the rebuild status to update
Not all of the best practice operations are supported on every controller or configuration.
http://www.dell.com/storagecontrollermanuals/
Thanks
Daniel,
Thanks. To put it offline. I did the following: omconfig storage pdisk action=offline controller=0 pdisk=1:0:5
Checked that the status was indeed Offline:
...
ID : 1:0:5
Status : Critical
Name : Physical Disk 1:0:5
State : Offline
...and then I removed (hot-swapped) the drive and put back he replacement. Then I checked:
omreport storage pdisk controller=0 vdisk=2
and it shows physical disk 6 only, as the member of the virtual-disk. Then (you mentioned it can take minutes to start a rebuild) I saw that vdisk2 was still (after 45 minutes) with one member. Then I tried to add the disk as a dedicated hot-spare:
omconfig storage vdisk action=assigndedicatedhostspare controller=0 vdisk=2 pdisk=1:0:5 assign=yes
and I'm welcomed with this message:
Invalid physical disk value. Read, pdisk=1:0:5
The physical disk cannot be assigned as dedicated hot spare to the virtual disk you selected. Possible reasons include: Invalid physical disk id, insufficient physical disk space, unsupported mix of SAS and SATA type physical disks.Valid values for physical disk are: None
------------------------------------------------------------------------------
As per the the output of: omreport storage pdisk controller=0
I see that physical disk 5 is SAS (I knew) and physical disk is SATA (not aware of that fact, it says that in "Bus Protocol"). What do you think? Should I get a SATA on with 3 TiB right? There is something I can do?
Daniel My
12 Elder
•
6205 Posts
2719
1
Posted April 16th, 2019 13:00
You cannot mix SAS and SATA within a virtual disk, so if the virtual disk is using SATA drives then the replacement needs to be SATA.