Start a Conversation

Solved!

Go to Solution

45288

May 23rd, 2019 17:00

When Will I know RAID is rebuilt? (Virtual Disk 0 on RAID Controller in Slot 6 reconfiguration has started)

I am extending a RAID5 virtual disk (3 1TB physical disks, added 2, so when completed 5 1TB PDs) on a PowerEdge T420 which has a PERC H310 controller.

After inserting the new drives and seeing they are Ready (as seen in iDRAC and also in Ctrl-R BIOS), I used the Lifecycle Controller to reconfigure the virtual disk, adding the 2 new physical disks to the RAID5 VD array.

I see in the Lifecycle log this message:

Virtual Disk 0 on RAID Controller in Slot 6 reconfiguration has started.

However, it has been over 24 hours and still no new progress messages. There does not seem to be any way to see the % complete. All I know is the physical disk LED lights are flashing, so I assume that means I/O activity. The iDRAC displays the original capacity of the 3 PD VD, it hasn't updated with the additional capacity.

Is there any way to see progress status?

What will be the completion message written to the Lifecycle log?

Will this ever complete? I am running low on patience and need to get more disk space ASAP.

30 Posts

May 28th, 2019 20:00

After nearly 120 hours (during which I was stressed out, even after confirming a recent backup was solid) I finally just did an ESXi OS restart -- and all is well. Thanks to theflash1932 for giving me some extra courage.

Mysteriously the expansion of the virtual disk required a system restart!?!? (I can find NOTHING in any documentation indicating that a PERC H310 firmware-based process requires a system restart.) I've extended many Dell PowerEdge disk arrays (or more often, replaced a failed drive) and none of those ever required system restarts.

I am guessing that message PDR54 (disk media error was corrected during recovery) caused VDR47 (disk media error was corrected ) which prevented message VDR24 (Reconfiguration completed.) It was at about 36 hours into the reconfiguration/expansion that VDR47 was issued and the drives stopped being noisy, so I'm guessing they had all re-striped by then.

It remains a mystery why was there NEVER a VDR24 message written to the Lifecycle log indicating successful reconfiguration/expansion (even after the system restart). Dell support, are you listening?

9 Legend

 • 

16.3K Posts

May 23rd, 2019 17:00

It will probably finish. The H330 is a low-end controller, so will have to learn patience. 24 hours "might" be enough time.

What OS? May be able to see the status in OMSA from the OS.

Just checking … what is your Boot Mode set to? If set to BIOS, then your boot "disk" is likely MBR … the max MBR disk size is 2TB, so you will NOT be able to utilize the new space once the array has been expanded unless you have installed the OS in UEFI mode (GPT disk).

30 Posts

May 23rd, 2019 18:00

Thanks for offering ideas. OS is VMware booting from flash, I know about 2TB limits, I hadn't considered OMSA, that's a good idea to check, I think I should be able to connect remotely using the VMware VIB target... Oops, OMSA requires a Windows server OS, the only server OS available is currently unavailable (it is a VM guest on the ESXi host this VD expansion is running on, and will not boot due to lack of drive space, although ESXi has booted). Must wait for VD to expand, then extend the Datastore to allow the VM to boot. *sigh*

Still looking for answers to specific Q's in original post.

9 Legend

 • 

16.3K Posts

May 23rd, 2019 19:00

You can install the VIB and use OMSA, but it is more limited than the version that runs on Windows (yes, needs to be physical instance), so it may or may not show you the progress.

 

30 Posts

May 24th, 2019 19:00

Screenshots illustrate that 5 PDs are present but that only the original 0,1,2 are forming 1.8TB usable space in the VD. The 2 new PDs added still shows they are waiting to be assimilated into the VD ("Used RAID Disk Space 0.00 GB" and "Available RAID Disk Space 931.00 GB"). I also show a screenshot of a PD that is part of the VD with the "Used/Available" reversed).

30 Posts

May 24th, 2019 19:00

Capture2.JPG

 

30 Posts

May 24th, 2019 19:00

Capture0.JPG

 

30 Posts

May 24th, 2019 19:00

Google search for Dell messages found this manual "Dell Test Event Notification Message Reference
2014/10/07" archived at manualzz.com (link).

 

VDR12 : " reconfiguration has started." When event is generated, message will have the following substitutions:  = "Virtual Disk 0 on Integrated RAID Controller 1"

VDR19 : "Reconfiguration of has failed." When event is generated, message will have the following substitutions:  = "Virtual Disk 0 on Integrated RAID Controller 1"

VDR24 : "Reconfiguration of has completed." When event is generated, message will have the following substitutions:  = "Virtual Disk 0 on Integrated RAID Controller 1"

VDR40 : "The reconfiguration of has resumed." When event is generated, message will have the following substitutions:  = "Virtual Disk 0 on Integrated RAID Controller 1"

 

At least I know that there should be a message logged when this expansion ends.  I'll check back in tomorrow, hopefully with good news.

30 Posts

May 24th, 2019 19:00

It has now been over 48 hours. The hard drives were flashing and I could hear the read/write heads making their usual "busy" noises for most of that time. Now there is silence (yes the server is powered on and functional, all health is green/good). 

There are no messages in iDRAC or Lifecycle log indicating the VD array is extended, no messages of any problems, no apparent completion, progress or error messages. This is just **bleep**! Even if there isn't a progress % complete reported, there should at least be a completion message eventually, shouldn't there?!?!? When will I know it is safe to do a restart?

Here are some screenshots of what I see:VD status after 48+ reconfiguration to extend, still shows capacity of original 3-PD arrayVD status after 48+ reconfiguration to extend, still shows capacity of original 3-PD array

There have been a couple of correctable errors detected on the original RAID5 VD as reported in the Lifecycle log. These appear to be benign. Still no progress or completion message that the additional PD have been successfully added to the VD. 

Do I dare initiate an OS restart? Will that interrupt the reconfiguration (with tragic results possibly destroying the VD?)

 

30 Posts

May 24th, 2019 19:00

Capture1.JPG

 

30 Posts

May 25th, 2019 10:00

60+ hours into the reconfiguration/expansion. Still not done. Additional symptoms:

CTL37: A Ptrol Read operation started for RAID Controller in Slot 6. Detailed Description: A ptrol read operation was initiated by the controller or the user. Recommended Action: No response action is required.

PDR54: A disk media error on Disk 0 in Backplane 1 of RAID Controller in Slot 6 was corrected during recovery. Detailed Description: This message is generated after a disk media error is corrected on a physical disk. Recommended Action: No response action is required.

LOG007: The previous log entry was repeated 560 times. Detailed Description: This message is generated after a series of sequential duplicate events. Log entry count is the number of times the previous message was repeated. Recommended Action: No response action is required.

PDR54: A disk media error on Disk 0 in Backplane 1 of RAID Controller in Slot 6 was corrected during recovery. Detailed Description: This message is generated after a disk media error is corrected on a physical disk. Recommended Action: No response action is required.

LOG007: The previous log entry was repeated 342 times. Detailed Description: This message is generated after a series of sequential duplicate events. Log entry count is the number of times the previous message was repeated. Recommended Action: No response action is required.

So it would seem the automatic "Ptrol Read" (which happens weekly, I believe) kicked off in the middle of the reconfiguration/expansion and surfaced over a thousand recoverable disk media errors present on the Disk 0 (I assume that is PD disk 0), one of the original (oldest) disks in this server. Despite "No response action is required." is worrisome. I found this unsolved question (link) indicating there may be a bug in PERC firmware triggering false positive PDR54 messages. This server's components are up-to-date with current versions of firmware. The latest added PD (not PD disk 0) might have a firmware update available, but it is too late for that until the dust settles.

Meanwhile, I wait for the reconfiguration/expansion. It is a holiday weekend, I will let this run its course, hopefully it will come to a happy completion before 150 hours elapsed! 

Unless anyone has ideas?

 

P.S. In order to post the messages I had to edit the word "Ptrol" (removed the "a"). For some reason it is a forbidden word in the Dell community?!?!

30 Posts

May 27th, 2019 13:00

Remembered there is an CLI interface to the iDRAC called RACADM. Installed it but I don't see anything new that isn't already exposed in the GUI web page. The RAID reconfiguration/expansion is still either slowly continuing, has ended but hasn't written a completion message, or has become suspended/hung. It has been over 90 hours.

RACADM is available in the Drivers download section for the server. The "iDRAC 8/7 v2.50.50.50 RACADM CLI Guide" is on Dell's website and provides complete description of available commands. But it does not help me since I am unable to find a command to query the status of an underway RAID5 reconfiguration/expansion operation.

The only idea I have is that because PDR54 was raised during the operation it completed and issues message VDR47 instead of VDR24. Possible bug?

Any ideas from Dell technical support would be most appreciated... And specifically, what happens if I cause an OS restart of the server (which will re-enter BIOS startup)... will that destroy the VD?

 

racadm -r 192.168.1.241 -u root -p myPW lclog view -c Storage
--------------------------------------------------------------------------------
Message ID      = VDR47
Category        = Storage
AgentID         = iDRAC
Severity        = Information
Timestamp       = 2019-05-24 11:49:07
Message         = A disk media error was corrected on Virtual Disk 0 on RAID Controller in Slot 6.
Message Arg   1 = Virtual Disk 0 on RAID Controller in Slot 6
FQDD            = Disk.Virtual.0:RAID.Slot.6-1
--------------------------------------------------------------------------------
Message ID      = PDR54
Category        = Storage
AgentID         = iDRAC
Severity        = Information
Timestamp       = 2019-05-25 03:10:21
Message         = A disk media error on Disk 0 in Backplane 1 of RAID Controller in Slot 6 was corrected during recovery.
Message Arg   1 = Disk 0 in Backplane 1 of RAID Controller in Slot 6
FQDD            = Disk.Bay.0:Enclosure.Internal.0-1:RAID.Slot.6-1

   

9 Legend

 • 

16.3K Posts

May 28th, 2019 10:00

"what happens if I cause an OS restart of the server (which will re-enter BIOS startup)... will that destroy the VD?"

I don't have access to technical documentation that Support will have, and I haven't had a chance to do anything on a test machine I have access too, but I can answer this one:

No, the OS has absolutely no bearing on the VD reconstruction/reconfigure. It will simply pick up where it left off after the hardware reboot.

1 Message

April 25th, 2021 07:00

Thank you very much for the detailed information 

I am waiting for the reconstruction be finished and losing my mind... your post gives me hope and courage 

I will continue to wait for a bit 

Thanks again, geoperkins!

November 27th, 2021 13:00

@ChefbigAre you still waiting for the RAID reconstruction?! lol, curious if you ran into the same issue. I'm 6 hours in and still at 1%.

No Events found!

Top