PS6100 RAID10 > RAID6 expanding progress stuck at 0%
PS6100 FW5.2.2 (yes I know not absolute latest/greatest)
Switched RAID configuration from RAID10 to RAID6 almost 24hrs ago. Progress is stuck at 0%
Load on this shelf is very low (SANHQ average IOPS shows 100) We've done this same thing before on other shelves and it's always gone much faster than this. Is it stuck? Any way to check what it's doing behind the scenes?
In the event log there's nothing really except for the initial two events:
Expanding disk array from 10 drives, RAID 10 to 11 drives, RAID 6.
Expanding disk array from 12 drives, RAID 10 to 12 drives, RAID 6.
In the GUI, go to (highlight) the member that you are expanding, on the status tab, in the upper right hand corner is the "RAID Status" section, this shows the RAID status as "expanding" and the progress in percentage.
That's the percentage I was talking about as being stuck at 0%.
Anyway, after 28hrs the percentage jumped from 0 to 24%, then an hour later it jumped to 78%. (and yes I was hitting the refresh button in the GUI, even though it does dynamically update for the most part)
Could have sworn in the past it went up smoothly but oh well.
We have the same problem - running 5.2.4 Firmware.
Migration from RAID 50 to RAID 6 was started at 10:30am this morning, at 5:30pm - 7 hours later - its reporting Progress 0%. Both the GUI and cmdline report the same thing...
GROUP1> mem select MEMBER2>show
_____________________________ Member Information ______________________________
This must be the slowest RAID expansion known to mankind, after nearly 24 hours the smallest of the arrays is only at 36% and the SAN has been idle because it was a Sunday:
support exec raidtool
You are running a support command, which is normally restricted to PS Series Tec
hnical Support personnel. Do not use a support command without instruction from
After 48 hours the RAID migration progress so far is a dismal 66% - for one small Array and its a tiny 4 Drive array, not the three main 14 drive arrays in the Member.
RAID LUN 3 Ok.
05 Drives (43,44,45,46,10)
RAID 6 (64KB sectPerSU)
Capacity 5,872,337,879,040 bytes
Expansion Underway (%66.34 complete)
4 Drive RAID 5 --> 5 Drive RAID 6
This is slower than glacial, if thats possible, so after 48 hours we still have no idea how long a RAID 50 to RAID6 migration is going to take as the first array isn't even complete.
I'm only reporting this because others will be wanting to do a RAID50 -> RAID6 migration to meet Dell's new August 2012 Best Practice guidelines.
Short answer is a Member might need a month to perform a migration?
It would be quicker to drain the member, delete and re-create it as a RAID6 member, and migrate the volumes back, if I had enough space to store 70TB of course.
GUI now reports Progress at 7%, which is odd as I'm 7% of the way through a conversion of a single 14 drive RAID set with two more to go... but at least its now updating as its just moved to 8%.
I'm hoping now that we have migrated what I assume are the "system" drives (the small RAID5 array) that the migration might speed up.
Might put a bottle of Gwertz in the fridge to celebrate when this migration is complete.
I'd rather be able to choose how fast migration takes, slow, medium or fast - as I'd normally choose medium, and accept some performance impact as the migration is a one off change. If the member is empty of volumes you might choose fast, so its still a valid option.
It is a very similar discussion that I've had about drive replacement after failure taking up to 30 hours to complete. Then lo and behold Dell have sped up their drive replacements in the latest firmware.
During a drive failure/replacement there is a huge risk window (of a second disk failure or a URE) if a RAID rebuild takes 30 hours - I'd rather it was done quickly, and now that is what Dell do.
Same goes for a "migration", the longer it takes, the bigger the risk of something else going wrong... for instance having a drive failure/replacement during the migration. If the migration was quicker, then we wouldn't even be contemplating doing a vacate to change RAID policy. Quid pro Quo.
It always made sense to mirror the failing drive to the RAID5(6) (system?) drives if there was enough space allocated for that process, as the drives could also work in parallel. Unforunately you don't always get warned that a drive is going to fail, so an array rebuild may still be necessary.
Ideally RAID6 would still have two hot spare drives instead of one, however a drive failure should no longer be a near death event, but using the small RAID5(6) array would still allow them to act as a temporary hot spare drive if there is space available, until a replacement hot spare drive is available.
Migration of the first 14 drive RAID set is now at 20% after 1.5 days, so the estimate of 7 days per RAID set to migrate from RAID 50 to RAID 6 with 2TB drives appears to be accurate.
Total of 3 RAID sets (with 2TB SATA drives) = 21 days plus 3.5 days (80 hours) for the small 4 drive RAID 5->6 conversion.
GUI shows conversion is at 20% when in fact only one of the three main Array sets are at 20%.
I'll give the Forum a rest until something new comes to light, as it may be some time before this migration completes if the RAID6 member migration does take 25 days to perform.
Some estimate of how long a migration is likely to take would be a big help, if people know a RAID6 migration might take close to a month then they might reconsider doing the migration!
With 1TB SATA drives the RAID 6 migration should take a more palatable 7-10 days.
With 3TB SATA drives presumably 50% can be added to my estimates.
It does sound like a long time, and i agree that it should be possible for one to select the priority that the conversion runs on.
What i am a bit more curious about, is the fact that it does not seem like all your customers have received the information about new raid recommendations.
Atleast I have not recieved that information :-P
Is it sent out to non-us costumers or is it handled world wide? My e-mail with all our serial numbers for arrays is registered on equallogic.com so it seems like a mistake that i do not get that rather vital information.
Joe S586
9 Technologist
•
729 Posts
1349
0
Posted August 16th, 2012 08:00
In the GUI, go to (highlight) the member that you are expanding, on the status tab, in the upper right hand corner is the "RAID Status" section, this shows the RAID status as "expanding" and the progress in percentage.
Also, this process can take a while.
-joe