Info regarding DMX

1) I don't have a symdisk output available.. can you please post an example output ?? It will be easier to answer your question...

2) number of hotspares defined depends on a lot of variables.. it depends on the code level, on the number of drives and speed/size of drives in your storage.. A better answer requires a lot of "it depends" thus it's better if you give more details on the environment.. Code level, box type, number/speed/size of drives in the backend .. This will help in explaining you actual numbers

Ident - Identifier of processor
Symb - Symbolic name of processor (01A in your example)
Int - Interface (each processor have 2 interfaces, C and D)
TID - target identifier or if you prefer, address of drive in the look managed by processor.

When you want to dig into a single specific drive you still use "symdisk show" but supply also a "drive identifier" obtained merging symb,int and tid.

symdisk show 01aC:0

Can you please post output of command "symdisk -hotspare list" ??

When you buy the storage (or when you upgrade it), our salesman use a tool (direct express) that handles all the math behind hotspares (and a lot of other things). Thus when you order your box, it already have "the right" number of spares. I think that 1% of capacity is a good tradeoff .. recent codes have stricter rules, requiring more spares, spread at key positions in the backend.

The output from symdisk list is:

symdisk -sid xxx list |more

Symmetrix ID : 000125772xxx
Disks Selected : 1290

Capacity(MB)
Ident Symb Int TID Vendor Type Hypers Total Free Actual
------ ---- --- --- ---------- ---------- ------ -------- -------- --------
DF-1A 01A C 0 SEAGATE C146X15 52 140014 125 140014
DF-1A 01A C 2 SEAGATE C146X15 52 140014 125 140014
DF-1A 01A C 4 SEAGATE C146X15 52 140014 306 140014
DF-1A 01A C 6 SEAGATE C146X15 52 140014 306 140014
DF-1A 01A C 8 SEAGATE C146X15 52 140014 306 140014
DF-1A 01A C A SEAGATE C146X15 52 140014 306 140014



Actually, we have DMX 3 in our place and the code on it is 5671 and we have 146 and 300gb drives and a total of 1290 drives...and the drive speed is 15k rpm and we have hot spares of 18...can u explain why?

Message was edited by:
Stefano Del Corno

This is the output from symdisk list

symdisk -sid 01 list -hotspare

Capacity(MB)
Ident Symb Int TID Vendor Type Hypers Total Free Actual
------ ---- --- --- ---------- ---------- ------ -------- -------- --------
DF-1A 01A D 1D SEAGATE C146X15 0 0 0 140014
DF-2A 02A C 1D SEAGATE C146X15 0 0 0 140014
DF-5A 05A D B SEAGATE T300155 0 0 0 286102
DF-11A 11A D 1D SEAGATE T146155 0 0 0 140014
DF-16A 16A C 1D SEAGATE C146X15 0 0 0 140014
DF-1B 01B C 1D SEAGATE T300155 0 0 0 286102
DF-2B 02B D 1D SEAGATE C146X15 49 0 0 140014
DF-6B 06B D 1D SEAGATE T146155 0 0 0 140014
DF-15B 15B C 1D SEAGATE T300155 0 0 0 286102
DF-16B 16B D 1D SEAGATE C146X15 0 0 0 140014
DF-1C 01C D 1D SEAGATE C146X15 0 0 0 140014
DF-2C 02C C 1D SEAGATE C146X15 0 0 0 140014
DF-15C 15C C 4 SEAGATE C146X15 0 0 0 140014
DF-16C 16C C 1D SEAGATE T300155 0 0 0 286102
DF-1D 01D C 1D SEAGATE C146X15 0 0 0 140014
DF-2D 02D D 1D SEAGATE T146155 0 0 0 140014
DF-15D 15D C 1D SEAGATE C146X15 0 0 0 140014
DF-16D 16D D 1D SEAGATE T300155 0 0 0 286102

Thanks a lot guys....:)

Michael,

can you describe benefits of using permanent sparing option ? So the hot spare takes on a roll of the failed drive, so when the failed drive does get replaced ..data is not copied it to, it simply becomes an available hot spare?

With permanent spare option, let's say i have disk group 2 and i happen to have a drive fail in that disk group, so my hot spare kicks in. Does it become a member of disk group 2 at this point ?

Thanks

alo can you please let m eknow what do you mean by

AND that this spare drive is in a suitable "backend" location to fully maintain hardware fault tolerance otherwise the Permanent Sparing operation will NOT be performed ...I am unable to clearly understand this point....

Dynamox I'm neither Michael nor a "microcode" expert however I've personally seen permanent swaps and when you replace failed drive, the service processor builds a brand new binfile (at least that's what happened when I replaced a failed drive) , thus I suspect that hotspare becomes part of your disk group 2 while replaced drive becomes part of hotspares group...

Regarding advantages of permanent sparing .. You have less data moving around the backend .. Replacing a drive with "good old" dynamic spare requires a first rebuild/copy of data from failed/failing drive to hotspare and later another copy when you replace failed drive. With permanent sparing you rebuild/copy only once., actually reducing time required to rebuild integrity of your data. You obviously need more drives (but I'll give better details answering another questin in this same thread).

Just as told before, I'm neither Michael nor a "microcode expert" .. However as already told elsewhere, let's pick a RAID5 volume as an example.
Building a RAID volume requires 4 / 8 / 16 slices on different drives. But the slices needs to be on different loops, in different power zones, on different processors, to ensure your data is safe, even in case of failures in the backend. Our code will do all the best to avoid any SPOF between you and your beloved data. One of the things that matters, is location of slices in the backend.. But you already know that the three most important things are location, location, location

When the code invokes a permanent hotspare, faces same issues while creating new devices.. Thus the "new" slice should respect all restrictions coded in the microcode. If the code can't find a suitable drive, the hotspare won't kick in.

I'm pretty sure Michael will be more then happy to give an overview of all the different rules our code follows while creating devices.. But in a word: our microcode DO care about your data

Message was edited by:
Stefano Del Corno

won't kick in at all or will revert to dynamic sparing ?

Hi Stefano,

I am only a SymmWin & Configuration "expert" . Are you sure you are not a microcode expert (you seem to know this at least as well as I do )..... I am really only expanding on your earlier comments:

Hello All,

First to summarize:

One or more traditional "hot" or Dynamic Spare will be invoked by the microcode to ALL hypers of a failing drive as a temporary ("floating") mirror to ensure that all data is protected until the failing drive can be replaced by an EMC CE.

A Permanent Sparing "swap" is different. It will update the backend layout of the Symmetrix System bin file so that a suitable spare drive will take on ALL the attributes of the failing drive. The Symmetrix volume numbers are unchanged, all device attributes, channel assignments, meta & SRDF relationships, etc, are completely untouched. Consider the configuration change as analogous to (but NOT the same as) an "Optimizer" swap....

Yes, the BIG benefit of Permanent Sparing is that we copy the data ONCE:

Once the automated configuration change (i.e. Permanent Sparing "swap") has occurred the spare becomes the failing drive and the data is copied / rebuilt from the protection device(s).

One minor complication - IF there were any unprotected hypers on the failing drive an additional Dynamic Spare would have been temporarily invoked before and during the automated swap to protect this data (and act as the copy source for this unprotected data after the swap).

- SO the benefits that flow on from this are that we do not need / use ANY mirror positions for a permanent spare (assuming that all hypers on the failing drive were originally protected). Therefore no impact on TimeFinder Mirror operations, no impact on SYMCLI scripts (due to an invoked "hot" spare).

- AND even if the additional Dynamic Spare IS required (temporarily) it only copies the data from the unprotected hypers of the failing drive (NOT the entire drive). The Dynamic Spare only remains invoked for the duration of the automated swap and data copy / rebuild process (it does not require any EMC CE or PSE Lab intervention to be split). Therefore we automatically minimize the impact on TimeFinder Mirror operations and SYMCLI changes.

With an automated configuration change and only ONE data copy / rebuild we minimize performance impact, we minimize the inconvenience of the traditionally invoked "hot" spare and we minimize the time that any data is unprotected.

- Finally the failed drives can now be replaced at ANY convenient time (the "bad" drive is now flagged as a "Not Ready" spare in the Symmetrix System bin file). An EMC CE does not need to visit the Data Centre as a matter of urgency. The Not Ready spare can be replaced at ANY time with no impact (since there is NO data copy or rebuild required).

- One further advantage of Permanent Sparing is the option to now use the "EMC Certified Data Erasure" feature. This billable feature will erase the data on the failed drive (now a Not Ready spare) to US DoD specifications before it is removed from site....

The "disadvantage" of Permanent Sparing is the need for additional spares in the Symmetrix bin file:

- A spare is a "spare", that is it can be used by the microcode as a either a Dynamic Spare or as a Permanent Spare BUT as previously discussed we need suitably located spares for every drive type (SSD or HDD), size, speed & format (512 or 520 byte) in the configuration to make Permanent Sparing a viable option.

Now this rolls onto the second question (or really a requested clarification):

As Stefano has already stated when building Raid-5/6 or Mirror protected volumes the SymmWin program selects suitable backend locations for the Raid members or mirrored pairs.

- The rules ARE complex (and vary with Symmetrix model & Enginuity family) but are designed by Engineering to avoid single points of failure. Therefore the SAME stringent requirements that apply when creating NEW logical volumes are also applied to the "relocation" of ALL hypers present on the failing drive.

- Basically the SymmWin program will look at the physical backend location of the failing drive and then look at the location of all of the suitable (same type, size, speed & format) and "Ready" spares to see which one meets our fault tolerance "rules" for every hyper of the failing drive.

- If SymmWin fails to find a suitable candidate (i.e. no "Ready" spares of the same type, size, speed & format OR in a suitable backend location) then the microcode will automatically invoke Dynamic Sparing:

One or more traditional "hot" or Dynamic Spare will be invoked by the microcode to ALL hypers of a failing drive as a temporary ("floating") mirror to ensure that all data is protected until the failing drive can be replaced by an EMC CE.

- With Dynamic Sparing the microcode will invoke (almost) ANY ready spare as a temporary "floating" mirror to protect the data. The Dynamic Sparing rules allow any spare of any size or speed to be invoked (we still need spare drives of the same type & format).

- The physical backend location of a Dynamic Spare relative to the failing drive is NOT a concern. However, we in configuration have always recommend a minimum of two traditional "hot" spare drives per Symmetrix to avoid the situation where a Dynamic Spare is located on the SAME DA slice as the failing drive - i.e. invoking a dynamic spare in this circumstance would cause a severe performance impact.

O.K. I hope this helps clarify all the mysteries.

This posting is much longer than I intended but you asked for it .

However, if you still want more information I highly recommend the excellent "Drive Sparing in EMC Symmetrix DMX-3 and DMX-4 Systems" White Paper from EMC Applied Technology (available on Powerlink).

Best Regards,
Michael.

Hi Michael .. it looks like you REALLY expanded my 2 cents .. now we have a million dollar answer
I'm sure I'm neither a microcode expert nor a symwin&config expert .. Probably I simply read carefully some good documents someone gave me

Michael,

thank you for very detailed explanation ..i do have a follow up question.



so this is pro-active sparing ? The code is suspecting the driving is getting ready to fail so the code invokes hot spare. But if the drive really spun down and won't spin back up, the unprotected hypers are gone. Is that a true statement ?

Thanks

Hi Dynamox ....



I'm not Michael .. however sometime it happens .. sometime the code detects a failing drive (before it actually fails) ..



If you have "real" (unprotected) BCVs, your statement is true.

I had to re-establish huge metavolumes due to a failed drive (and its unprotected tiny slice)...

-s-