i have got the following error msg on 30th jul. just wanted to ensure no issues with that director now. how to ensure that?
symevent -sid xxx list -fatal
Symmetrix ID: xxx
Detection time Dir Src Category Severity Error Num
------------------------ ------ ---- ------------ ------------ ----------
Wed Jul 30 12:52:29 2008 DF-6C Symm Director Fatal 0x0040
A Symmetrix Director is not responding
The DF (disk director) is a redundant director. Meaning that if one goes DD, it's dual initiator will take over and control all of the it's drives as well as it's own. You will still have access to all of your disks, but you lose redundancy.
You should open up a service request to have this Symm looked at. The error indicates that director 6c (should be a back end disk director going by the DF) had a fatal error and is no longer responding. An error like this would have prompted a dial home but if your not sure have the lab look into it would be a good idea.
Symevent may help you .. In case processor was dead (DD) you can check with symcfg if all DF processors are Online. However I think it's better to open a SR and have someone dial the box and check health
A board in DD doesn't need to be replaced. PSE lab will diag why processors dropped DD and eventually reboot them. It happens also with drives. Sometime PSE spins down your "supposed-to-be-broken" drive and spins it up again .. If it spins up fine, it was a false error. And trust me it's easier to have a broken disk
However before replacing any single hardware part the CE have to carefully read all SR notes from PSE .. And sometime the CE can't even touch hardware if a PSE isn't logged in the box, issuing commands with a very special account (usually your CE will log on symwin using "CE" account while the PSE uses a different and powerfull user) and checking command output before giving directions to your local friendly CE.
Just as an example .. when you have 2 broken drives the CE can't change them. He needs a PSE to check box status and tell him what drive to replace first and when to replace second drive...
And before running any script CE have to run beloved KTS (key to success) script that does a boring and long check of many different aspects. And trust me, KTS will report a not working DF.
In a word .. there are a lot of people checking each other to avoid a single mistake to bring down your business.
in which case you want to have that DF replaced asap any way. This can be done online since the other DF is taking over all disk related IO anyway.
1 Question comesto my mind: if another DF goes offline which causes a DF to take over on this particular card (the one with the first failing DF) the system is still running fine, but replacing any of the 2 failing DF boards causes som disks to loose connectivity. How is this handled ?
Just to add a little on Stefano's comment about many people checking each other, the scripted procedures also perform various checks to verify the condition of the system and if it is safe to proceed. If any issues are found, a warning will be reported and require PSE assistance.
The director replacement script will check to see if any directors other than the one specified for replacement are in a failed state. If any are found, the script will post a warning. Many scripts (if not all) check for newly failed directors throughout the script.
The drive replacement script has a step(s) that check that all of the volumes on the drive to be replaced have ready and valid mirror/RAID members.
In response to RRR's question about having to replace a DF dual initiator pair, each with a failed processor: This would be considered a rare and very serious condition. High level PSE and / or development engineering assistance would be required to determine the best course of action. This would be determined on a case-by-case basis depending on the cause of the failures, so I cannot say exactly how the problem would be corrected. All measures would be taken to recover the situation without disruption.
MikeMac1
2 Intern
•
292 Posts
1328
0
Posted August 6th, 2008 06:00