symdisk list does not show disk failure but emc says possible disk failure

Question

Dear SAN Admins, I had a weird 'disk failure' issue yesterday which resulted in an outage in our production environment. There was a thin meta with a inconsistent meta member which resulted in a database panic and the file system going offline on a AIX box. Also this thin meta was a part of SRDF/A replication. Until we knew the issue, EMC tech support dialed home and worked on the SRDF repair and got the file system back online and we resumed production. Later on we came to know there is a 'possible' disk failure on our VMAX and a CE will be able to identify that. symdisk list -failed does not show any failed disks. Any possible causes ? (The frame is on latest code 5876). Regards Taz

taz_at_emc · Accepted Answer

Alright.. here is what I got as an RCA for this issue:

Upon analysis, EMC has told that the SRDF state was in split state prior to the meta member of that particular R1 metahead got corrupted. As per EMC, due to longer split times and production data continously hitting one of the R1 meta member, the meta member got corrupted which resulted in this downtime.

So to be on a safer side, I think its a best practise to split the R1-R2 only on purpose and once the change is done, change the mode but not keep the R1-R2 in split state for longer times.

Hope it helps.

Taz

sauravrohilla · Answer

I have some doubts, you said there was a possible disk failure which CE will be able to identify? Why did not the support see it?

EMC repaired your SRDF and that fixed your production data? What was the exact issue as both seems irrelevant?

Symdisk list -failed resulted nothing could mean either there was not a failed disk or it was already sync'ed and replaced.

regards,

Saurabh

Sreehari_PwC_UK · Answer

I'm not too sure if this would be of much help, but has permanent sparing occurred? Regards, Sreehari

Avinash V · Answer

Few thing to be checked - What is the SE version. Ensure it is at minimum v 7.4 Try 'symdisk -sid XXX -spare_info list -v' and let me know if you see the failed drive information.

Avinash V · Answer

My understand is that for the symdisk command to list a failed disk, the disk needs to go to NR state. This state needs to be passed from ucode to SE.

You can refer to the symaudit logs..

Refer https://support.emc.com/kb/73973

Tandok · Answer

Check the symaudit log and symevent list if you see any events of drive failures... Check the date when your issue occured

taz_at_emc · Answer

Symmetrix Command Line Interface (SYMCLI) Version V7.5.0.0 (Edit Level: 1604)

built with SYMAPI Version V7.5.0.0 (Edit Level: 1604)

No failed drive information in the second command. Am awaiting an RCA from emc on this.

A-J1 · Answer

I guess they were performing a proactive disk replacement, What i guess happened was, The TDAT for the meta encountered media errors (0311's) basically bad tracks, another drive in the same raid group might have been in the process of rebuilding due to permanane sparing so we have a raid group where few tracks are invalid on one member and the other raid member is having un-rebuilds for the same stripes. this is a track level dual raid failure.

So the drive that encountered bad tracks needs to be replaced proactively even though it has not failed completely.

VMAX

symdisk list does not show disk failure but emc says possible disk failure

Was this post helpful?