SAN file system mounted as read only

are there more than one file system from that VMAX on this host ? If yes, did it stay up ?

That's a good question. Yes it does also mount "/bckp" for the dataserver to dump the databases. Unfortunately going through the log files I believe nothing was a happening around that time. Also I believe we should have tried to touch a file and confirm it  like how we did it on the "/db" folder.

I opened a case with red hat and this is the response I got

As per the primary analysis on the error message, it seems that there is some issue with storage array. When a scsi layer timeout occurs, the SCSI layer must abort the command so that it cannot complete after we've given up waiting.

Can you please check with your storage team and make sure that there are any hardware issue or error logs available.

You may refer following article for details.

lpfc "SCSI layer issued Device Reset" messages in RHEL

https://access.redhat.com/solutions/39590

are you using PowerPath or DM-MPIO ? Did any other hosts connected to that VMAX experience any issues around that time ?

We were using powerpath (EMCpower.LINUX-5.7.4.00.00-003.el6.x86_64) . No only this host which is why its bizzare. Didn ot see any other meessages. Also we connect this host to 4 director ports on the VMAX 20K . Other machines which use the same Dir ports did not see this issues.

Can you explain the difference between the two ? In this case I believe its symopt

/sbin/powermt display dev=all

Pseudo name=emcpowera

Symmetrix ID=000192605947

Logical device ID=0733

state=alive; policy=SymmOpt; queued-IOs=0

==============================================================================

--------------- Host ---------------   - Stor -  -- I/O Path --   -- Stats ---

###  HW Path               I/O Paths    Interf.  Mode     State   Q-IOs Errors

==============================================================================

   2 lpfc                   sdi         FA  8gA  active   alive      0      0

   2 lpfc                   sdg         FA 10gA  active   alive      0      0

   1 lpfc                   sdc         FA  9gA  active   alive      0      0

   1 lpfc                   sde         FA  7gA  active   alive      0      0

Pseudo name=emcpowerb

Symmetrix ID=000192605947

Logical device ID=072B

state=alive; policy=SymmOpt; queued-IOs=1

==============================================================================

--------------- Host ---------------   - Stor -  -- I/O Path --   -- Stats ---

###  HW Path               I/O Paths    Interf.  Mode     State   Q-IOs Errors

==============================================================================

   2 lpfc                   sdh         FA  8gA  active   alive      0      0

   2 lpfc                   sdf         FA 10gA  active   alive      1      0

   1 lpfc                   sdd         FA  7gA  active   alive      0      0

   1 lpfc                   sdb         FA  9gA  active   alive      0      0

it was a typo on my part, should have said "SymmOpt". Looks good right now. Going back to the logs, specifically this string:

Error:Mpx:Bus 2 to Symmetrix 000192605947 port 8gA is dead

did you see it listed for other FAs as well ?

Dynamox,

EMC did do an RCA and they believe this is outside EMC infrastructure.  I believe they did  good job on looking  in to all switch logs, SAN, and system logs and also worked with Red hat vendors. Nothing conclusive. In the mean time I did move the san devices to a different machine just to rule out the possibility that it was a machine issue.

Hopefully it does not happening again

Thanks,

Ram

We had a similar issue in our environment where the file system went to Read Only mode and on further investigation from  Red hat confirms it's a Multi Path Set up issue.

SriNer,

Can you elaborate on that . What does it mean by  Multi Path issue ?

--Ram