ORACLE DATABASES DONT COME BACK UP AFTER MIGRATING TO VPLEX

Fellow SAN Admins:

We migrated from LUNs from a CLARiiON Array to our new VPLEX Environment, recently. Everything went fine from a Storage perspective as well as the Linux/Operating System level. However, after the migration, our DBAs complained that they were not able to bring all their Databases UP at the same time. It was a shared database server running around 7 different databases. The error messages they encountered was related to a low shared memory issue. Nothing was changed during the migration from the OS side. This was the error message:

ORA-27300: OS system dependent operation:semget failed with status: 28
ORA-27301: OS failure message: No space left on device

If you run into similar complains after a migration, please request your UNIX Administrator to bump up the value of "kernel.sem". I doubled the value on the host to proceed further and complete the migration successfully.

If you are curious to know about the technical details, this is what we observed:

Each path belonging to an Invista LUN required one semaphore. [There were no semaphores used for the paths belonging to the CLARiiON LUNs on RHEL. CLARiiON LUNs do not use up a semaphore. It is only after the virtualization/encapsulation step that the server uses up one semaphore for each path]. The server had the recommended value of "shared memory/semaphores" from Oracle.  This particular server had around 28 LUNs presented to it. Each LUN had 4 Paths as detected by powerpath. This totalled to 28 X 4 = 112 semaphores. Oracle had only 16 semaphores for itself after this migration. [128 - 112 =116]

The "kernel.sem" parameter set on the server originally  was:

     kernel.sem=250 32000 100 128    ß Old Value

This left only (128 - 112 = 16) Semaphores for the Oracle databases to come back UP.

To move forward, we had to bump up this value to:

      kernel.sem=250 64000 100 256    ß New Value

Once this was done, the databases came back UP like a charm.  Your UNIX SA can check the number of semaphores on a give Linux server with this command before and after a migration  -  "ipcs –s | grep root | wc –l". You are are to encounter this issue if you have a large number of LUNs presented to the Server.

Just thought, I'd share this piece of information with you all. If you have any further queries, please write back to me. I can try to answer.

Regards,

-Kennedy