Fellow SAN Admins:
We migrated from LUNs from a CLARiiON Array to our new VPLEX Environment, recently. Everything went fine from a Storage perspective as well as the Linux/Operating System level. However, after the migration, our DBAs complained that they were not able to bring all their Databases UP at the same time. It was a shared database server running around 7 different databases. The error messages they encountered was related to a low shared memory issue. Nothing was changed during the migration from the OS side. This was the error message:
ORA-27300: OS system dependent operation:semget failed with status: 28
ORA-27301: OS failure message: No space left on device
If you run into similar complains after a migration, please request your UNIX Administrator to bump up the value of "kernel.sem". I doubled the value on the host to proceed further and complete the migration successfully.
If you are curious to know about the technical details, this is what we observed:
Each path belonging to an Invista LUN required one semaphore. [There were no semaphores used for the paths belonging to the CLARiiON LUNs on RHEL. CLARiiON LUNs do not use up a semaphore. It is only after the virtualization/encapsulation step that the server uses up one semaphore for each path]. The server had the recommended value of "shared memory/semaphores" from Oracle. This particular server had around 28 LUNs presented to it. Each LUN had 4 Paths as detected by powerpath. This totalled to 28 X 4 = 112 semaphores. Oracle had only 16 semaphores for itself after this migration. [128 - 112 =116]
The "kernel.sem" parameter set on the server originally was:
kernel.sem=250 32000 100 128 ß Old Value
This left only (128 - 112 = 16) Semaphores for the Oracle databases to come back UP.
To move forward, we had to bump up this value to:
kernel.sem=250 64000 100 256 ß New Value
Once this was done, the databases came back UP like a charm. Your UNIX SA can check the number of semaphores on a give Linux server with this command before and after a migration - "ipcs –s | grep root | wc –l". You are are to encounter this issue if you have a large number of LUNs presented to the Server.
Just thought, I'd share this piece of information with you all. If you have any further queries, please write back to me. I can try to answer.
Thanks for sharing the tip on this.. A good example of the knowledge sharing we hoped would occur and why we believe this forum will be beneficial to VPLEX customers.