colby,

Not sure if you looked at this document, but it might be worth a review to ensure you covered the best practices first: www.equallogic.com/.../DownloadAsset.aspx

Also, this link has a boatload of info for Oracle: www.equallogic.com/.../default.aspx

-Joe

Hi Joe,

We've been through a significant amount of the reading on those links.  The articles were helpful to us for setting up the EqualLogic array to connect from the OS perspective.

It's the Oracle Universal Installer (OUI) that will not 'discover' the ASM disks.  It appears that the ASM disks are being created and seen from the cmd line.  I was hoping that someone could shed some light on the reason the OUI doesn't see the ASM disks.  I think this is an Oracle issue, but I was hoping someone at Dell had come across and solved this during their testing of various Oracle installations.

Basically I haven't been able to find anyone specifically seeing what we're seeing so I put out the feeler's as far as I could reach them.

Thanks again,

Colby

It appears that you are using EqualLogic HITKit

did you implement the work around which Dell recommended? like adding a file in /etc/udev/rules.d directory?

if so give me the contents of

1. /etc/sysconfig/oracleasm

2. ls -l /dev/dm*

3. ls -l /dev/oracleasm/disks

3. ls -l /dev/mapper/eql*

The info you asked for is below:

[root@dr-dbnode2 ~]# cat /etc/sysconfig/oracleasm

#

# This is a configuration file for automatic loading of the Oracle

# Automatic Storage Management library kernel driver.  It is generated

# By running /etc/init.d/oracleasm configure.  Please use that method

# to modify this file

#

# ORACLEASM_ENABELED: 'true' means to load the driver on boot.

ORACLEASM_ENABLED=true

# ORACLEASM_UID: Default user owning the /dev/oracleasm mount point.

ORACLEASM_UID=grid

# ORACLEASM_GID: Default group owning the /dev/oracleasm mount point.

ORACLEASM_GID=asmadmin

# ORACLEASM_SCANBOOT: 'true' means scan for ASM disks on boot.

ORACLEASM_SCANBOOT=true

# ORACLEASM_SCANORDER: Matching patterns to order disk scanning

ORACLEASM_SCANORDER=""

# ORACLEASM_SCANEXCLUDE: Matching patterns to exclude disks from scan

ORACLEASM_SCANEXCLUDE=""

[root@dr-dbnode2 ~]# ls -l /dev/dm*

ls: /dev/dm*: No such file or directory

[root@dr-dbnode2 ~]# ls -l /dev/oracleasm/disks/

total 0

brw-rw---- 1 grid asmadmin 8, 113 Jan 17 13:30 ASMDATA1

brw-rw---- 1 grid asmadmin 8,  65 Jan 17 13:30 ASMDATA2

brw-rw---- 1 grid asmadmin 8, 145 Jan 17 13:30 ASMDATA3

[root@dr-dbnode2 ~]# ls -l /dev/mapper/eql*

brw-rw---- 1 root disk 253, 12 Jan 17 13:34 /dev/mapper/eql-8-cb2b76-074e2e258-08670a9557f4f0dd-oracleasmdisk1

brw-rw---- 1 root disk 253, 13 Jan 17 13:34 /dev/mapper/eql-8-cb2b76-074e2e258-08670a9557f4f0dd-oracleasmdisk1p1

brw-rw---- 1 root disk 253,  6 Jan 17 13:14 /dev/mapper/eql-8-cb2b76-076e2e258-53670a9556a4f075-oraclefra1

brw-rw---- 1 root disk 253,  7 Jan 17 13:14 /dev/mapper/eql-8-cb2b76-076e2e258-53670a9556a4f075-oraclefra1p1

brw-rw---- 1 root disk 253,  3 Jan 17 13:34 /dev/mapper/eql-8-cb2b76-0d8e2e258-55270a955854f0dd-oralceasmdisk2

brw-rw---- 1 root disk 253,  4 Jan 17 13:34 /dev/mapper/eql-8-cb2b76-0d8e2e258-55270a955854f0dd-oralceasmdisk2p1

brw-rw---- 1 root disk 253,  8 Jan 17 13:34 /dev/mapper/eql-8-cb2b76-0fde2e258-3ac70a955884f0dd-oracleasmdisk3

brw-rw---- 1 root disk 253,  9 Jan 17 13:34 /dev/mapper/eql-8-cb2b76-0fde2e258-3ac70a955884f0dd-oracleasmdisk3p1

adendukuri, thanks for the response.

I did not follow a specific rules.d work around.  I googled for a related work around but did not find it.  Can you point me to the work around you're referring to?

Yes, we are using the HIT Kit - installed as per the guide:

     Host Integration Tools for Linux - Installation and User Guide - Version 1.0 - January 2011

The installation does create a rules.d file:      99-equallogic.rules

The info requested to follow...

[root@dr-dbnode2 ~]# cat /etc/udev/rules.d/99-equallogic.rules

#-----------------------------------------------------------------------------

#  Copyright (c) 2010 by Dell, Inc.

#

# All rights reserved.  This software may not be copied, disclosed,

# transferred, or used except in accordance with a license granted

# by Dell, Inc.  This software embodies proprietary information

# and trade secrets of Dell, Inc.

#

#-----------------------------------------------------------------------------

#

# These udev rules should serve as a guide for setting up system-wide UDEV

# rules for Dell EqualLogic storage devices.

#

# Rule #1: Set the SCSI subsystem device timout to 60s for every Dell

# EqualLogic device detected.  It is not recommended to change this from

# the value set here.

ACTION=="add",SUBSYSTEM=="block",ENV{ID_VENDOR}=="EQLOGIC",ENV{ID_FS_USAGE}=="",\

RUN+="/bin/sh -c 'echo 60 > /sys/$DEVPATH/device/timeout'"

# Example Rule #1: The recommended read-ahead size is between 1024 and 4096.

#

# Higher values are better for sequential access by prefetching this amount (in

# kb), at the cost of random access performance.

#

# Uncomment the following two lines to set the read ahead size to 1024kb for

# all Equallogic devices:

ACTION=="add",SUBSYSTEM=="block",ENV{ID_VENDOR}=="EQLOGIC",ENV{ID_FS_USAGE}=="",\

RUN+="/bin/sh -c 'echo 1024 > /sys/$DEVPATH/queue/read_ahead_kb'"

# Example Rule #2: per-WWID matching

#

# You can override values on a per-volume basis, based on the SCSI WWID value.

#

# Cut and paste this rule as many times as necessary, replacing the example

# WWID (in the RESULT=="..." line) with the actual WWID of the SCSI device you

# wish to match.

#

# You can also replace the setting being altered, for example to change the

# disk IO scheduler instead, change 'read_ahead_kb' to 'scheduler'.

#ACTION=="add",SUBSYSTEM=="block",ENV{ID_VENDOR}=="EQLOGIC",ENV{ID_FS_USAGE}=="",

#PROGRAM="/sbin/scsi_id --whitelist --device=$tempnode",

#RESULT=="360c090629cc917ce72cbb40000003000",

#RUN+="/bin/sh -c 'echo 2048 > /sys/$DEVPATH/queue/read_ahead_kb'"

We managed to figure it out...

It turns out that a previously failed -silent run of the OUI had left some Oracle files hanging about.  The OUI not showing the ASM disks because they were associated to the failed installation.  After removing 'ALL' the files from the failed installation and a reboot we were able to get through the full install successfully.

I also added the following to the end of the /etc/udev/rules.d/99-equallogic.rules to ensure the mode and ownership is correct.

This properly sets things on the /dev/sd* block devices but doesn't change the /dev/mapper/eql*...

I'm thinking that we'll need to get the /dev/mapper/eql* devices set properly as well...but the install succeeded with root:disk as the owner...we'll find out after a reboot...?

------------------------------------------------

[root@dr-dbnode2 ~]# cat /etc/udev/rules.d/99-equallogic.rules

<...same contents as above with the following appended...>

ACTION=="add",SUBSYSTEM=="block",ENV{ID_VENDOR}=="EQLOGIC",ENV{ID_FS_USAGE}=="",\

RUN+="/bin/sh -c 'chmod 660 /dev/%k'"

ACTION=="add",SUBSYSTEM=="block",ENV{ID_VENDOR}=="EQLOGIC",ENV{ID_FS_USAGE}=="",\

RUN+="/bin/sh -c 'chown grid:asmadmin /dev/%k'"

------------------------------------------------

adendukuri - Is this the work around that you were mentioning?

Thanks for the response,

cs

No, That is not the work around which I am talking about

1. the best practice is to have /etc/sysconfig/oracleasm file edit and should have

ORACLEASM_SCANORDER="dm-"

# ORACLEASM_SCANEXCLUDE: Matching patterns to exclude disks from scan

ORACLEASM_SCANEXCLUDE="sd"

2. after failed install you may have to write zeroes to the cluster related ASM disks using dd command

3. it is important that you need to have dm devices and to create them equallogic engineering team came with workaround that create dm devices. This is a must have work around in place. Else you will have problem in bringing cluster nodes except the one which comes up first. this is Oracle undocumented requirement. This is must because this dm device maintains persistant name across all the nodes otherwise not possible with "sd" devices.

4. that rules.d file creates such dm devices which will have different major (253 which is constant) and minor ( variable number) for each of device.

I am positive that if you reboot the cluster nodes you may not able bring up one of the node even though you built the cluster first time successfully.

hope this helps

Hello adendukuri,

I think I see what you're saying, we do not have any /dev/dm-X devices created.  I have been searching for some details of the work around but have no success finding anything other than this post.  I'm still digging, but do you happen to have any further information on the details?

Looks like I'll be reading up on udev...

For the current installation we're running a standalone Oracle instance so it's been fine after reboots.  This installation is destined  to be a DR site setup.  However, we are shooting for consistency across our sites so it will matter from our primary Oracle RAC setup.

Thanks again,

Colby

P.S. Furhter to our conversation - it appears that my configuration may not yet be supported - see below.  I missed reading this statement in any documentation we had before I started our installation.

DELL-Adam M 08-17-2011 4:04 PM - "DISCLAIMER: An Oracle RAC configuration with ASM using the EqualLogic HITKit is currently not officially supported by Dell. There are engineering efforts ongoing to provide its availability as a supported configuration, but is unavailable as such at this time."

en.community.dell.com/.../getting-more-iops-on-your-oracle-workloads-running-on-red-hat-enterprise-linux-and-dell-equallogic-with-eql-hitkit.aspx