Oracle 11g with ASM - disk layout strategy

Dynamox,

We are not sure customers have the DataGuard protection in their environment. If so, RAC->RAC and standalone->standalone failover should be done respectively. We took care of the similar cases before.

Louis

Louis,

What about the questions i ask ..any wisdom to share?

The answer for your #1 question is ture. General speaking, different datastore can satisfy different performance requirement of tiers.

I am fully understand your #2 question paravirtulized disks mean.

The #3 question gives me the concept that fdisk or VLM will bring more workload on the OS layer. Why not directly provide more VMDK for the different partitions.

Louis

Dynamo, I have some thoughts like to share with your. My thoughts is following your questions starting with “A:”

1) do you dedicate datastores for data files and separate datastore for log files ?

A: I suggest to leverage ESM separated store to hold Oracle data files, because Oracle instance has various behaviors on those file, that means IO profile is different. General speaking, Oracle Data file like R5 or R 6 FC/SAS, Redo file like R1/R10 FC/SAS, Archived logs like R6 NL_SAS. If you would use F_Cache to improve Ora performance, you would have a look at Oracle database on EMC VNX Best Practices http://powerlink.emc.com/km/live1/en_US/Offering_Technical/White_Paper/h8242-deploying-oracle-vnx-wp.pdf   

2) do you use paravirtualized disk ?

A: This depend your favor on Oracle instance mobility or performance, paravirtualized disk / RDM would stick Ora instance on a Physical ESX server, and the performance with using paravirtualized disk / RDM would roughly 10% Oracle Performance improvement, if you have deployed F_Cache, the difference is small. 

3) do you simply present a big vmdk file and then fdisk it and "oracleasm createdisk"  or do you use LVM and then "oracleasm createdisk"

A: I prefer “a big vmdk file”, because VMFS would cross all spindles connected with ESX servers and co-exist of VMFS and ASM is a bit duplicated.   

Your thoughts? 

Eddy

paravirtual adapters

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1010398

3) yes, i present vmdk files to the host but what do you do with them in the OS itself.

Eddy,

1) we only have one pool on VNX with EFD and SAS drives in raid-5 config. I am not going to put NL_SAS in the same pool until EMC allows pool configuration with different raid types (i am not using anythign but R6 for NL-SAS)

2) what do you mean mobility ? Are you saying i can't VMotion/sVMotion paravirtual controllers ? News to me

3) the question is what you do with vmdk devices presented to the Oracle box, do you fdisk that vmdk file into multiple partitions or do you use lvm. We are going to use ASM regardless.

Dynamox,

we are looking at starting virtualizing our Oracle 11g systems. Some systems are RAC and some are standalone. 
Currently we use ASM with ASMLib to manage disk.  I am trying to get some feedback on what other folks are doing in terms of their disk strategy. Here are my questions :

I work in the team that builds,  validates and produces many of EMC's Oracle the white papers.  I have put my answers and links in line.

1) do you dedicate datastores for data files and separate datastore for log files ?

This is an older paper, and things have moved on, but it shows how we
P2V'd a database and server 
laid out ASM on VMFS for performance and compared to RDMs.
Configured ASM including using fdisk with a 64KB offset on your partition as ASM is a Volume Manager.
Let me know if you do not 
http://powerlink.emc.com/km/live1/en_US/Offering_Basics/White_Paper/h6502-virtual-infrastructure-oracle-v-max-vsphere-psg.pdf

As far as I know, EMC do not have an official position on single or multiple datastores. If you are using vmdks on VMFS then I would recommend separate data stores.

2) do you use paravirtualized disk ?
VMware recommend using the paravirtualized scsi controller we used it, without issue, for our first "Virtualized RAC" paper.
Once again it uses RDMs but also details general guidelines for   
http://www.emc.com/collateral/hardware/white-papers/h8123-oracle-rac-symmetrix-fast-vp-vsphere-wp.pdf

The following papers build out RAC on VNX
http://www.emc.com/collateral/hardware/white-papers/h8124-oracle-e-business-unified-storage-vmware-wp.pdf
https://community.emc.com/docs/DOC-12147 (It is on NFS but the rules for the underlying configuration still apply)

I believe that Eddy has already pointed you to Deploying Oracle Database Applications on EMC VNX Unified Storage
http://powerlink.emc.com/km/live1//en_US/Offering_Technical/White_Paper/h8242-deploying-oracle-vnx-wp.pdf

3) do you simply present a big vmdk file and then fdisk it and "oracleasm createdisk"  or do you use LVM and then "oracleasm createdisk"

If you are using ASM then you want multiple LUNs in your disk group for ASM to balance IO and prevent lun queuing on the guest.
As above ASM is an volume manager so I would use  fdisk with a 64KB offset on your partition. If you want to grow your disk group add another LUN.  

4) what other questions should i be asking ?

Not an exhaustive list but some things I think you should consider

What size of cluster is being used, cpu/core count? Oracle Licensing

RDM or VMFS:Effects on Backup and Replication.

Migration to/from physical: downtime and mechanisms.

Build your VM template from a fresh Guest install or VMconverter.

You don't mention OS: RedHat vs SusE. VMware have an OEM deal with and offering licensing and support for SuSE on VMware with certain bundles.
http://blogs.vmware.com/vsphere/2011/02/suse-linux-enterprise-server-for-vmware-now-available-to-many-more-vmware-vsphere-customers-.html

Currently, I have only tested RAC with RDM (no vMotion due to SCSI bus sharing must set to physical) and NFS.
However, if you want to use Oracle RAC and vMotion/DRS then you can now use vmdks on VMFS thanks to the multi-writer flag that enables sharing at the disk level, see vmware KB 1034165 and probably a future EMC whitepaper on the topic

http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&externalId=1034165&sliceId=1&docTypeID=DT_KB_1_1&dialogID=182416168&stateId=1%200%20182414901

Here are the main public EMC anding pages for virtualizing Oracle

http://www.emc.com/solutions/samples/oracle/virtualization-oracle.htm
http://www.emc.com/solutions/application-environment/oracle/oracle-virtualization-vmware.htm

Lastly, we have a large pool of Oracle SMEs out there in the field who are able to do a deep dive on this topic with you.

Hope this helps.

Allan

thanks Eddy. I don't feel right about having one to one relationship between vmdk and ASM disk. If my standard is to present 100G vmdk file to the host, i would prefer to partition it so where would be more queues in the OS and not just one big 100G device. You know what i mean ?

Dyamox,

From my perspective, The unit ASM managed is Disk recognized with OS after scanning SCSI bus, on the other words, it is LUN presented with Storage.

Here is some ASM CLI for you reference,

-- Add disks.

ALTER DISKGROUP disk_group_1 ADD DISK

'/devices/disk*3',

'/devices/disk*4';

-- Drop a disk.

ALTER DISKGROUP disk_group_1 DROP DISK diska2;

And one VMDK file is mounted with a VM guest, it is a disk, which can be managed with ASM and it would not be used after associated with a ASM DG.

In your case, a 100G VMDK attached with this Ora box, after VM stating, ASM would see a 100G disk.

Thoughts?

Eddy

Thanks Jeff, i downloaded that presentation yestarday, very helpfull. This is my issue with using RDM LUNs. Even in vSphere 5.0 the limit is still 256 LUNs. My current physical oracle boxes have a lot more than 256 LUNs. The presentation says to use use less bigger size LUNs, i don't like that strategy ..at least not on my physical boxes where i want to give my OS a lot of disk queues.

Let's forget about SRDF/Timefinder requirement, let's say my DBAs will use RMAN to clone their databases. What do you see customers do in terms of datastore sizing, vmdk sizing and LVM management once vmdk is presented to the host.

Thanks

Wow! That's alot of RDMs. Did you attend the VMworld Europe event by any chance? I ran into another customer who hit the 256 limit on RDMs and came up after my technical session there to discuss options. Thought that might be you.

At any rate, if you are using RMAN for cloning and such and are willing to live with the limitations of VMFS in terms of RAC nodes, then I would say there is certainly no compelling reason to do anything else. Certainly, the performance overhead of VMFS in a vSphere 5 context is tiny, very low single digit percent from the testing that I have seen. Not enough to worry about in other words. For a purely virtualized setup like you are describing, I have shifted my recommendation away from RDM in the direction of VMFS.

In terms of the physical layout, the following are points to keep in mind:

• Use the same methodology as a physically booted database server for laying out the storage objects onto disk. For example, online redo logs almost always belong on an isolated set of RAID 10 LUNs. If you have any significant redo activity going on, that's certainly our advice for best performance.
• The default for datafiles is RAID 5, but I have seen workloads where RAID 10 worked better. Your mileage may vary. Good old Oracle performance tuning is called for here. Look at your AWR reports, storage and OS-level stats and of course ESX stats to determine the nature of your workload and the type of storage required. Small random read and write intensive workloads call for RAID 5. Anything involving large block sequential I/O cries out for RAID 10. Because Oracle does so much I/O, this is a critical area of tuning, and is frequently wrong. There is no rocket science here, and virtualized environments are very similar to physically booted environments in most respects.
• VMFS creates the potential (along with ASM on physically booted servers) for intermixing workloads on the same storage objects. Not only can this damage performance, it can make tuning very difficult, as the workload on one database running on a completely different server may be adversely affecting performance on your server. You will spend forever trying to figure that out. Keep things very simple, and be cautious about creating very large VMFS file systems with lots of stuff on them Rather, create smaller VMFS file systems on their own discrete storage objects.
• The presentation contains a number of other recommendations for settings in VMFS, the VMs, ESX, and the storage. Be sure to apply these as appropriate.

Please let us know if you need further help. That's what we're here for.

these are no RDMs, these are physical servers that we will be trying to virtualize at some point.

Jeff Browning wrote: Small random read and write intensive workloads call for RAID 5. Anything involving large block sequential I/O cries out for RAID 10. Because Oracle does so much I/O, this is a critical area of tuning, and is frequently wrong. There is no rocket science here, and virtualized environments are very similar to physically booted environments in most respects.

i don't know about RAID 10, RAID-5 might be sufficient for redo/archive lgos. VNX does a very good job with large block sequential I/O, it will be written as full stripe write causing even less write penalty than RAID 10.

you had anythign specific in mind ? Redo log example is using 4+1R5