Oracle Licensing on VMware - no magic

Question

Bart Sjerps just posted this on Oracle Specialists. I thought it needed to be opened up here. There seems to be a lot of confusion on licensing when customers consider running Oracle databases on VMware. Part of the confusion is caused by Oracle on purpose (classic FUD) by suggesting licensing is more expensive on VMware than on physical servers. The reality couldn't be more different - I strongly believe that many customers can actually *save* on database licenses by going virtual. But to understand how to achieve this, you need to know a few things - I hope I can clear this up in a short explanation. I will keep the discussion to Oracle database licenses and ignore application/middleware etc. for now. License models Customers typically license their basic database by one out of three options: License by CPU (core) - the more CPU cores, the more licenses are needed. There is a processor core factor depending on the type of CPU and can be 0.25, 0.5, 0.75 or 1.0. License by named user - the more named users, the more licenses are needed. The amount of CPU's is not important, neither the amount of total databases. Typically one license pack per 25 users. Enterprise License - the customer negotiates a contract for the whole company and afterwards can deploy as many databases on as many servers/cpus as he wants. If a customer uses 2 or 3, then it does not matter if they run virtual or physical. But there are also no license savings possible without re-negotiating their contracts. I don't want to go as far as suggesting to customers to change their license models so we leave this as-is for now. In my experience, most enterprise customers use either cpu licensing or enterprise contracts. Some have different licensing methods for different business units. Oracle can be very creative in customer-specific contracts so expect to find a different situation for each individual customer. But let's assume CPU licensing for the sake of this discussion. Maintenance & support Users typically buy the CPU licenses  but then have to pay maintenance for the time they use the licenses. Yearly maintenance cost is about 25% of license (list price). I have no information on typical discounts. I expect customers to get at least 50% discount off the price list (but only on licenses, not on maintenance AFAIK). Database Edition and options The plain database license comes in 3 versions (for servers): Standard Edition One - Maximum 2 processors, no options allowed. Only used for testing and very small deployments Standard Edition (SE) - Maximum 4 processors, no options allowed. Only used for smaller sizes and workloads (but stay tuned) Enterprise Edition (EE) - No limitations and on top of EE, you can have many licensed features. Most customers will use this, at least for production databases On top of the basic Database license, most customers use a set of options, each requiring additional licenses per CPU. The most common options are: Real Application Clusters (RAC) - allows many servers running the same database (active-active clustering) to allow scale out performance and high availability. Real Application Clusters One Node - same but one database can only run actively on one node. For high availability only. Active Data Guard - remote replication using log shipping. Note that standard Data Guard is free, but Active Data Guard allows the standby database to be opened for read-only purposes and offers some extra features. Partitioning - allows tables to be split up in smaller chunks. Absolutely required when running large databases and no downtime can be tolerated. Eases administration work and offers some performance benefits. Real Application Testing - allows workloads to be recorded and re-played on another database to do performance and functionality testing Advanced Compression - allows database blocks to be compressed - requiring less storage and boosting performance (in most cases). Diagnostics Pack / Tuning pack - provides automated reports. Oracle AWR (Advanced Workload Reports - a performance reporting tool) is part of Tuning Pack. In my experience, nearly all customers have partitioning. Most customers have tuning/diagnostics pack. Some customers have RAC. Some customers have the other options. There are more options available but these are the most common. Many customers have 3 or more options - sometimes the options cost more than the base database license - especially if they use RAC they will have most of the other options, too. Running on a cluster If a database runs on a cluster, then Oracle assumes the database can make use of any processor in the cluster. This is independent on what kind of cluster is used (so can be MSCS, HP MCSG, Vmware, Oracle RAC, etc). This is basically the foundation for all FUD and confusion. For example, if you deploy a VMware farm (cluster) of 16 servers, and all virtual machines run all kinds of stuff (file/print, exchange, apps, etc etc) and only one tiny virtual machine in the corner, with only one virtual CPU runs a small Oracle database, you would expect only to pay for one CPU core - but Oracle's reasoning is that this tiny VM can be dynamically moved (VMotion) to all nodes in the cluster and on any processor. Therefore, all CPU's have to be fully licensed by Oracle. So in this case, running the single database on a (small) physical server would be cheaper than running on a VM in the farm. Total cost of the stack In a typical database server deployment, the cost of the database licensing is far greater than the cost of the hardware + OS licenses combined. I have no hard numbers but I assume the average DB license cost (plus options) is 10 times larger than the cost of the server + OS. So a $5,000 server would typically require $50,000 on licenses. Then because maintenance is 25% yearly, the total cost of licenses over a 3 to 5 year period is even higher - so for a 5 year TCO the total license cost might be $75,000 (assumption - could also be closer to $100,000 - and no, I didn't make a mistake with an extra zero, Oracle *really* is this expensive). Utilization It is very hard to size a typical Oracle database based application. There are no good methods or calculations to figure out how much CPU power, disk I/O and memory is needed to run a given app. So historically, project teams size their database servers for peak loads, and because they cannot predict how big the peak load is, they double the resources 'just in case'. The end result is that most database servers are way oversized in terms of CPU and memory. Most physical deployed database servers will average on about 10-15% CPU load (or less). However, they will peak to higher loads at certain times, such as monday morning when many users log in, or when month/quarter/year-end batch processing is started, etc. Then, the utilization numbers can be influenced by other tasks of the processors. Some common causes of 'artificially high' CPU loads on database servers: CPU is involved in storage mirroring (i.e. Host Level Mirroring - using Oracel ASM or a Unix volume manager) CPU is involved in file transfers over the IP network Backup (non-serverless, using CPU, Network and I/O bandwidth) Customers run the application server on the same machine driving up CPU load - This can drive up CPU load from 10% to 90% or more !! Same for Middleware and Enterprise Service Buses (Think Oracle BEA, IBM Websphere, SAP Netweaver, etc) A  bunch of monitoring/management agents burn CPU cycles (Tivoli, BMC, HP Openview, CA, etc). Each agent maybe consuming 1% but add it up and you have another 5-10% overhead. Administrators generate database dumps / exports and run their own reports, scripts and tools. They run ad-hoc queries as well that should not be on production. Poorly tuned database servers cause paging and other CPU overhead - hard to diagnose but driving up CPU and I/O significantly. Database admin tasks (table reorganizations, (re)building indexes, converting tablespaces, ...) And so on... All of these cause the processors, expensively licensed for database processing, to do other stuff. So if a server is running at 15% utilization, then the utilization caused by the database workload itself might only be 10% and the rest caused by other stuff (whether needed or not). Needless to say that Oracle likes customers to use their expensive licensed CPU's for other tasks because it forces them to buy additional CPUs sooner and therefore drive their license revenues. Isn't life great for an Oracle rep? ;-) Number of databases Most customers run many databases. For the average enterprise customer that I visit, 100+ databases is a normal number. A big global that I visited runs 3000+ Oracle databases worldwide (and this is only the scope of this specific project team). Imagine the cost of licensing all these databases on all individual servers... Why so many? Well, customers do not like to share multiple applications on one database (and often this is not even supported). So if you run SAP ERP, Oracle JD Edwards, your own banking app and a few others, they all require their own production database. For each production database, you might find an acceptance environment, test system, development server, maybe a staging area to load data into the data warehouse, maybe a firefighting environment, a standby for D/R, a training system and so on. Customers will rarely share production environments on the same server (unless virtualized or at least with workload management segregation). Sometimes they share a few databases for non-prod on a server. So for, say, 100 databases, the average customer runs between 30 and 50 (physical) servers. Power of big numbers It does not require rocket science to understand that many of these databases do not require peak performance at the same time. A development system typically drives workload during daytime (when developers are coding new application features). A data warehouse runs queries during the day and loads in the evening. For a production system it depends on the business process. An acceptance system might sit idle for weeks and then suddenly peak for a few days preparing for a new version deployment into the live production system. And so on. So what if you could share resources across databases - without influencing code levels, security, stability and so on? If that would be possible - you would not size for 'peak load times two' anymore. You would size for what you expect and assume an average utilization of, say, 70% over the whole landscape. If one database needs extra horsepower, there is enough available in the landscape. How much license cost would you save by bringing down the number of CPU's so that utilization goes up from 10% to 70%? What would be the effect on power, cooling, floor space, hardware investments, time-to-market? What would be the business advantage of not limiting production performance of a single server, by whatever was sized during initial deployment? Risk avoidance? What would be the business advantage of solving future performance issues by just adding the latest and greatest Intel server in the cluster and Vmotion the troubled database over? Wasn't this exactly why we started server virtualization in the first place about 8 years ago? And why EMC aquired VMware? Wouldn't you think the average Oracle sales rep is scared to death when his customer starts considering to run his databases on a virtual (cloud) platform? Would it make sense for him to drive his customers mad with FUD around licensing, support issues and whatever he can think of to prevent his customers going this way? Even threatening to drop all support if they continue to go in that direction? If Oracle is scared of losing license revenue, wouldn't you think there is a huge potential for savings for our customers here? The journey to the private database cloud So how should we deal with this? A few starting points Oracle supports VMware. Period. Any other claim of Oracle reps can be taken with a grain of salt (to be more specific: it's nonsense). Oracle does NOT certify VMware. Then again, Oracle does not certify anything except their own hard- and software. But IMO, support is all you need and the discussion around certification leads nowhere. Oracle might ask the customer to recreate issues on a physical server if they suspect problems with the hypervisor. Isn't it great that we can do this easily with Replication Manager? ;-) Oracle only supports Oracle RAC on VMware for one specific version (11.2.0.2). Any other version with RAC is not recommended on VMware because of support issues. Expected to change in the future. Both EMC and VMware offer additional support guarantees for customers deploying Oracle on Vmware. So where Oracle pulls back, EMC and VMware will fix any issue anyway. Performance is no longer an issue. With Vsphere 5, a single virtual machine can have 32 virtual processors, 1 TB ram and drive 1 million iops. Only the most demanding workloads would not fit in this footprint. But with customers running hundreds of databases, maybe we should start with the 95% + that DO fit and make significant savings there. By the time we're done, VMware will have Vsphere 6 and who knows what happens then. How to get around the licensing issue As I said, Oracle requires licenses for all servers in a cluster. So how do you limit the number of licenses? By deploying an Oracle-only VMware cluster. Only run Oracle databases here. No apps, no middleware, no fileservers, and try to move everything off that does not relate to database processing. No host replication, no storage mirroring, etc. Say you have a legacy environment with 10 servers, each with 16 cores, so you have 160 cores licensed with oracle EE and a bunch of options. Average CPU load is 15% but let's assume 20% to be conservative. I claim that a single VMware cluster with 3 servers each with 32 cores will easily do the job. Now we have 3 * 32 = 96 cores to be licensed. 96/160 = 0.6 = 60% so we saved 40% on licensing right away. Probably the average CPU load on the whole cluster will still be much less than 70% so we can gradually add a bunch more databases until we average out on 70%. If the old system was not running Intel x86 but SPARC, PA-RISC or POWER cpu's then the processor factor was probably 1.0 or 0.75. Intel has 0.5. So for 96 cores (Intel) you would need to pay 48 full licenses. Another 33% savings. The savings of 40% on licensing will easily justify an investment of a nice new EMC storage infrastructure with EFDs, FAST-VP and all other goodies. Do you think the customer will push us hard for a $0.01 lower GB price competing HDS or Netapp if we just saved them millions in Oracle licenses? But the story does not end here. Additional savings Let's assume the customer needed High Availability and scale-out performance and was running Oracle RAC. RAC is the most expensive licensed option and you need at least two for a two-node cluster. But VMware allows for HA (High Availaiblity clustering) as well. Using VMware HA instead of RAC, you would have to fail-over and recover the database in case of an outage - if the customer cannot tolerate this then he needs to stick with RAC (only for mission critical databases!). But most customers can live with 5 minutes of downtime in case a server CPU fails and in that case, replacing RAC with VMware HA can save them another big bunch of dollars. Let's assume that with virtualization you justified the investment in a nice EMC infrastructure with Flash drives to replace the competitive gear. Now the Oracle cluster is no longer limited by storage I/O's and can drive more workload out of the same 3 VMware servers in the cluster. But you can also replace host mirroring (where applicable). You can implement snapshot backups to get the I/O load away from the production servers. You removed the middleware and apps stuff from the database servers - reducing CPU utilization and allowing even more headroom for DB consolidation - all without buying extra licenses from Oracle. You have a customer who wants even more? What if they create TWO database clusters for VMware? One for Production (running Oracle Enterprise Edition (EE) with all the options they need) and one for Non-prod (running Oracle Standard Edition (SE) without options - good enough for test/dev and smaller, non-mission critical workloads). I bet the number of non-prod databases will be much more than prod. By removing the expensive options AND moving from Enterprise to Standard Edition, you saved another ton of money on Oracle licensing as SE is much cheaper than EE. But be aware - the devil is in the details and using Standard Edition is not for the faint-of-heart (for example, you could no longer clone a partitioned database to a SE enabled server because of the missing license and functionality). Still if the customer is keen on saving as much as possible, then this might be the final silver bullet... Do they run a huge Enterprise Data warehouse? Carefully find out if they have troubles with it and see if you can position Greenplum - saving another bag of money and speeding up their BI queries. But be careful, in an Oracle-religious shop it might backfire on you... Reality Check I had this discussion already with a few enterprise customers. And found that although the story is easy in theory, the reality is different. If a customer already has the 160 CPU licenses purchased from Oracle, then the Oracle rep will not happily give a money-back in return of the shelfware licenses. So in that case the customer can only save on maintenance and support. But having enough licenses on the shelf, he would not have to purchase any more for the next 5 to 10 years. So talk cost avoidance instead of immediate savings. And again, if they are licensed by user or have a site license, then saving on licenses will be a tough discussion. Still, the savings on power/cooling/hardware/floorspace would still be significant enough to proceed anyway. And don't forget the other benefits of private cloud of which we all know how to position: they are no different for Oracle than for other business applications. Final thought For this to work you need a customer that is willing to work with you and be open on how they negociated with Oracle, and a team of DB engineers to work with you to make it happen. If internal politics cause significant roadblocks then you will get nowhere. It's not an easy sell but the rewards can be massive. We're only just starting to figure out how to convince customers and drive this approach. Feedback welcome and let me know if you need support. Resources Landing page with Oracle 's price list: http://www.oracle.com/us/corporate/pricing/price-lists/index.html Download 'US Oracle Technology Commercial Price List' for the database license document. Read the fine print because it's not always as simple as it seems. Processor Core Factor information: http://www.oracle.com/us/corporate/contracts/processor-core-factor-table-070634.pdf

jeff_browning · Answer

So far as I know, Oracle Specialists is only available to EMC folks, one of the reasons I posted this here. Thanks for your comments. You are correct on the issue of CBT and the Data Guard license requirement, annoyingly.

jweinshe · Answer

Thanks for reposting this - is the Oracle Specialists lists where this originally was posted public or only available to EMC people?

A couple of things I'd like to draw attention to in Bart's post

1) Regarding the maximums with Oracle Standard and Enterprise Editions, instead of using the term processor, I believe much would be gained by referencing processor sockets and cores per processor. I believe Standard is limited by # of sockets, regardless of cores per socket, whereas Enterprise is licensed by core, regardless of how many cores per socket. This further expands the suitability of Standard Edition for many workloads.

2) Starting with Oracle 11gR1, Oracle actually added another scenario when Active Data Guard licensing (and hence Oracle EE) is required. If you are using a block change tracking file on a physical standby (for example, to offload the I/O associated with RMAN backups entirely from the production system and instead do backups on the standby), you must now purchase a license for Active Data Guard Option.

From http://download.oracle.com/docs/cd/E18283_01/backup.112/e10642/rcmbckba.htm

"You can only enable block change tracking at a physical standby database if a license for the Oracle Active Data Guard option is enabled."

3) Bart wrote "Oracle only supports Oracle RAC on VMware for one specific version (11.2.0.2). Any other version with RAC is not recommended on VMware because of support issues. Expected to change in the future."

It's 11.2.0.2 and above - I say this because 11.2.0.3 is now out and available and IS supported by Oracle when running Oracle RAC on VMware.

Overall, a very good post and highlights many of the things I've heard Jeff say over the years.

BartS · Answer

Hi Jay,

Thanks for your feedback, I wasn't aware of the BCT issue. Good to know. Same for the RAC support - I thought Oracle made an exception for support probably because one large customer had pushed them hard for this.

Regarding the processors vs cores vs sockets issue: very true and I admit I used information in somewhat older sources where they just discussed processors.

The fact that Oracle seems to limit sockets on SE and counts cores when using EE is adding to the confusion. Still, the overall message does not change. On the contrary, if it is true that the limit is by socket then Oracle SE can drive even more workload and all the more reason for customers to consider deploying their non-mission critical stuff on a separate VMware farm which is SE (not EE) licensed.

But you probably run into some operational issues there like I said. For example you could not restore a database that uses Advanced Compression (licensed on EE) to a server that is SE licensed (any ideas on how to deal with such stuff are appreciated!)

Any feedback on how this works out with customers is also welcome. We're just starting and I really want to know if and how customers can apply this to really reduce cost.

BTW the reason that I didn't post this on EO@EMC myself is that it is a bit agressive with respect to our Oracle partnership. But then - they start to compete with us too, so why not...

Thanks Jeff for putting this up here and boldly go where I didn't dare going before :-)

You triggered me to post it on my own blog, too...

Best regards,

Bart

jweinshe · Answer

Luckily I'm just an EMC and Oracle (and VMware) customer so besides giving my Oracle sales reps heartburn I'm generally not too worried with what I say as long as I stick to facts.

First and foremost I'm an Oracle E-Business Suite DBA. Given that Oracle requires Enterprise Edition for Oracle E-Business Suite and I believe that holds true for JD Edwards and Peoplesoft, when I come to a customer they're already licensed for X processor licenses of EE. I then work to leverage those EE licenses in a VMware environment by creating a VMware "island" of 2 or more vSphere hosts and limiting all the Oracle DBs to that vCenter cluster of Oracle DB hosts. That way I've met Oracle's licensing requirements, I'm able to start consolidating Oracle workloads and avoid purchasing more Oracle licenses and I don't have to worry about the EE vs Standard Edition issues.

On my wish list is for Oracle to publically consider VM to host affinity rules in vSphere 4.1 and above a valid form of segmenting Oracle workloads from the main vSphere cluster and also to consider vSphere at least the same sort of hard partitioning as it considers OracleVM.

Regarding the Advanced Compression restore issue, couple of things

1) If you're doing that restore due to a DR type scenario, the Oracle licensing allows for a 10 days per calendar year grace period. Basically, they don't want you worrying about buying new licenses while you're in a disaster situation.

2) One thing customers who buy Advanced Compression and other EE add ons need to consider is if they're in a virtualized environment, they'll actually get even more benefit out of those licenses. In a physical environment they've licensed Advanced Compression to probably one physical host running one DB. By licensing Advanced Compression for a vSphere host and running the DB on that host (heck, with free ESXi even), they can now run multiple Oracle VMs and leverage Advanced Compression across all of them. This is one of the reasons why I see virtualization even for very small Oracle environments to make good sense.

I suspect there's probably many Oracle customers out there using features like Advanced Compression without realizing the features they're using require an additional license. For example in Advanced Compression, you need to buy that license if you're using SecureLobs with Compression or Deduplication, table level compression, or even if you use certain levels of RMAN compression. It's very easy to accidentially start using a feature requiring a license without knowing it.

Final bit of babble (sorry!), I'm a huge fan of customers virtualizing Oracle EE under VMware and licensing Advanced Compression... not only for the reasons I listed above, but as that customer's data grows, when they look at buying EMC's FAST Cache or such, they're now getting even more benefit since more data is stored in each block. Of course, as Darryl rightly pointed out in another thread, in very high transactional environments you may NOT want to do that on certain tables because of all the transactions against the table.

BartS · Answer

Jay,

I was not thinking about SE not being licensed for partitioning, advanced compression and the like, but more about the functionality simply not being available in the binaries. Thus, apart from being licensed correctly or not, it will simply not work (or at least this is my expectation).

In other words, if you run a prod database with partitioning or advanced compression, then the blocks on disk are compressed. If you clone it using EMC replication manager (clones or snaps) then the copy will have compressed blocks (and/or partitioned tables).

Now if you try to mount this database clone image on a server with SE binaries, it will probably refuse because it lacks the binary code to understand these data structures.

My idea of using SE was for test/acceptance/training/staging etc. where you would use EE for the real production environment. But then cloning from prod to the other systems will probably cause some issues.

btw good info, thanks!

jeff_browning · Answer

Jay:Interesting point on the use of Advanced Compression with VMware. I had not thought of that! We really need to look at the use of EE add-ons in a virtualized context, and evaluate the potential savings in terms of license costs for the add-ons themselves.In terms of your approach of using a physically isolated Oracle ESX cluster, that is certainly the most conservative position, and forces Oracle to acknowledge that your VMs are not creating licensing havoc for you. Check here if you want to see an analysis that Sam Lucido and I performed on this issue. (Pay special attention to the section on Licensing obviously. ) There is certainly some indication from the field that Oracle is granting special dispensation to good customers for the use of host affinity on this stuff. I have said more often than I should (remember that I am an attorney), that litigation is likely on this eventually. (It has already happened in Canada.) If you simply read the Oracle license agreement, as well as the technology price list, it is clear that Oracle licensing only attaches on server sockets where the code actually runs. It is difficult to reconcile Oracle's position on this with their own licensing documents. Clearly, you should be able to provide Oracle with convincing evidence of actual usage (in the form of VMotion logs for example), and avoid any licensing surprises.Glad to hear that you are a happy VMware customer with EBS. We love it!Regards,Jeff

jweinshe · Answer

Just curious - how did the litigation in Canada turn out?

dba_hba · Answer

Nice to see VMware openly address this and explain their position. http://www.vmware.com/files/pdf/techpaper/vmw-understanding-oracle-certification-supportlicensing-environments.pdf Allan

Oracle

Was this post helpful?