Oracle Licensing on VMware - Reprise

I agree with Dave.  DRS host affinity is a clear, auditable method of partitioning your ESX cluster.  VMware has officially come out in support of this.  Read their statement here:  Understanding Oracle Certification, Support and Licensing for VMware Environments

In terms of auditing the real question boils down to, "Does Oracle recognize the extensive APIs and vmware.log file(s) as provided by VMware vSphere?” It’s been my experience (somewhat dated) that  Oracle bases licensing audits, in the most part, on the database audit tables. I did a quick Google search trying to answer the question “Does Oracle recognizes VMware API & logs” as a validated means of auditing?” Unfortunately, I didn’t find any useful information one way or the other!

So let me ask the million dollar question: Is there a public statement by Oracle recognizing any third party APIs or logs as a means for auditing?

On another note I don’t see the advantages of using DRS, host affinity, in are large cluster as compared with a narrow Oracle centric cluster with the exception of some minor uptick in management. A narrow Oracle centric cluster has these benefits:

• Eliminates the concern of using DRS, host affinity, in a large cluster
• Simplifies management: no worries that you have the proper rules wrapped around DRS, host affinity
• Narrow Oracle centric cluster have the possibility of better deterministic performance
  • I understand this is debatable but, an Oracle centric cluster simplifies virtualization tuning, best practices and testing only for Oracle
• Isolation has value

What do you think?

True, for large organizations, it makes perfect sense to have dedicated clusters.  For Organizations that have less than 20 databases, it eases the burden of having many ESX clusters.

You misunderstand the question. I am not suggesting that DRS host affinity would work as a form of host partitioning. That has to do with what CPUs have to be licensed on a given server, which is different from my question.

Rather, I am suggesting that entire servers be licensed, but that VMware auditing logs be used to determine on what servers Oracle software has run.  Please let me know if this distinction is clear.

I'm not sure that Oracle will recognize DRS host affinity as a valid means of partitioning Oracle.  This Oracle document, which provides Oracle's guidance on partitioning, would seem to clearly state that DRS Host Affinity, VMware CPU Pinning, or anything else is not a valid a means to partition the server for the purposes of Oracle licensing.   Like Sam says, during an audit, they'll use Oracle audit logs to determine where Oracle is running.

For my understanding, how are the Oracle auditing tools capable of finding out on which physical server or CPU the database has been running? Consider Cisco UCS or probably any other blade system where the physical servers are stateless (i.e. no CPU id, no fixed MAC address etc).

If I run Oracle on a UCS blade, then shut it down (including the OS), then via UCS tooling make the same OS run on a different (but equally configured) blade, then I guess Oracle has no means of finding out about the move.

Equally, in a VMware landscape, if you have, say, 4 ESX servers and you use host affinity to run Oracle only on host 1 and 2, but you have performance issues on host 1 for a few days and decide to change the affinity settings to allow the same Oracle DB vm's to run on host 3 for a while, then after a few days you change the settings back... then how is the Oracle tooling capable of detecting the physical underlying hardware change?

I bet the only way to find out is using the VMware audit files and, if they are plain ASCII files they can be tampered with. So probably they are no good as licencing audit trail !?

Jeff, I understand - but I think I'm agreeing with you: DRS/Host Affinity would likely not be recognized by Oracle as a valid means of partitioning your HA cluster to limit licensing.  I've heard Dave at HOB contend the same thing you've heard, which is essentially: "DRS/Host Affinity SHOULD be fine" and as a purely theoretical and ethical point, I would agree: it really should be fine.  The VMware logs should be recognized as a valid means of determining where Oracle ran and for how long.  However, I'm not sure that's something I would hang my hat on if I were a CIO - what with such big $$'s at stake.

My methodology would use the VMware logs, not the Oracle logs. ESX maintains logs However, you make an interesting point. I don't think the Oracle logs would contain this. When an instance starts up, the only thing that is contained in the logs is the fully qualified domain name, which would not change in any event. Do you know of a way that Oracle logs could be made to work in this manner? That might be an interesting challenge.

Jeff,

You hit upon Oracle's point: They have no method to audit host affinity, cpu affinity or similar therefor they don't support or have no statement on the technologies (Licensing Lmbo). I've been around some Oracle audits, while consulting for Oracle, (more than a few years ago), and its been my experience that audits and auditors are very agreesive. Narrow Oracle centric clusters are a strategy wraps more control around the Oracle infrastructure and doesn’t limit the customer in their use of Oracle with VMware. Win, Win.   

To everyone else on this thread: please jump in, legal experience or not, as this discussion is all about share different viewpoints.

OK Sam & Jeff, here more thoughts from my side.

I guess you have to separate two things:

-     What is really allowed within the limits of licencing contracts (and the law)

-     What can be reliably measured using available tools (logs, audit trails, etc)

I don't know the contents of licence agreements but I guess it says something like that you have to licence Oracle on a specific processor (i.e. a piece of silicon in a black plastic case with a serial number printed on it).

The fact that I can ditch audit trails when moving (i.e. vmotion) a running database from a licenced physical processor to another server with an unlicenced processor and back without being detected, means I am violating the policy - it's just that they can't catch me.

(but I risk that the auditors are smarter than me and find out a way to figure it out anyway and use it to legally send me a huge bill)

Which suggests that using VMware affinity to keep Oracle DB within certain physical CPU/server boundaries is playing (strip) poker with licence policies and the hostile auditors.

They might also be smart enough to claim that (for the same reason) even if you might not have really moved the database anytime to unlicenced CPUs, the technical possibility was there all along and therefore we owe them licence fees anyway.

I would not recommend to customers to do it this way...

All,

Let me start by bringing all of you into the courtroom. There are three issues that you will observe me as counsel provide to the jury as part of my allowed instruction (I am not an attorney in real life).

I offer the definitions in this paragraph only to make this post as self-sufficient as possible and not with intent to talk down to anyone.  Case law refers to precedents established in other court cases. Case law can supplement statutory law enacted by legislative bodies and can stand in lieu of statutory law when applicable statutory law does not exist.

Back to the courtroom:

1. You cannot prove a negative.
2. Just about every contract that any of our organizations enters into or sees contains a clause that essentially states that the only terms that are binding have been reduced to the fully-executed instrument itself as well as other explicitly-referenced written material, and all verbal agreements and/or representations are null and void. The language usually allows for the obvious possibility of written, fully executed amendment.
3. As often as original documents are unavailable, courts in all fifty states recognize case law called "best available image.” This refers to copies which become recognized as legally binding as if they were the original. “Best available image” case law has been universally recognized since 1989 when as the Medical Records Systems Analyst for 23 hospitals and 7 clinics in 3 states, I had responsibility to begin architecture for imaging of medical records.

I offer the following as the exclusive binding source documentation for this post:

• The Oracle License and Services Agreement (OLSA)
• Oracle’s published Software Investment Guide
• Oracle’s published Licensing Data Recovery Environments – an extract of the Software Investment Guide

The OLSA is bi-laterally executed between Oracle and the customer. I have seen many of them as you can imagine in my role as the lead of House of Brick’s Oracle Enterprise License Assessment business line. The OLSA always contains:

• A reference to other unnamed published documentation, which given the nature of the reference, refers to the Licensing Data Recovery Environments extract of the Software Investment Guide as well as the current Oracle Support policy. Because of the nature of these external OLSA references, I make it my business to occasionally archive current versions of the Software Investment Guide and Licensing Data Recovery Environments document such that any given OLSA can be clearly tied to the external terms in effect as of the OLSA’s execution.
• A Non-Disclosure Agreement as to the OLSA’s terms. As Oracle publishes template OLSA copies, a reasonable and customary interpretation of the NDA’s coverage is pricing (discounts) as well as any other non-standard qualitative terms that the customer may have negotiated with Oracle. We have initiated for customers and seen Oracle approve non-standard qualitative OLSA terms. However, we have never bothered to initiate sub-cluster licensing terms as there is no need to do so.

The sum and detail of the stock published OLSA and its external Software Investment Guide for at least the last decade and to-date with respect to processor-based licensing is two-fold:

• Customers must license all physical cores or sockets on a host where Oracle executables are “installed and/or running” (with physical cores factored per Oracle’s published Core Factor Table, and potentially subject to the so-called 10-day rule [whose terms became more restrictive sometime during 2007]). Notice the tense. Oracle customers are contractually obligated for licensing the physical servers where the Oracle executables are and have been, not where they might go. To imply otherwise without explicit contractual inducement would not be unlike concluding that I am legally obligated to purchase transportation to or obtain a visa for destinations that I clearly have the capability of visiting but where I have neither ever been nor yet made a determination to even visit.
• Furthermore, customers must pay a license to cover the use of remote mirroring at the storage unit or shared disk array layer to transmit Oracle executables to a SAN whether or not that set of Oracle executables is “installed and/or running” on any physical host connected to the SAN.

When discussing Oracle licensing, there are two common areas of confusion.

1. Oracle itself defines both of the terms “Hard Partitioning” and “Soft Partitioning” in its Partitioning document, published since 2002. The document makes clear that both “Hard Partitioning” and “Soft Partitioning” have to do with carving up physical cores/CPUs within a single physical server. It is Oracle’s right to define the terms as they relate to the OLSA. I frequently see Oracle reps (knowingly?), customers and others confuse these terms over into a discussion of licensing physical servers within a sub-cluster. As often as unintentional term confusion occurs, I believe it important for authors to make corrective edits after the confusion has been pointed out, especially on a topic with such scaled financial ramifications.
2. To say that Oracle has not made a statement supporting vSphere DRS Host Affinity Rules (as has been asserted in this thread) is beside the point. That is clearly in the realm of proving a negative. That DRS Host Affinity Rules may or may not be used as a mechanism to assure that Oracle executables are not “installed and/or running” on unlicensed physical hosts is customer-discretionary and irrelevant to the OLSA’s binding terms. I might add that methods existed to restrict VM movement to a sub-cluster of physical hosts long before the vSphere 4.1 advent of DRS Host Affinity Rules.

I tried to force both clarifications into Gartner Group’s Chris Wolf’s November 10, 2010 blog thread via comment posting.  I feel that I failed to get Mr. Wolf to directly address my point before he elected to close the thread.  I feel that I failed both verbally and in writing during the pre-publication interview process to preventively insert the clarifications into TechTarget’s Beth Pariseau’s May 4, 2011 article titled “Oracle licensing for vSphere 4.1 irks VMware pros.” Furthermore, the TechTarget article’s title unfortunately implies that Oracle may have licensing terms specific to vSphere version 4.1, which it does not, or licensing terms specific to VMware technologies in any way, which it does not. Thankfully the VMware Corp. November 2011 whitepaper (to which I contributed) “Understanding Oracle Certification, Support and Licensing for VMware Environments” got it right. I can confirm that the entire white paper received all the multi-disciplinary pre-publication review that you would imagine appropriate for a piece of its importance and ramifications.

House of Brick is occasionally invited by customers and/or partners to get on the phone with Oracle to deal with Oracle licensing concerns that Oracle has verbally conveyed to the customer.  We always welcome these opportunities.  The most recent call was a couple months ago. An Oracle Corp. licensing specialists was on the call. The impetus for the call was that Oracle had previously told the customer that all nodes in a physical cluster must be licensed, not just those where Oracle executables are “installed and/or running.” We asked the Oracle licensing specialist about the statement.  He informed us that it was Oracle’s policy.  We asked where the policy was written.  He replied that it is an unwritten policy. At this point, we reminded him of the binding OLSA language whereby all verbal conveyances and understandings were voided at the execution of the document leaving only the printed OLSA terms in effect. That was the end of the discussion. The customer left the call entirely and appropriately satisfied as to their full legal compliance with their proposed sub-cluster licensing scenario. This vignette is by no means an isolated incident. The outcome is always predictable and consistent.

A previous post on this thread states ‘I've heard Dave at HOB contend the same thing you've heard, which is essentially: "DRS/Host Affinity SHOULD be fine"...’  The quote could be interpreted inappropriately out of context of my consistent, persistent statements. What I have always said is that any method (manual or automatic) to restrict movement of Oracle executables to a licensed sub-cluster of physical hosts is fully OLSA-compliant and legally defensible. At the release of vSphere 4.1, I added the statement that DRS Host Affinity Rules happens to be one, the newest, and the easiest of multiple available methods of restricting Oracle executables’ movement.

There are comments on this thread that I believe fall under the umbrella of attempting to prove negatives. Oracle has no legal obligation to go on paper with sub-cluster licensing clarifications specific to vSphere. In my view, Oracle has no financial incentive to do so.  Stop looking for that to happen.

Occasionally people actually insist that I produce an Oracle-authored document that says Oracle will not ding them for not licensing all nodes in a physical cluster. That strikes me like asking me to provide an Oracle-authored document that says Oracle will not make a claim against me for choosing to drive a Ford as opposed to a GM. Not only is it asking me to prove a negative which I cannot do, but it delves into an area that is none of Oracle’s business since it is not listed in my OLSA obligations.

I see no need for Oracle to comment on their recognition of the validity of vSphere logs to prove vSphere VM movement. vSphere logs may well be the best available mechanism to document Oracle executables’ travels.  Although vSphere logs may not be identical to “best available image” case law, “best available image” is instructive to the audit log/verification concept under discussion. As of November, 2010, the majority of the world’s servers are virtual. ~85% of those virtual machines are VMware. That is tenured technology and no one is questioning the de facto authority of those logs. Oracle has the OLSA-granted right to present themselves at any time without notice for the compliance inspection of their customers’ physical hardware. Oracle has options to audit the current and historical location of where their executables are “installed and/or running.” That those options may rely on mechanisms not provided by Oracle is beside the point.

As for the question on the thread as to whether vSphere logs could be maliciously manipulated, security industry experts have said that eighty percent of security breaches come from within the firewall. I see a corollary to that statistic. Oracle has plenty of financially incented prospective friends inside its customer organizations. It is sad that human integrity is such that posting of large rewards has become necessary to incent individuals to turn in their employers for knowing theft of software. To me, OLSA obligations are like the U.S. tax code. I believe in paying every penny I owe. However, beyond that, it is my discretion to who or what I donate and in what amount. I have no patience with individuals or entities that premeditate the creation of OLSA compliance issues. I similarly have no patience with the knowing spreading of FUD by some professionals in what could be construed as extortion of funds beyond customers’ executed contractual obligations. I will continue to vigorously promote and defend the legal rights of both software vendors and their customers even if that means I induce accelerated hair loss through rapid, frequent hat swapping.

What Oracle verbally represents on this or any other contractual topic for that matter is of absolutely no relevance whatsoever. Look to the OLSA and extant external referenced documentation as of the OLSA’s execution date. Should you still be dissatisfied, look to court history of judgments against customers attempting to exercise their sub-cluster licensing rights. Let me save you some time. Relevant court history does not exist. There is a simple reason: the terms of your executed OLSA are clear, sufficient, binding. The terms explicitly invalidate any and all verbal representations.

Given what I have presented here, I see no legal, risk management, or moral need for and am not inclined to encourage and/or endorse air gap clusters strictly for Oracle licensing purposes. I am concerned that such talk may scare parties away from investigating, understanding, and asserting their legal rights. Furthermore, in my observation, such talk gets quickly quoted out of context of the original motive. I believe such talk has a similar long-term effect on the market as paying ransom for hostages has on future hostage taking. The SMB market has every contractual right to leverage sub-cluster licensing to facilitate crossing the chasm over to Business Critical Application virtualization. That larger enterprises may find it convenient to use air-gap cluster separation of their Oracle workloads is great, but to do so has absolutely no bearing whatsoever on their legally binding contractual obligations to Oracle Corporation.

Some on the thread have suggested this or that course of action to minimize risk with Oracle. Occasionally people express concern to me that Oracle has far deeper legal pockets than they do. I believe those that make these points do not understand the power of their position. I cannot think of one reason that Oracle could possibly want this thing to go to court and indeed I believe Oracle has everything to lose minimally moving forward if and when it does. Once such a judgment was out, it would cause an immediate slow-down on non-contractual cash donations that some of Oracle’s customers have felt induced to make.

Interesting point, and very interesting and useful discussion. I have a fulltime job assisting Oracle clients independently  with licensing issues and would like to share my 5 cents with you and look forward to hear your (dis)agreements.

1. I don't care what's in the SIG. I don't care what's in the Partitioning whitepaper. I don't care what's in the 'licensing data recovery' whitepapers. Look at the footnote and understand why. It serves no contractual purpose at all, period.

2. Oracle does take stance toward clients who defend themselves against the practice of 'licensing the entire cluster'. I've adviesed multiple clients to argue their case with Oracle Legal, in writing. Not one time Oracle Legal replied, only LMS (specifically the regional LMS manager). That must be because the Legal department don't want to burn their fingers on this item. LMS will always answer this (freely translated):

"Our agreement is defined in the OLSA. Such a license agreement only describes under what conditions you are allowed to use Oracle software within your organization. If a certain usage type has not been described in the agreement, it does not mean that this type of usage is therefore allowed".

I've seen multiple lawyers laugh at this statement. The OLSA is a restricted license, and all restrictions are in place. And if they were not, Oracle would certainly not be allowed to randomly come up with new restrictions as they see fit.  "Where the Programs are installed and/or running" covers it all. But Oracle tries to get the money, and use LMS as a vehicle. Why? Because LMS is funded by Sales and have a revenue target.....simple as that.

In any situation I've run into, I will tell the client not to pay for the whole cluster if the software is not 'installed and/or running' on every ESX. DRS or not. We have a saying here "wie eist bewijst", or "who demands, must deliver the proof". so if Oracle would demand licenses for the entire cluster, oracle must proof that software is installed and/or running on every ESX....it's not for the client to proof it is NOT running. DRS or not.

It's not the client's fault that Oracle is not exhaustive in their description of OLSA interpretations. Microsoft does it much better, and has for each virtualisation example contractual binding scenario's written out in the binding Product Use Rights documents.

In every case I've run into, Oracle eventually backs out. Unfortunately many un-informed client's don't dare to push back and simply pull their wallet and pay. If any such case would make it to court, upon the client's initiative, Oracle would lose a lot of money as hundreds if not thousands of clients would want their money back....and rightfully so. With the agressiveness Oracle is auditing in this region, it's only a matter of time until a client will sue them for attempted theft. It would be an interesting case to follow.

Now if you compare the SIG with any signed OLSA - customized or not - you'll find that many things could be interpreted differently than they are being interpreted by Oracle's auditors. Virtualization is just one out of many issues.

LVP:

I would suggest Daniel Hasselick with License Consultants and Jason Keogh with iQuate as the two folks in this area.

Regards,

Jeff

Jeff,

Great resources.

LVP may also want to review this blog entry from Michael Webster giving the “right questions to ask” and knowing your Oracle obligations when the Oracle Sales Team (think audit) questions you:

http://longwhiteclouds.com/2012/07/21/fight-the-fud-oracle-licensing-and-support-on-vmware-vsphere/