Running IT for an IT company has some benefits, but it also comes with unique challenges. At EMC, one challenge our IT people at EMC like to joke about is the criticism they regularly get from 40,000 “armchair CIOs.”
But here’s the thing. Some of those critics have really good ideas, such as tools and practices learned from working closely with our customers and partners. Unfortunately, IT has had no systematic way to gather and use that find of information. It’s especially hard when different groups within the company offer conflicting advice.
That changed last week when EMC IT hosted its first internal “IT Technology Advisory Board,” a half-day interactive discussion with top technical leaders from across EMC’s businesses. Senior IT leaders shared their progress and plans for building EMC’s own Private Cloud, and sought questions, opinions and advice from the company’s thought leaders. Half-way through the meeting, everyone switched roles. Product teams shared their thinking and strategy for enabling and managing Private Clouds, and IT asked questions and offered suggestions as would a customer.
You’ve probably guessed by now, correctly, that a few people were unable to resist the opportunity to do a little IT bashing. After all, many had been unable to find anyone beyond the IT help desk to resolve their complaints, and were people in the room that could actually do something about it. The IT folks handled most of the flare-ups pretty well. More important, IT admitted how much they struggle to understand the true scale and priority of a particular issue across EMC—especially when groups within the company disagree.
That’s the second big reason IT has for holding these advisory-board meetings. Let’s gather EMC’s top technical talent, and their divergent perspectives and priorities, into one room and get them talking with each other. Consensus can be a beautiful thing. Even if getting there isn’t always so. As you’d expect, there was a wide spectrum of opinions on IT’s Private Cloud strategy and plans, with skeptics often triggering the more-interesting discussions.
There were some pleasant surprises, with most coming from audience Q&A. For example, IT was asked if they were documenting decisions and assumptions somewhere to be revisited in the future. At first the answer sounded evasive: “we’re documenting what we call ‘limitations’ that we want to re-examine later. But we’re not documenting a lot of operational assumptions.” But how can [EMC’s technology leaders] help you if we don’t understand your operating assumptions?
The surprise answer: “Because cloud changes everything. Most of the assumptions we’ve had for years need to change.” To emphasize the point, our CIO was blunt: “We can change what we need to change.” Wow.
Our Oracle environment, which I briefly describe in an earlier post, unsurprisingly triggered a lot of discussion. Like critical systems in most corporate IT shops, it’s built today to handle peak workloads. If we had the environment fully virtualized, with the right monitoring and automation in place, IT could build to normal loads instead, and use additional internal resources during peaks. We’re not going to get there all at once, however. We’re already most of the way there, but some applications will take longer. Especially back-end stuff currently running on large scale-up server iron. But that doesn’t mean we’re not working on it as aggressively as possible.
Wait a minute. IT has already been virtualizing our company’s mission-critical apps? Including Oracle? Isn’t that the polar opposite of the approach we advise most customers to take? Yes, it is. But unlike most IT shops, we wanted to learn as much as we can as fast as we can, and that comes from taking on the toughest use cases.
For example, when does it make sense to use external resources in a Private Cloud? By virtualizing (or trying to virtualize) mission-critical business and database applications, IT learned that scale-out friendly workloads benefit most from private-cloud flexibility. That is, until a single Intel based server can handle the kind of scale-up loads that currently have their homes on 120+ core big-iron.
I was disappointed, though, to see some people on both sides apparently missing the whole point of the exercise. I should have expected it, of course. Years of living through a stereotypically dysfunctional company IT-user relationship makes for habits that die pretty hard.
What about people leading the discussions? This was all about seeking opinions and advice, so presenters were at their collaborative best, right? Actually, most did pretty well and engaged people in healthy, productive dialog. Many of them rarely speak in front of large groups, never mind lead animated discussions, so that’s saying a lot.
Unfortunately, a few presenters—on both sides—apparently had no idea what their audience wanted or needed. A couple IT presenters seemed to be talking to other IT people, deflecting audience challenges by offering to “take the discussion offline.” But the hardest to sit through was presented by one of our product-development organizations. It was a standard product-portfolio pitch, with barely a passing reference to Private Cloud—or, for that matter, anything specific to EMC IT’s needs or interests. Worse, it provided no insight on that product team’s strategy or plans beyond, in essence, “we plan to merge several small blobs into a few big ones.” The good news is that this kind of thing was the exception and not the rule.
This kind of discussion between our internal IT team and our product/service thought leaders is a first for EMC. The inaugural meeting brought surprises, good and bad, and substantive discussions. There were some disappointments, and some disagreements. There were also times when everyone saw eye-to-eye, such as when one product architect observed: “You can’t fix a business process problem with technology.” To which IT replied, “Bingo!” And we’re convinced that close collaboration between product development and IT operations as vital to a successful journey to the Private Cloud—for EMC and our customers. We have a long way to go, but I think this as a good start.
How many of you have this kind of interaction between your company’s IT and users? What’s worked well, and not so well, for you? I look forward to your thoughtful comments.