In Chinese philosophy, Yin and Yang describe how opposing yet complementary forces are interrelated—one cannot exist without the other. If Yin represents a contracting force, Yang balances it with expanding energy. Two halves that complete each other yet change continuously to seek a new balance.
When I forecasted 2016 data management predictions for the latest issue of CMS Wire, I realized this is the Yin and Yang year of the lifecycle of data. Major parts of the data creation market are beginning to ebb, which will cause others to flow. For starters, Hadoop meets its Yin force this year after five-to-seven years in a Yang state. Everyone knows the current, crowded market can’t sustain hundreds of small Hadoop vendors. It’s already beginning to contract due to its own tremendous momentum and prolific innovation.
Companies have been gluttons, gobbling up each new Hadoop flavor of the month without much regard for what will happen when it’s time to pick two from the 97 flavors tried. Hadoop platform consolidation will create an undercurrent of inside-the-firewall data complexity that will echo loudly as companies struggle to determine which data gets stored, tracked, integrated, used or simply abandoned. What finance will want in its on-premises solution, for instance, will differ widely from marketing’s cloud-sharing data environment.
As organizations consolidate to fewer platforms and therefore a combined view of all the data, someone has to physically reconcile all the different data sources. As the Yin of Hadoop connects with the Yang of data complexity, data from countless “Big Data or Hadoopp science projects” will be left behind because no one knows what to do with all those flavors and forms of content.
Navigating and integrating all these data sources will be the next big challenge as parts of the market contract. Companies with just a handful of Hadoop implementations will fare better during the Yang of data complexity as they’ll have far fewer sources to reconcile. It’s not unusual, however, to find hundreds of data-driven projects with teams of developers who spent years experimenting and exploring new and creative ways to find business answers. These companies are going to need help, and I know just where they can find it.
In this year of Yin and Yang, I’m hopeful we’ll find an opportunity for harmony between two previously opposed camps: shadow IT and traditional IT. For years, we’ve heard and seen firsthand how shadow IT emerged out of necessity to drive innovations outside the walls of the IT department. Greater accord between these factions now is needed as increasing data complexity requires more traditional IT tools and skills, including database management, modeling and analytics.
Traditional IT has the ways—and hopefully, can get the means—to perform data and business-related modeling, which became a lost art during the halcyon Hadoop heyday. Most developers had no reason to do anything other than back-of-the-envelope modeling then as they could quickly and more easily standup another cluster to look at a new variable. If it didn’t work, they just halted the experiment and tried something else.
This approach won’t work once platform consolidation begins. The renaissance of the art of modeling and its corresponding best practices, allows companies to understand, track, and deliver the best data sources and information outcomes for fueling both decision making and business analytics. Additionally, companies will be forced to stand by their database management commitments or risk creating islands of disjoined disparate data sources. Developers will have to build models—and they’ll likely need help from their traditional IT counterparts. Likewise, the shadow IT camp can benefit from the advantage of best practices and data management solutions from the IT department’s toolkit.
Striking the right balance between technology and business stakeholders is key to making the most of this year of ebbs and flows. It starts and ends with following best practices. Figure out which business use cases best benefit from data management and then do your homework, assess ROI and be realistic about how much your company’s culture can absorb. Put stock and value in meaningful analytics and push out as much actionable insight as possible to the ultimate end-users. Align on-premises and hosted environments as needed to deliver the best of both worlds.
That’s the goal of this Yin and Yang year. Follow the market’s ebbs and flows, then continually seek ways to equalize and harmonize people, processes and technologies. In doing so, you’ll find the whole is much greater than the sum of its parts.
How do you plan to achieve data management equilibrium this year? Connect with me on Twitter at @joschloss to share your thoughts.