The Four “W”s of Successful Big Data Projects

It seems everyone, including me, is talking about Big Data project pitfalls. I recently authored an article for CMSWire on the topic, describing the roadblocks organizations typically face when embarking on a Big Data initiative.

When I talk to customers, they complain that an overall lack of organizational alignment, access and knowledge are causing Big Data projects to fail. Others point to massive confusion that the many vendors are generating by claiming, “we do Big Data” and “you need to do Big Data this way.”

In “10 Things You Need to Know About Big Data Now,” Joyce Wells, editor of Database Trends and Applications, remind us that Big Data is not new, and neither are the most commonly accepted variables to it: volume, variety and velocity. In my mind, what’s new is the tendency to get so hung up on Big Data technologies instead of the result the technologies should enable.

Remember: Big Data is a means to an end—not the end itself. That’s why I tell people to focus on the four “Ws”— Why, What, Who and When. It’s a back-to-basics approach that works.

Start with “why” you’re embarking on the project. Simply wanting to analyze Big Data because you think that’s what you’re “supposed” to do isn’t enough. You need a business objective, such as a desire to know where your next new market should be, how to make a product more desirable to current markets or how to reduce operational costs in a particular area.

Then start thinking about “what” you need to learn in order to answer the business questions. “What” and “why” should always go hand-in-hand, but when it comes to determining what you want to know, get specific. What can you discover from your data (all of your data, not just “Big Data”) to help get the answers? You may very well already have the data you need, so collecting more only will slow a project.

For one national retailer we worked with, it was the desire to learn more about what specific customer behaviors drove onsite sales transactions. Notice the level of specificity. There could have been any number of good reasons “why” they were doing this project, but when it came to “what” they wanted to know, it was focused. This enabled them to uncover a strong correlation between dressing-room visits and actual sales transactions. This knowledge led to an increased focus on encouraging shoppers to try on merchandise, which in turn generated a 50 percent increase in sales.

Aligning “what” and “why” is hard, especially because business requirements and priorities can change faster than IT can address them. There are many factors outside the organization that cause businesses to reshape themselves—competition, macroeconomics, consumer behavior. And, they can all happen at once or sporadically, which is where the “when” comes into play.

It can take considerable time to build technology solutions to address business problems. Unfortunately, it takes very little time to change your mind and reprioritize the areas of business that have the greatest potential to deliver value. That’s why I advise against tackling something so broad that any disruptive market change could derail the project. Rather, pick something smaller that creates value and then build on the success of what you’ve learned to explore more.

In working with a Midwest university, we addressed a persistent and rather consistent problem regarding student retention. For every percentage point of increased retention, the college could return $1 million to its bottom line. So, the analysis focused on which factors most contributed to dropouts and what could be done to raise retention. Very focused.

Lastly, to get the right answers, determine “who” has access to corporate data in order to look for relationships, patterns and correlations worthy of further analysis. Resist the urge to make the team too broad and ensure they have top-down endorsements to access what they need. Sure, it’s great to have people who understand Hadoop and machine learning, but don’t put your learning projects on hold until you hire a data scientist. Your current team understands the business. When you combine that understanding with unlimited access, it can produce more powerful and faster returns than one magical new hire.

Once you have a strong understanding of what you want to learn and why it’s important, then create a reasonable timeline for discovery and assemble the right team to get the answers. Focus on business issues and you can drive successful project outcomes.

Which of the four “W”s do you relate to most? Drop me a line at

About the Author: Darin Bartik