This article was originally published by Booz & Company.
For some, “big data” is a revolution poised to transform everything from how we gather intelligence on terrorists, to how we manage our supply chains, to how we brush our teeth. For others, it’s an evolution, little more than the latest twist in the decades-old notion of business intelligence. Sorting out its true potential is no easy task.
Fortunately, Tom Davenport, the President’s Distinguished Professor in Management and Information Technology at Babson College and a long-standing expert on business analytics, has turned his attention to the issue. His newest book, Big Data at Work: Dispelling the Myths, Uncovering the Opportunities (Harvard Business Press, 2014), avoids the common focus on the technological marvels of big data. Instead, it examines the present and future potential for solving real business problems, and the management and strategic challenges that big data raises as companies strive to put it to use.
Davenport recently spoke with strategy+business about big data’s potential for transforming not just how companies gather and analyze information, but how they use it to learn about their external business environments, make decisions, and develop new strategies.
S+B: In previous books, you’ve written extensively about business analytics, and now you’ve turned your attention to big data. What’s the difference between the two?
DAVENPORT: It’s interesting. Initially, I didn’t see much of a distinction, and I thought that I could kind of rest on my laurels and not write a book about big data—because the fact is that the analytical tools and approaches used are not all that different for big data. But when I started talking to companies and data scientists, I realized that there really were some fairly substantial differences—some that have yet to be fully articulated and some that are already in evidence.
First, much of the data used in big data analysis is unstructured. It’s amazing that we can now analyze text and voice and video. But you need to figure out how to structure all that data and get it in a format where you can analyze it, and it takes a lot of time and effort to get the data into useful rows and columns. Even once it’s structured, it’s not easy to analyze, for example, the meaning of people’s comments on social media or in blogs or customer reviews. I was talking to a professor at USC who studies political discourse in social media, who pointed out that it’s tough in that realm because computers are totally incapable of detecting sarcasm.
Second, the sheer speed of the flow of data is now like a river, very fast moving and big, that just keeps on coming. This brings up a real management challenge: We simply don’t have the continuous decision-making methods that can make effective use of the continuous data stream we have at our disposal. So we need to shift our approaches to both how we monitor data and how we use it in decision making.
We’re already seeing this issue arise in the sentiment-analysis applications where people are looking at what their customers are saying about them on the Internet via social media and other channels. The technology allows them to monitor the data as it comes in, and they say, “Oh, look—opinion is up, it’s down, it’s up, it’s down.” But they don’t really know what to do with the results, and they never set up any criteria for when and how they should take action.
To make matters worse, data moves and changes so quickly that in order to make decisions about it, you have to create thousands of new models a week—a speed that people just aren’t capable of. In fact, as humans, we can no longer understand a lot of the reasons that one marketing effort, for instance, might succeed more than another. I think the whole realm of managerial decision making, at least in marketing—and probably in some other areas as well—is changing. The best we can do is to take a high-level overview of how these systems are working and hope that they don’t get off track. It’s not that different from what has already happened in financial services, with flash crashes and the like. And the companies that can develop these high-speed decision-making capabilities in response to the sheer speed of big data will be taking a big step forward.