Overcoming Big Data’s Challenges
Companies that have the most success leveraging big data are getting three things right: talent, technology, and culture.
Lately, it’s almost impossible to talk about business strategy without mentioning the transformative potential of big data. (There’s even an indie rock band with the name.) Many companies are actively using advanced data analytics, and others are just getting started. But as the beginners are finding out, it’s not as simple as just buying some new technology and hiring some statisticians. Gaining real business benefit from investments in big data and advanced analytics means overcoming three key challenges: talent, technology, and culture.
Talent. In order to make real progress with big data, companies need an elite core of data scientists: highly trained professionals who know how to use the advanced statistical algorithms and machine-learning protocols that are necessary to handle large and varied amounts of data coming in at high velocity. As I have discussed in earlier posts, handling the growing amount of unstructured data requires special skills.
Unfortunately, there aren’t that many people today who are trained to handle and analyze data sets of this type and magnitude. Graduate schools are producing more and more trained data scientists, but inevitably, companies that make a bet on big data will be drawn into a war for talent. They need to be prepared to fight that war.
Inevitably, companies that make a bet on big data will be drawn into a war for talent.
And having an elite core of data scientists is only part of the battle. Just as important is the cultivation of a group of people that I call bilinguals — people who can speak the languages of both business and analytics, and who can therefore translate between the advanced data scientists and the nontechnical decision makers responsible for day-to-day business operations.
After all, it’s not enough to simply gather the data and analyze it — you also have to be able to apply the analysis to the business decisions you make every day. What we’ve observed many times is that often the advanced mathematician and the businessperson will talk at cross purposes. What they need is someone who understands enough about the possibilities, potentials, and procedures of big data to help craft the solution that will ultimately be of value to the businessperson.
Technology. Having the right technology infrastructure is an essential component of big-data success. And here, companies have taken different paths. One approach is to make a very big investment in the technology stack required to handle big data. We sometimes call this the Field of Dreams approach: “If you build it, they will come.” Unfortunately, what has often happened is that companies have overshot the mark. They’ve invested more in the technology than the organization was prepared to absorb, and then they had to scramble to find enough use cases to justify the investment. Other companies have taken the opposite approach: They’ve built only the technology stack required for their immediate needs. This is akin to someone who builds a house only for the family he or she has today, not realizing that quite soon another bedroom, bathroom, or spare guest room will be needed.
The most successful path is the one that lies in between these poles. It’s better to plan investments for the midterm, to anticipate some of the new use cases the company might not be ready for yet, but can anticipate. If your company is like most, the number of use cases requested will grow as decision makers experience firsthand the benefits of more robust data and analytics.
A final point on this: It may seem obvious, but having the right talent in place — both data scientists and bilinguals — before you go shopping for technology increases the odds of getting the tech investment right.
Culture. This might be most difficult challenge to overcome. When it comes to bringing big data into the corporate culture, there are two kinds of companies. Some firms — Google, Amazon, Netflix, and LinkedIn, for example — have big data baked into their business model. These companies have no culture issue around big data, because big data was part of the equation from Day One.
Then there are the more traditional, mainstream companies — firms that have built and cultivated their management teams by heavily valuing intuition and experience as a way to make decisions. Executives from these types of companies are often heard to say things like, “I know my customers. I’m in the field all the time. I don’t need all this extra data that you’re bringing me. And I don’t understand the statistical black box that comes with it.” But no one is advocating using analytics as a replacement for judgment and intuition. Rather, analytics has the potential to become a much more powerful aid to judgment and intuition.
The bigger picture, however, is that companies and their senior leaders have to realize that the era of big data is here and, even if they themselves don’t fully embrace it, ignoring it is not an option. Because big data is so prevalent now, the technology and analytic solutions will also continue to improve, increasing the amount of insight that can be gleaned from each byte.
It’s similar to the inventions of the microscope and the telescope in the 17th century: Suddenly, scientists were able to see cells and planets in ways they couldn’t even have imagined. And in the same way that those discoveries kicked off an era of scientific invention, the advent of big data has already fostered innovation in developing new business models and solving business problems in new ways.