Making experiments pay

A version of this article appeared in the Autumn 2020 issue of strategy+business.

The Power of Experiments: Decision-Making in a Data Driven World

by Michael Luca and Max H. Bazerman, MIT Press, 2020

The current pandemic is a dramatic case study in why experiments matter. How else are we to know which tests accurately identify COVID-19 infections and their telltale antibodies, whether and how well drugs already at our disposal mitigate the suffering inflicted by the coronavirus, or whether the vaccines currently in development will actually protect us from it? Randomized controlled trials are the only way to answer these questions short of playing Russian roulette with the lives of large numbers of people.

Experiments have been around for a long time. In The Power of Experiments, an introductory paean to the benefits of experimentation for corporate and government decision making, Harvard Business School professors Michael Luca and Max Bazerman peg the first recorded experiment to the reign of King Nebuchadnezzar in Babylonia circa 600 BC.

As the Old Testament story goes, Nebuchadnezzar attacked Jerusalem and took a group of young Israelites, including one named Daniel, as servants. He ordered that they be acculturated for three years, in part by being fed the same food as the royal court, which the Israelites considered “ritually unclean.” But Daniel proposed that he and three other prisoners be fed only vegetables and water for 10 days, at which point a guard would compare their health to that of the rest of the prisoners, who would have been fed the local fare. At the end of the experiment, the guard judged the vegetarians healthier, and Daniel and his three comrades were able to continue eating the less objectionable diet.

About 2,600 years later, Luca and Bazerman write, “We are in the early days of the age of experiments.” By this, they mean that experiments have spread far beyond the boundaries of science labs and medical trials. In 2018, Google ran more than 10,000 experiments. Amazon, Facebook, Uber, Yelp, and TripAdvisor run thousands of experiments per year. As these names indicate, technology companies, especially those with digital platforms, are at the forefront of corporate experimentation. That’s because it is relatively easy to run experiments on their large, captive audiences.

Some of these experiments pay off to the tune of tens of millions of dollars. For instance, eBay was paying Google about US$50 million annually to advertise against search results. Members of the auction platform’s economics research team wondered whether their practice of placing ads by search terms that included the company name, such as eBay or eBay shoes, actually made sense.

“Working within eBay’s economics research team, economists…ran a series of experiments, seeking to find out exactly what the financial returns on Google ads were,” relate the authors. “They turned Google ads on and off, experimentally varying across markets. They tracked the traffic coming to eBay from Google ads. And, critically, they also tracked traffic coming from organic links — that is, from people who were going to eBay from unpaid Google search links.”

Obviously, eBay received no advertising clicks when it didn’t place ads, but the experiments found that in markets where the company didn’t advertise, it got a spike in traffic from unpaid organic links. “Evidently, users who Googled ‘eBay’ (or another eBay-related search term), who had been clicking on the ad because they saw no reason to scroll down to the organic link just below it, were now instead clicking on the first organic search result. For these searchers, eBay essentially swapped in free organic clicks for each advertising click lost,” explain Luca and Bazerman. “In other words, much of the money eBay was shelling out to Google each year was a waste.” After the results of these experiments were published, 11 percent of large companies that were buying search ads in the same way as eBay discontinued that advertising.

Experiments can help companies make more money, but sometimes they can get companies in hot water, too. As the authors point out, using experiments to optimize sales can backfire if the experiments lead to policies that leave customers feeling abused, such as “shrouding” fees until checkout. In addition, many companies do not tell customers when they are the subjects of experimentation, assuming that clauses buried in unread user agreements are all the permission necessary. That may be, but customers who learn they have unknowingly been part of an experiment can end up feeling manipulated, creating a backlash that is damaging to brand equity and company reputation.

Many stories in the book illustrate the benefits and pitfalls of experimentation, and they are interesting enough. Happily, however, the authors also extract lessons from the stories for leaders who want to use experiments to bolster the decision-making prowess of their companies.

One of the lessons is that the ability to conduct experiments should not be restricted to scientists ensconced in centers of excellence. Instead, there should be an “experimental infrastructure” that is available to managers and teams throughout the organization. Another lesson is that managerial judgment is a key element in successful experimentation: For instance, an experiment might find that a coupon is highly successful in the short run, but if the coupon promotion ends too soon, it might not reveal the deleterious effects of the discount in the long run. A third lesson is that companies should be transparent about the experiments they are running to avoid provoking customer mistrust and dissatisfaction.

The Power of Experiments is a fast, accessible read that offers a good overview of the applications, promise, and perils of experimentation. It’s a good place for executives to start as they consider whether and how to use experiments inside their companies.