A version of this article appeared in the Autumn 2021 issue of strategy+business.
Noise: A Flaw in Human Judgment
by Daniel Kahneman, Olivier Sibony, and Cass R. Sunstein, Little Brown, 2021
Companies live and die by the ability of the people who work within them to make sound judgments. Their judgments determine what strategy to follow, where to invest R&D funds, how to set prices, who to hire and promote, and a myriad of other decisions. Some of the decisions are one-offs; others are made repeatedly. There’s just one problem, assert Daniel Kahneman, Olivier Sibony, and Cass R. Sunstein: “Wherever there is judgment, there is noise — and more of it than we think.”
Noise is the major source of variability in judgment and, thus, a major cause of decisions that miss their mark, according to the professorial supergroup (henceforth, KSS). Kahneman was awarded a Nobel Prize for his work as a behavioral economist; Sibony is an expert on decision-making who teaches at HEC Paris and Oxford’s Saïd Business School; and Sunstein is the Harvard prof whose work on nudges has been influential in public policy.
Noise is also the title of the trio’s new book, a 400-page tome that should leave executives who take the time to wade through it more than a little unsettled. Their uneasiness should stem from the likelihood that they have been underestimating the negative effects of noise on decision-making in their organizations. When KSS asked 828 senior executives in a variety of industries how much variation they expected to find in expert judgments, their median answer was 10 percent.
In reality, the variation in expert judgments can be four to five times that. When two members of KSS ran a noise audit for an insurance company, they discovered that the median difference in the pricing determined by its underwriters for identical policies was 55 percent, and the median difference in the payouts determined by its claims adjusters for identical claims was 43 percent. “One senior executive estimated that the company’s annual cost of noise in underwriting — counting both the loss of business from excessive quotes and the losses incurred on underpriced contracts — was in the hundreds of millions of dollars,” they write.
The insurance company is the tip of a noise iceberg that KSS explores in excruciating detail. Depending on the judge, the sentence for a convicted heroin dealer in the U.S. can be from one to 10 years in prison, and for bank robbers, from five to 18 years. Fingerprint examiners change their opinions when shown the same prints weeks apart — a disconcerting discovery for those of us who believe that fingerprints are incontrovertible evidence. The rate of false negatives by radiologists screening mammograms for breast cancer ranges from zero to more than 50 percent. The variance in employee performance appraisals attributed to noise having nothing to do with actual performance is 70 to 80 percent. The probability that two interviewers will produce similar ratings after meeting the same job candidate is 62 to 65 percent.
KSS pegs this depressing litany of judgment variability to three types of noise. Level noise is variability in the average responses of different individuals. The differences in sentencing severity among judges across bank robberies is an example of level noise. Pattern noise is variability in the responses of individuals to specific cases. The differences in the pricing recommendations of the insurance company’s underwriters for a specific policy is an example of pattern noise. Occasion noise is variability in the responses of the same individual. A radiologist who reads the same mammogram differently on different days is an example of occasion noise.
Wherever there is judgment, there is noise — and more of it than we think,” write Kahneman, Sibony, and Sunstein.
Unfortunately, there is no simple solution to noise. It is bundled up in many of the cognitive flaws that define us as human beings, which all three members of KSS have studied and written about in the past. But the authors do offer readers advice for reducing the level of noise in decision-making processes. Their first recommendation is to upgrade the quality of those making judgments and decisions — some of us are less noisy than others. If you can’t get better judges, try appointing a decision observer, whose job is to suss out decision biases and noise.
Noise also offers a valuable set of guidelines for “decision hygiene” that can help reduce noise. “We chose this term because noise reduction, like health hygiene, is prevention against an unidentified enemy,” explain the authors. The guidelines include sequencing information, so that judges don’t leap to conclusions too early; aggregating multiple, independent judgments to capture the so-called wisdom of crowds; using shared scales to ground individual judgments; and breaking up complex judgments into more comprehensible components.
As fascinating and eye-opening as this examination of noise is, the book can be something of a slog because KSS devotes more space to admiring the problem than solving it. This is understandable given the authors’ contention not only that the deleterious effects of noise are largely unrecognized, but also that when they are identified, people tend to react with disbelief or outright resistance. For instance, after the outrageous variability in criminal sentences was exposed in the 1970s, it took a decade for Senator Edward Kennedy to laboriously shepherd the Sentencing Reform Act of 1984, which created a commission to establish guidelines and mandated sentencing ranges for judges, through Congress. As a result, sentencing variability dropped significantly. But in 2005, the Supreme Court struck down the law’s sentencing mandates (making them recommendations instead) — and the disparity in sentences doubled.
Why do we resist efforts to reduce noise? The final chapters of Noise to examine seven objections to decision hygiene, ranging from the expense of addressing noise to the squelching of individual prerogatives. This extended show of objectivity notwithstanding, KSS sticks to the tune: “Our general conclusion is that even when the objections are given their due, noise reduction remains a worthy and even an urgent goal.”