Caveat emptor, CEO
A short list of questions can help leaders avoid the potentially harmful consequences of flawed management studies.
Purpose-driven organizations. Disruptive innovation. Reengineering. Five Forces analysis. Shareholder primacy. For better or for worse, many of the ideas that leaders adopt — and sometimes bet the future of their companies on — come from the academic world, and virtually all of them are promulgated by academic research. That’s why the rigor and validity of management studies should be as great a concern inside C-suites as they are in colleges and universities.
It’s also why Dennis Tourish’s take on the current state of management research is disturbing. “It has…become evident that various forms of research malpractice are common in our field,” writes the professor of leadership and organization studies at the University of Sussex in the introduction to his book Management Studies in Crisis: Fraud, Deception and Meaningless Research (Cambridge University Press, 2019). “I’m talking about outright fraud such as inventing data, but also about plagiarism, self-plagiarism, poor-quality statistical analysis, and p-hacking.” Tourish goes on to support this contention with a tour of the current state of management research that is akin to Dante’s tour of hell.
Leaders could ignore management research. And if things are as bad as Tourish says, that might not be an outlandish proposition.
Tourish has particular scorn for the research on two concepts currently in vogue — authentic leadership theory (ALT) and evidence-based management (EBM). ALT, an offshoot of James MacGregor Burns’s transformational leadership that was popularized by Harvard Business School professor and former Medtronic CEO Bill George, holds that leadership success derives from the alignment of who you are on the inside with how you behave on the outside. After examining the research, Tourish concludes, “ALT is little more than a series of fables, designed to reassure us that leadership is simpler than it is and that introspection can lead us all to salvation.” Be still, my cynical heart.
EBM, popularized by b-school professors Jeffrey Pfeffer and Robert I. Sutton, posits that managerial decision making should be based on the “best available evidence.” This sounds logical, until you realize that the best available research includes the seriously flawed management studies that Tourish is calling out. “A narrative is adopted in which the world of management practice is simply ‘out there,’ awaiting discovery,” he explains. “The issue of how to adjudicate between different truth claims is purely technical — we need to do more studies, until the issue in question is resolved. I am not so sure.” Thus, EBM can be hoisted with its own evidential petard.
Tourish spends most of the book delving into the problems in management studies, and he suggests some solutions for the academic audience. Unhappily, though, he doesn’t offer similar advice to business leaders.
So what options are available to executives? Leaders could ignore management research. And if things are as bad as Tourish says, that might not be an outlandish proposition. But a more palatable alternative might be to reverse engineer some of the flaws pointed out in the book and reframe them as a short set of questions for identifying and avoiding suspect research. Those questions might look something like this:
1. What’s the claim? Read the findings in the conclusion section of the study. When you translate the jargon and parse the findings, what’s left? Here’s an example from a study cited by Tourish: “Authentic leadership is positively correlated to follower perceptions of leader behavioural integrity.” Translation: Leaders who behave with integrity are seen as behaving with integrity.
If the claim is tautological or otherwise obvious, consider moving on.
2. How was the claim studied? Read the sections that describe the design of the study. Ask yourself whether the design is rigorous. For nearly a century now, the findings of the Hawthorne studies conducted at the Western Electric Company by Elton Mayo and Fritz Roethlisberger have been cited as proof that simply paying attention to workers improves productivity. Tourish points out that the two-year-long first stage of the study involved just five female workers, a host of changes (including more rest periods and a different incentive system), and no control group.
Also ask yourself if the design is relevant to you and your company. Lots of management studies use students as subjects. How they respond in a survey or an experiment may have little or nothing in common with how your company’s workers will respond. If the study is poorly designed or if its design is irrelevant to your situation, the odds that it will offer you useful insights are long indeed.
3. How “significant” are the results? Statistical significance isn’t necessarily significant in the real world. Moreover, so-called p-hacking, that is, manipulating the analysis to lower the probability that the results of the study might have occurred by chance, is rife, according to Tourish. If the results of the study are based solely on p-values, view it with suspicion.
4. What other explanations could there be for the results? The fact that a hypothesis was tested and a result was produced doesn’t mean much in and of itself. Correlation is not the same as causality. Tourish notes that there is a high correlation between the divorce rate in Maine and the per capita consumption of margarine. But, if you live Down East, changing what you put on your toast is unlikely to save your marriage. If the study doesn’t establish causation, you may be wasting your time.
It’s hard to say whether the shortcomings of management studies are any greater now than they were in years past. Tourish convincingly argues that they have been flawed from the start, back when Frederick Taylor laid the foundations of scientific management. However, there is no doubt that many more studies are being conducted today than in the past, and also that the pressure on academics to publish studies is greater than ever. This could be a boon to business leaders, but it also means that discretion is sure to be the better part of valor when it comes to tapping management studies as a source of insight.