The Science of Subtle Signals
By analyzing overlooked behavioral cues, researchers are creating a new understanding of organizational effectiveness.(originally published by Booz & Company)
In 2006, when Vertex Data Science — a US$724 million private company based near Liverpool, England, and one of the world’s largest providers of call center outsourcing — wanted to improve the performance of its telephone sales operators, the managers went looking for an unusual kind of self-understanding. They enlisted the aid of Alex Pentland and his colleagues from the Human Dynamics Group at the Massachusetts Institute of Technology’s Media Lab, the elite research institute for digital technology founded by technology pioneers Nicholas Negroponte and Jerome Wiesner. The researchers at the Human Dynamics Group were best known for their experiments in human–machine interplay and wearable computing: using portable devices built into eyeglasses and clothing to track movement and other human activity. They traveled to Vertex’s operations offices in Inverness, Scotland, to set up electronic devices that analyzed the speech patterns of the operators on the call center floor. The devices captured neither the specific words that the operators used nor the logic of their conversations, but only the physical voice signal: the measured variations in tone and pitch. Even so, Pentland and his researchers predicted accurately, after only a few seconds of listening, the ultimate success or failure of almost every call.
Successful operators, it turned out, speak little and listen much. When they do speak, their voices fluctuate strongly in amplitude and pitch, suggesting interest and responsiveness to the customer’s needs. Operators who speak with little variation come across as too determined and authoritative, but by speaking invitingly, being responsive but not pushy, a skilled operator can let callers find their own way to a sale. “Like a mother speaking singsong to a baby,” says Pentland, “variation sounds perky and inviting. If operators do it right, they’re almost certain to be successful.” Armed with this understanding, a company like Vertex can train its operators to converse more effectively, and can seek new hires who exhibit these speech patterns. If a call starts going badly, a supervisor can detect the signs quickly enough to switch it to another operator. Early experiments have suggested that these insights can improve a company’s telephone sales performance by 20 percent or more. And the same is true of other forms of corporate communication. “In pitching business plans, for instance,” Pentland points out, “consistency of tone and pace is key to getting your plan rated highly.”
This story is a straightforward tale of managerial intervention and success. But it also throws down a profound challenge to the prevailing views of organizational effectiveness. Most explanations of human behavior in the business world presume that people — be they employees, consumers, or executives — are influenced most by meaning and reasoning. It’s what gets said that matters, not how it is said. But the performance of these telephone operators and a growing volume of other evidence suggest that this view is seriously flawed.
In a wide variety of facets of everyday business, the keys to sustained success may actually lie in understanding the kinds of signals that are ordinarily overlooked: tone of voice, body language, the ways people congregate (or don’t), the time spent on tasks, the rhythms of workplace activity, and the patterns of social networks. Those on Pentland’s team — and their counterparts at other research institutions, such as Xerox’s Palo Alto Research Center (PARC) and Intel Research in Seattle — are designing new ways to track and make sense of such indicators. The resulting new science of subtle signals may lead not just to more profitable sales pitches, but also to a richer, deeper understanding of the practice of management and the way organizations work.
Anyone in business knows through painful experience the pervasive problems that exist because our knowledge of organizations is imperfect. Key information fails to flow to those who need it, departments fight with one another, and managers make decisions on fine-sounding theories rather than real information. Most mergers and acquisitions never realize the “synergies” that were envisioned. All this experience, and more, suggests that there is good reason for believing, as organizational theorist Elliott Jaques asserted a decade ago, that “management is in the same state today that the natural sciences were in during the 17th century.… There is not one single, well-established concept in the field of management on which you can build a testable theory.”
But what if sensors and networks of sensors could transform organizational research much as microscopes and new forms of dissection transformed medicine in the 18th and 19th centuries? Instead of revealing the cell and microbe, these devices would uncover patterns of activity that usually go unobserved in organizations: the dynamics of person-to-person relationships and the ways they affect managerial decisions and organizational practices. Imagine, for example, an automatic system that could detect a breakdown in the trust on which a creative team depends and flag specific steps that could fix it, or one that could map out the complete flow of information and knowledge within an organization — even what happens at the coffee machine or during social gatherings — and identify key hubs of exchange or bottlenecks.
At the MIT Media Lab, Pentland leads a team of about a dozen researchers who have developed a range of small, wearable electronic devices that can easily and accurately gather the kinds of social data needed for such analyses. These devices track not just the physical location of the people who wear them, but also the finer details of a person’s movement— in effect, his or her body language — and several distinct features of his or her vocal behavior. And by taking note of people’s proximity to others and the patterns of their movement, the team can foster new insights into collective human behavior: the subtle differences between effective and ineffective teams, and the structures and incentives that either improve or block collaboration.
For example, computer scientist Tanzeem Choudhury — a former student of Pentland’s currently dividing his time between Intel Research and the University of Washington — and several colleagues have begun to experiment with “dense sensors,” wearable stickers equipped with radio frequency identification (RFID) transmitters or motion detectors. The data from these sensors can be analyzed and compared to broader community data, such as crime and traffic statistics, to build models that describe and even predict the daily patterns of people’s lives, and their ever-evolving social networks. Choudhury’s team is exploring the idea of designing “smart environments” that would respond intelligently to people’s needs — automatically introducing crucial information into a discussion, for example, even when no single individual might recognize its vital relevance.
And in a still more ambitious study earlier this year, Pentland, teaming with David Lazer and Nancy Katz of Harvard University’s Kennedy School of Government, put sensors on hundreds of volunteers and recorded streams of data as they went about their business, from their morning commute through the lunch hour and into evening, capturing data about each meeting and encounter. The data revealed precisely who interacted with whom, how frequently, and whether the interactions happened in the workplace or elsewhere — over a few beers, for example. If someone gave a presentation to a group, the sensors would show how stressed he or she felt, as reflected in variations in the rhythms and pace of his or her speech; they would also reveal if the person felt confident and appeared that way to others, and who in the room responded with genuine interest.
As social network specialists have been saying for years, beneath the formal organizational chart of any company lie hidden webs of social interactions that we rarely talk about, webs whose existence we may not even acknowledge. The health or dysfunction of these social networks can determine the effectiveness of a team, a large group, or an entire firm. “Ignoring these influences when you’re running a company is crazy,” argues Pentland, “because the data shows that it is at least as important as our rational behavior.” To be sure, most senior executives know full well that the “soft side” of their operations is the most important enabler of a well-functioning team or company, but the tools for monitoring or measuring the soft side have always been expensive and unreliable. Even the most diligent social network analysts can’t sit in a corridor and observe a company for hours on end. They are limited to interviewing or surveying people about whom they communicate with, and the answers may not always be accurate or complete. The technologies of sensing and observation may be about to change all that, while increasing the number of factors and indicators that can be measured in the process.
In short, these sensors may make it possible to track the unconscious and instinctual side of human behavior, along with the collaborative and social side, in a way that helps some companies outcompete their rivals. Sensors may put management research on a much more empirical path, providing fine-grained data that could lead to a more innate, reliable understanding of how organizations work. And like many scientific advances, they may also raise a host of new ethical concerns. To understand corporate behavior thoroughly, researchers like Pentland and his team have to monitor and analyze people’s behavior in unprecedented detail, putting potentially sensitive data on subtle personal cues and social habits into permanent computer storage. It’s not yet clear how researchers can balance the desire for workplace privacy with the equally compelling drive to understand how corporations really work — and provide their host companies with competitive advantage in the bargain.
An MIT Media Lab researcher wearing
Mapping Cognitive Channels
In 2005, in experiments conducted jointly with Jared Curhan of the MIT Sloan School of Management, researchers from the Human Dynamics Group asked MBA students to take part in simulated face-to-face negotiations. One student played a middle manager taking a job in a new division, and the other the vice president of that division. They were asked to negotiate the manager’s salary package, with real monetary rewards at stake for the participants. The negotiations often lasted an hour or more. Yet in just five minutes, an electronic sensor could predict with 87 percent accuracy which person would come out on top, merely by cueing in to bodily movements and manner of speech, ignoring words and strategy. As the sensor data revealed, successful middle managers tended to be strong on “mirroring” behavior — unconscious mimicking of the gestures and movements of their conversational partners. This demonstrated empathy and understanding. In contrast, the most successful vice presidents tended to talk more and control the pace of the conversation, a social behavior that the researchers referred to as “engagement.” For both participants, a consistent emphatic tone, conveying confidence, was also critical.
This type of research confirms in the business setting what some psychologists have suspected for years — that human behavior can often be predicted with remarkable accuracy by paying attention to so-called thin slices of what people do. Malcolm Gladwell popularized this concept in his 2005 book, Blink. After watching 15 minutes of video of a married couple conversing, trained observers can tell with 90 percent accuracy whether the marriage will last. By observing a doctor speaking with a patient for 45 seconds and attending only to his or her pitch, rhythm, and intonation — which convey warmth or lack of it — analysts can identify those doctors likely to be sued for malpractice.
In short, people have two distinct “channels” of communication — the obvious verbal and rational channel, through which information flows linguistically, and a nonlinguistic channel that we often ignore, but that carries at least as much information.
From an anthropological point of view, it’s not surprising that a lot of human influence takes place nonverbally. Apes, chimpanzees, and other primates — our close evolutionary cousins — lack anything like our facility for language, yet still lead sophisticated social lives. They organize groups for hunting, collective defense, and child rearing. All this takes place through nonlinguistic means, by displays of power, meaningful noises, and facial expressions. Instincts for this kind of communication enabled humans’ ancestors to form strong, cohesive groups, and human beings still possess those instincts, alongside more recently evolved talents for language and reason.
Some of the most famous social psychology research of the last century documented the extent of group influence on individuals. For example, in a 1951 experiment directed by Solomon Asch at Swarthmore College, researchers asked experimental subjects to say which of three lines on a paper matched the length of another line, using lengths so different that the correct answer was obvious. If they heard a number of people give the same wrong answer, many people followed along with the crowd, completely ignoring the clear input of their senses. Recent experiments conducted with magnetic resonance imaging (MRI) machines by Gregory Berns of Emory University suggest that peer pressure can alter how people actually see the lines.
In other words, people in group situations don’t consciously weigh the options and then deliberately (or timidly) choose to conform. Instead, the conforming happens automatically and unconsciously. Those dynamics happen so often, and so consistently, that they inevitably play a role in the ways people make decisions in the business world. But this unconscious behavioral channel is generally ignored in most management thinking; even a writer like Gladwell, whose work often falls at the nexus of business and psychology, didn’t recognize that these behavioral channels are so obvious that they can be picked up by machine.
The idea of using sensors to capture these subtle signals began to emerge in the late 1980s. At PARC, a young researcher named Mark Weiser coined the phrase “ubiquitous computing” in 1988; he distributed electronic badges that transmitted information about where PARC employees were walking, so that people working there could see crowds forming on electronic displays. Weiser, who later became PARC’s director of research, passed away in 1999; by that time, other researchers, including John Seely Brown, Xerox’s chief scientist, and Alex Pentland at the Media Lab, had begun linking ubiquitous computing with the emerging idea of social network research, tracking the patterns of connection in informal communication among people.
Pentland began developing technology to probe network influences after an experience in the early 2000s serving on the board of a Media Lab initiative to create spin-off laboratories overseas. Nothing in that initiative had gone quite as planned.
“We had some of the most brilliant and powerful people in the world,” he recalls, “but our work was a disaster, just an incredible disaster. People were making decisions that were, on the face of it, ridiculous. Two days later you’d think, ‘How in the world did I go along with that?’ It was as if your brain had been turned off.”
Pentland’s board experience led him to recognize the enormous power of nonverbal communication. The leading directors were all extremely charismatic and certain of themselves; everyone else went along with whatever they said, almost without thinking. “This experience really affected me,” says Pentland. He began studying the scientific literature on nonlinguistic human communication, a body of research that is extensive, but mostly qualitative. And then he focused on building devices to measure that communication. “You need instruments,” he says, “because as people we can’t really observe others objectively.”
The instruments he and his colleagues developed exploit modern telecommunications technology in a number of ways. They can program personal digital assistants (PDAs) and specially configured “smart” cell phones to keep track of their owners’ proximity to others, using the unique identifiers built into cell phone and Bluetooth transmitters to identify each individual’s location. Other electronic badges complement these locators with more precise position measurements — based on global positioning system (GPS) data, they are accurate within two meters — as well as capturing audio signals and measuring upper-body movements with an electronic accelerometer.
Gathering this data is just the first step. Pentland and computer scientist Nathan Eagle have developed a method they call “reality mining” for analyzing and drawing meaning from the data. In one study, about 100 students carried reprogrammed Nokia cell phones around with them for nine months; researchers then analyzed the voluminous data set for patterns in the behaviors of both individuals and groups. They found they could make accurate predictions on where any person was likely to be seen at a certain time of day, and whom they’d probably be talking with. They could also build up accurate pictures of the networks of friends or co-workers to which the students belonged and identify their most important social links.
The ultimate aim of this kind of work — as another collaborator, Mark Mortensen of the Sloan School, points out — is to go far beyond the capabilities of traditional social network analysis, which mostly relies on human recall, with all its attendant weaknesses. “A lot of workplace communication takes place through spontaneous interactions, the watercooler kind of stuff,” he says. “You can sit people down with paper and ask them who they interacted with, how, when, and so on over the last three months, but the results are always biased.”
Those limitations, says Mortensen, make current analyses of team behavior inadequate. “People decide to work on something, in virtual teams or whatever, and afterward, if they didn’t kill anyone, they write a book about it. But what they say is purely anecdotal and there’s no science in it. We want to begin building a real science that is quantitative.”
Self-Awareness, Stress, and Groupthink
It may seem overly reductionist to try to understand people through signals from gadgets hooked to their belts. But a number of major organizations have already lined up to try it out. In collaboration with Thomas Malone of the MIT Center for Collective Intelligence, Pentland’s research team has begun to use sensors to observe creative group behavior at a major German bank. (One preliminary finding: People who maintain lots of e-mail and face-to-face contact report high job satisfaction and personal productivity; those who socialize less, even with the intention of getting more work done, express overall less satisfaction.) Soon, working with Eric Brynjolfsson at the MIT Center for Digital Business, they’ll be helping network hardware company Cisco Systems improve one of its emergency call centers. The Human Dynamics Group from the Media Lab is also running projects with automobile engineering teams at Nissan, with several universities, and with a Boston hospital. (Indeed, sensor technology may well have medical applications, as two clinical trials showed that a significant lack of social signaling activity, readily detected by the sensors, correlated strongly with well-known signs of clinical depression.)
In the business setting, a company might use this type of computer-augmented self-awareness to train its negotiators. Or its sales force. By identifying social signals linked to persuasiveness, a manager could help salespeople train themselves to achieve better results without necessarily working harder. Since the most persuasive gestures, body language, and voice styles can be identified, a CEO giving frequent pitches to shareholders could presumably be trained to do so far more effectively. In another setting, companies hiring new staff might use sensors to match their employees more effectively to their jobs. Someone who is genuinely interested in a project (or in any endeavor) tends to display lots of activity and variability in both voice and gesture, and often speaks more rapidly.
Alternatively, a company might use sensors to monitor the response of a new hire as he or she mingles with the different project teams and comes into direct contact with the teams’ work. By basing staff placements on observations of these real-life (but usually ignored) reactions, an organization could create a more fulfilling and productive environment. And marketers are already beginning to use wireless sensors to see how people respond to a new product design, both consciously and unconsciously. After all, about 80 percent of all new products still fall well short of sales expectations, even though companies spend millions on focus groups and surveys to probe consumer interest. If these traditional methods rely too much on the conscious, linguistic communication channel, they would largely ignore the reactions that matter most.
The sensors could be used in other applications as well. For example, many studies have shown that workplace burnout is a serious issue that costs companies millions each year. But because people tend to hide stress, it can be very difficult, if not impossible, to detect. Sensors may change all that. In a trial study, Pentland and student Michael Sung fixed physiological sensors on students playing poker for real monetary stakes, and monitored bodily movements, skin conductance, and heart rate. They found that they could identify moments of especially high stress (as later reported by the participants) with 80 percent accuracy. They could also tell about 70 percent of the time when players were bluffing. Pentland suggests that this kind of monitoring might be useful for identifying people who are potentially headed for burnout, and who therefore require more detailed monitoring.
One could easily list hundreds of other ways that sensors might make enterprise more efficient. Improving our understanding of individual behavior and what influences it may be only the beginning. Mortensen foresees wiring up an entire team, division, or company, and gathering real information quickly on who interacts with whom, what kind of knowledge they share, and whether the interactions are successful. With networks of social sensors, organizations may soon be mounting a scientific, data-driven attack on the most baffling and damaging problems they face — those that stem from the myriad and mysterious dysfunctions affecting groups.
For example, the most serious problems confronting modern organizations may not be individual problems but group issues: internal polarization that inhibits discussion, or endemic “groupthink.” Teams drift toward mindless decisions because no individual wants to “stand out” from the perceived consensus. These and other dysfunctions are extremely difficult to detect before they cause damage, because they involve nearly invisible patterns in the behavior of many individuals.
Using social network analysis, however, organizations have been able to improve information flow among different parts of their operations. Research led by Steve Borgatti, chair of the organization studies department at Boston College’s Carroll School of Management, has used surveys and interviews to map interactions between the engineering and manufacturing departments of a large organization. Borgatti and his colleagues discovered a problem that no executive could have been aware of — that almost all communication between the two groups passed through one particularly skilled and approachable person, who was consequently overwhelmed and often behind schedule. After identifying this hidden problem, executives introduced other go-betweens to share the load and improve the departments’ coordination.
Sensors and Sensibility
Sensors, working all the time or close to it, could gather far more accurate data about information flows happening on a minute-by-minute basis. In one study, after constructing a social diagram of a company, the Human Dynamics Group researchers could actually see polarization taking place — as if the company had been put under a microscope. “You’ll see two people going at it in a meeting,” says Pentland, “and then polarization growing around them,” reflected in the way people respond to the two main figures, and gather around them in distinct factions. Analysis of the sensory data in this case showed two people, in particular, trying to lead — both very active, with voice and body language conveying determination and authority. Neither individual showed the kind of mimicry or voice variation that would convey empathy; in other words, neither backed down. Soon others began to be recruited into the two opposing teams. If a manager saw this type of pattern in real time, he or she could tune in to the emerging problem and try to defuse it — addressing the root of the tension and helping the two sides get through it.
Another valuable asset of sensors is their ability to track patterns over time. They show not only who interacted with whom, but precisely when, so managers and employees alike can see how activity in one place — say at an engineering department at corporate headquarters — flows out to influence production far away, at a factory, for example. This is the kind of thing sensors can get at, but questionnaires and surveys cannot.
It is not yet clear, of course, how corporations will handle the understandable concerns of employees and customers about the Big Brother intrusiveness of this type of data gathering. As with Google’s “street view” feature, in which images taken on public streets by a roving camera are routinely posted on the Internet as a part of the search engine’s mapping service, the perceived danger lies in the breadth of observation. People can never be quite sure what activity will be gathered and inadvertently exposed in the random tracking of an unsupervised set of sensors. Knowing this, people might censor themselves more, thereby cutting back the very type of informal and free-form creativity that most businesses need more of.
Pentland, Mortensen, and the other researchers insist that the ethical challenges the use of sensors raises must be taken seriously if the technology is really to be beneficial. “Companies shouldn’t just look at this as another way to spy on employees,” says Pentland. Using sensors for monitoring and control would be a surefire recipe for resentment and loss of morale, he says.
Several ideas might help companies prevent problems. Pentland suggests, for example, that the technology ought to be used on a voluntary basis, with individuals adopting it because they learn the benefits that it brings for both themselves and the company. An organization could store information on individuals’ own personal computers, rather than in a central location. It might also give people the opportunity, at the end of each day, to review the data that’s been recorded about their activities. They could have the option of deleting anything they’d prefer to keep private. The devices might be fitted with an additional button that would erase, say, the last 10 minutes of data, or data collection might be strictly limited to teams, time frames, and workplace settings where there has been explicit agreement in advance to allow the analysis. Although all these possibilities reduce the amount and quality of data that would be gathered, some steps along such lines will be crucial for giving people confidence that their privacy is being protected.
If privacy issues can be resolved, a new world of organizational understanding may be at hand. History teaches us that data, when it becomes available, leads to powerful transformations of human understanding and capability. For example, the great scientific breakthrough of Johannes Kepler, working out the laws of planetary motion, was made possible by the painstaking astronomical observations gathered during the last several decades of the 16th century by the Danish stargazer Tycho Brahe with his own handmade instruments.
In the 1980s, grocery stores first introduced barcodes merely as a technology to improve checkout efficiency and keep inventory automatically. But the resulting oceans of data on product flows have now completely transformed the retail business. Two decades from now, we may be saying the same thing about the wave of sensors currently poised to invade corporate life. By probing the otherwise invisible social interactions on which organizations ultimately depend, these sensors will make it possible to explain scientifically why a creative design team suddenly became dull and uninspired, why a group of brilliant advisors made a series of inane decisions, or why two groups on whose open sharing of details the company’s welfare depended had great difficulty in speaking to one another. Today’s executives and management theorists can only guess the answers to puzzles like these, but tomorrow they’ll have the equipment to find answers by direct measurement.
Pentland, Mortensen, and their colleagues refer to the future organization that knows itself through sensing technology and manages itself accordingly as the “sensible organization.” It will monitor the flow of information between its departments and between facilities, and make accurate maps of where its key knowledge resides so that employees can easily tap into the resources they need. On the basis of knowledge, the sensible organization will promote the kind of communications that can build trust and move quickly to defuse emerging problems, even before the people in the organization know a problem exists. It will monitor team dynamics through time, catching patterns of stress or stagnation, and intervening to keep people working together creatively. And it will uncover social patterns that today we cannot even recognize or talk about, but which can explain, more definitively than ever before, the shining success of one company and the dismal failure of another.
Reprint No. 07307
Mark Buchanan (firstname.lastname@example.org) is the author of The Social Atom: Why the Rich Get Richer, Cheaters Get Caught, and Your Neighbor Usually Looks Like You (Bloomsbury USA, 2007). Formerly an editor with Nature and New Scientist, he was a guest columnist for the New York Times in 2007 and holds a Ph.D. in physics from the University of Virginia. Visit his Weblog.