Skip to contentSkip to navigation

Are You Killing Enough Ideas?

Companies can improve their innovation performance by getting their formal and informal organizations in sync.

(originally published by Booz & Company)

Right at this moment, in a conference room or at an executive off-site meeting, a group of senior leaders of a large global company are wondering why they aren’t innovating enough. They’re probably focusing on a few specific questions: “Why don’t we have enough good ideas? How can we tell which idea is going to be the next big thing? Why is it so hard to get an idea from the drawing board into the market?” Most telling of all is the question: “Why do we still waste resources on ideas that people don’t believe in?” In other words, even though an idea has been effectively “killed,” it still remains on the agenda, with nobody fully willing to learn from the mistake, put it to rest, and move on to other endeavors.

At first glance, these seem like problems with the research and development function alone. But the underlying cause is more fundamental and broader in scope. For a company to excel at innovation, it must have high levels of creative capabilities, such as R&D, and production capabilities, such as operational execution. Moreover, the people with these capabilities need to work together seamlessly. For example, in setting up wise and decisive approval mechanisms that can separate the wheat from the chaff, companies must take a variety of factors into account — the designers’ original intent, the limitations of downstream production, and the needs of the marketplace — that are typically perceived differently on the creative and production sides.

In most companies, the creative and production capabilities are difficult to integrate. To understand why, it’s helpful to think about organizations as having two parts — a formal organization and an informal organization. Most organizations tend to emphasize one over the other, but both are always present. Production is largely governed through the formal side, which includes decision rights, reporting structures, established incentives, codified processes, and corporate metrics. These elements tend to enable an organization to be disciplined and efficient. To be sure, many informal connections exist in manufacturing and production, but if you want to change or influence something in a typical production process, you must do so using the formal levers at your disposal.

Creative work, however, is typically generated by the informal side — the part that doesn’t exist on paper but that influences behaviors through mutually understood rituals, shared values, social networks, commonly held ideas, and emotional connections to the work itself. These factors tend to motivate an organization to be flexible and responsive. Creative people sit in formal groups and have bosses, but they are more likely to draw on their informal connections with people at multiple levels, be passionate about searching for the best idea, and take risks that often don’t work out.

In addition, innovative ideas that have commercial potential need early support from disparate parts of the organization. Although some support can be provided through formal channels, innovators at companies such as Apple, Google, and Microsoft are masters at engaging the organization through informal means. They tap into networks of key influencers and into communities within the organization that are less reachable in formal, top-down ways.

Because real innovation requires that good creative ideas be efficiently produced and launched, a balance between the formal and informal structures is critical. If “creativity is king” and teams are encouraged to chase every idea, there will be a great deal of waste in the system. Winners for the final rounds of funding will be chosen on the basis of informal connections and relationships rather than merit — which often means that the political strength of sponsors, rather than customer interest or analysis of potential impact, will most influence decisions. In the aftermath of such a decision, the level of commitment tends to deflate during execution, and the products are less likely to win in the marketplace. Even if companies with a dominant informal organization get a winning product out in the market, they often lack the rigor to follow up quickly and eventually lose sales to more disciplined competitors.

Conversely, if the organization is weighted too heavily on the formal side, then nascent ideas aren’t allowed enough time to develop. They are subjected to processes and decision criteria similar to those used for running the existing business. This results in a bias toward incremental innovations that resemble the products and services that the existing business currently manages, rather than breakthrough innovations that might require a new way of operating and evaluating business success. In Vijay Govindarajan and Chris Trimble’s 10 Rules for Strategic Innovators: From Idea to Execution (Harvard Business School Press, 2005), the authors emphasize learning to forget existing approaches, because new products and services will need to be fundamentally different and even disruptive to the prevailing assumptions of the old industry.

Seeking a Deliberate Balance

By contrast, a clear and successful innovation strategy requires a conscious balance between the formal and informal aspects of the organization. Both are drawn in to create and support the capabilities that are most needed.

One company that paid attention to this balance was Bell Canada in the mid-2000s. Like most companies in the telecommunications industry, in 2003 Bell Canada found itself confronting a major technological change. The prevailing methods of communicating — through telephone lines and copper cables — were quickly giving way to broadband wireless connections and Internet protocols. For Bell Canada, the result would be shrinking profits and market share. Company leaders realized that if they were to compete in this rapidly changing telecom environment, they would need a significant cultural change: to shed the mentality of a regulated utility and bring emerging technologies to market quickly and successfully. For this massive shift, they needed to motivate and engage frontline employees: the technicians and clerks who worked directly with customers and who most visibly represented the company.

Bell Canada’s transformation effort had many dimensions. One important capability that the company built was the fostering of informal networks in support of these new ideas. For example, corporate leaders set up a “community of practice” for frontline managers, that is, an informal working group that shared knowledge and learned together to improve performance by helping people take more pride in their work. These managers had an intrinsic feel for what would excite their employees and what customers were looking for. This community swelled into a virtual movement across the organization, growing from a dozen managers to more than 2,000 in just a few years.

The virtual movement, in turn, helped enable product and service innovations. A group of leaders from the R&D organization was searching for ways to evaluate new ideas with input beyond that of the traditional R&D staff members and marketing focus groups. These R&D leaders ended up connecting the formal decision-making process with the informal community of practice. They created a Web-based platform that allowed people to more easily develop and put forth new ideas, using blogs, wikis, and crowd-sourcing concepts to make the responses more transparent. The community of practice provided real knowledge of the market, and it fostered enthusiasm for trying new approaches and leaving the company’s old way of operating behind.

Managing Failures Effectively

Whether in transforming a company’s innovation practices or in maintaining them over time, one of the most revealing indicators of effectiveness is the number of losing ideas. This may at first seem counterintuitive, if the goal is to take ideas to market. However, a high number of losing ideas indicates that the informal and formal aspects of innovation are working well together. It shows that the enterprise is creatively generating enough ideas, evaluating them to predict which will be successful, then applying internal discipline to drop support for those that won’t work while shifting time, money, and attention to driving the best into the market.

By contrast, when there is an ineffective balance between formal and informal structures, it often shows up as an inability to manage bad ideas effectively. After a formal decision has been made to advance some ideas but not to pursue others, the company expends considerable effort to plan the next steps for the winners. But no one thinks actively of planning next steps for the losing ideas, to put them to rest, free up their supporting resources, and (ideally) identify and share any lessons or insights gleaned from the experience. (See “How to Shut Down a Project Gracefully,” by Rita Gunther McGrath, s+b, Summer 2009.)

When the balance is ineffective, the informal parts of the organization quietly kick into high gear to “defend the abandoned children.” Advocates of the failed project continue to spend time and resources on it, out of either inertia or false hope. The pride of seeing a personal project succeed overwhelms the pride that people might otherwise feel in supporting the group’s best assessment of what will succeed in the market. Efforts to move that investment to other projects don’t quite work. This tends to occur most often in “hero cultures,” where recognition often accrues to the “creative genius” who pushed forward despite resistance.

The formal parts of the organization tacitly reinforce this pattern. Metrics, and thus career advancement and recognition, are tied to expected results. People know that an innovator’s role in a successful product launch could be part of the justification for a promotion, raise, or merit bonus. And they know that no one is rewarded for failure. Thus, in very formal organizations, no one may explicitly shun a failed product, but employees have little or no incentive to celebrate or learn from failures. People try as long as possible to turn them into successes, and dispose of them only when it is clear no hope remains.

In the worst cases, the process leads to a vicious circle: Winning ideas aren’t resourced appropriately, and thus don’t achieve their predicted success; this undermines the credibility of the development system and thereby encourages more people to pursue their pet projects as mavericks. Meanwhile, since no one captures the lessons from the products and services that failed, companies then make similar offerings, with similar poor results in the marketplace, again and again.

Setting the Process in Motion

How, then, can companies integrate the formal and informal structures to support the best ideas, effectively abandon the weaker ones, and drive superior innovation over time? Successful organizations call on a few useful design principles when developing their innovation systems.

1. Weigh the balance of formal and informal mechanisms to meet your company’s current needs. Companies tend to vary by industry and market, and especially by their life-cycle stage. For example, startups tend to manage innovation very informally; they have less of an established business to run, and everything must shift quickly as they learn from their first efforts to bring an idea to market.

It is typically only after several years that leaders recognize that they need formal protocols, decision rules, practices, and metrics, applied across the board, to manage complexity more efficiently and ensure that people are treated fairly. After that, they may overcompensate with formal rules and processes borrowed from other companies. These tend to conflict with the informal rituals — practices that, because of their responsiveness and adaptability, were once core to their success — thereby diminishing the effectiveness of both.

Similarly, in larger companies, leaders sometimes become aware that they need to reintroduce some informality to become less rigid and more responsive. They, too, can overcompensate, creating overly flexible skunkworks that aren’t plugged into regular operations and that put forth maverick ideas that aren’t taken seriously by the rest of the organization. Some companies create a pendulum effect as they alternately emphasize formal and informal mechanisms, introducing inefficient change efforts with each oscillation.

To avoid these patterns, look at your organization as a whole, particularly in relation to R&D. Figure out ways to shape the formal and informal structures together, integrating them from the beginning. One useful approach is to map informal networks, a technique developed in academic circles in which communications among people are diagrammed to reveal the patterns of information flow and identify the “hub-like” people who contribute to it. (See, for example, “The Community Network Solution,” by Karen Stephenson, s+b, Winter 2007.) Network mapping is still a relatively novel practice in the commercial world, even though senior teams and breakthrough innovators rely heavily on their informal networks for both gaining insights and building support. There is great power in identifying critical nodes for information flow in the organization and planting creative and socially sensitive people in those positions, to make the best use of their natural networking talents to sharpen creative ideas and obtain advance support for the best ideas. (See “Finding and Grooming Breakthrough Innovators,” by Jeffrey Cohn, Jon Katzenbach, and Gus Vlak, Havard Business Review, December 2008).

2. Capture budding ideas from the widest possible net and collect them centrally. In most organizations, a subset of people in marketing or product development are formally tasked with developing and testing new ideas. They are typically chosen not because they are the most creative and insightful individuals, but because they know how to navigate the development process once an idea has been formed: They know how to get it approved or funded. Truly innovative companies like Apple or Google, by contrast, make generating ideas an informal part of everyone’s job and motivate employees largely with nonmonetary recognition. A call center representative who notices a pattern among customers’ complaints is just as likely to think of an inspired way to fix it as the marketer applying analytical techniques with focus groups. And external innovators are sometimes the best source of ideas. (See “The Promise [and Perils] of Open Collaboration,” by Andrea Gabor, s+b, Autumn 2009.)

Unfortunately, many of the best ideas never get past the rounds of informal watercooler conversations. Although the informal organization is good at surfacing new ideas outside standard operations, these ideas can fade away without help from the formal organization to codify them. Thus the challenge is to capture informal ideas without constraining them, and to bring them into formal consideration. New communication and electronic interaction mechanisms can help. For example, Web-based systems such as Starbucks’s www.mystarbucksidea.com gather and codify ideas and even use decision-market approaches to evaluate them. Companies must develop ways to motivate employees and customers to contribute their thoughts, and set up an efficient formal mechanism that makes it easy for these ideas to enter the development system. The more ideas you capture, the more likely you are to hit some winners.

3.    Involve multiple perspectives in “go/no-go” choices from the outset, and thus make them stick. Most organizations have discrete formal groups and processes that use different lenses for evaluating ideas: Marketing represents the customers, finance evaluates the economics, and engineering determines feasibility for launch. They answer the questions in series, and then “throw the problem over the wall” to the next team. They may not even be aware of one another’s findings.

The principles of focused accountability or clear decision rights provide a purported rationale for this fragmented approach. Breaking up the innovation process often seems like the easiest way to make progress. However, it ignores the fact that truly effective innovation needs to integrate choices about customers, finance, and technology; without buy-in at the outset from all these groups, choices made upstream may be undone downstream. When the final decisions about launching the product must be made, the groups often find themselves at cross-purposes; either one group wins and the others lose, or they reach a rapid but weak compromise for the sake of consensus that satisfies nobody, including the customers.

More often than not, the wisest upstream choices are enabled by a timely combination of formal and informal interactions. Right at the start, convene disparate perspectives for rigorous and synchronized debate — a seemingly messy and difficult process that actually makes things easier in the long run. The answers to three major questions should determine whether an idea should be developed or not: (1) will customers want it? (2) can we produce it? and (3) will we be able to make money from it? If you are organizing this process, set up ways to bring the necessary people into one room — or, at least, on one long conference call — to talk through all these questions together. Bring different perspectives to the surface, have productive fights (substantive and candid debates), and come to the best answer. This is the only way to resolve the natural tensions between formal groups without forcing them to compromise. (As management thinkers dating back to Mary Parker Follett have noted, compromise is more likely to water down the result than get the best from conflicting ideas.) When they arrive at joint decisions, different functions feel mutually accountable for results, which inherently improves the chances for success in any process that requires handoffs and coordination later.

Of course, such conversations consume energy and time, which is why it’s important to focus them on the most important questions. It’s equally important to be clear about how the debate will be resolved, and how each function will abide by and apply the law and the spirit of the decision to future situations. You must deliberately develop informal skills to conduct in-depth and high-stakes conversations constructively, even across functional boundaries, and must reinforce those skills with formal decision-making and enforcement processes.

4. Motivate the right behaviors. Build incentives and practices, both formal and informal, that will support the right kinds of implementation when no one is watching. For example, make explicit changes to reinforce the abandonment of ideas that the group has killed. Truly innovative companies deliberately celebrate failure as well as success. Leaders in these companies ensure that informal communications, as well as formal incentives, support cohesion and focus as a source of pride: “Our company knows where it’s going, and we’re going there together.” These leaders also carefully plan for reallocating resources after killing an idea. The “funeral” for any failed project resembles a celebration rather than a somber mass. And the lessons from failed projects — including both technical breakthroughs and insights about the market — are applied in other projects.

Innovation and the Bottom Line

Even in a recession, innovation should remain a priority. A Booz & Company study conducted jointly with the San Francisco Bay Area Council Economic Institute found that although companies in the area have cut costs, they continue to make key investments in R&D. This region is, of course, known for innovation; it hosts the corporate headquarters of many technology-based companies and attracts 43 percent of the venture-capital investments in the United States. But the study also suggested that innovative companies anywhere may be relatively well positioned in the current recession, at least compared to other companies. They have generally acted quickly to reduce costs and conserve cash, and they are already placing strategic bets on new technologies to create advantaged positions once recovery emerges.

In short, when innovation investment is productive, it can position a company for success in a variety of ways, not just in bringing new offerings to market. By carefully integrating the informal and formal organization along critical dimensions, all companies can yield more benefits from their existing innovation resources.

Reprint No. 09303

Author profiles:

  • Zia Khan is a Booz & Company partner based in San Francisco. He has expertise in helping clients build the formal and informal organizational capabilities needed to realize major transformations. He previously led the San Francisco office of Katzenbach Partners LLC.
  • Jon Katzenbach is a Booz & Company senior partner based in New York. A cofounder of Katzenbach Partners LLC, he is the author of Peak Performance: Aligning the Hearts and Minds of Your Employees (Harvard Business School Press, 2000) and Why Pride Matters More than Money: The Power of the World’s Greatest Motivational Force (Crown Business, 2003). He has expertise in helping clients mobilize their informal organization to realize performance advantages.