Close Cookie Notice

Welcome to the MIT CISR website!

This site uses cookies. Review our Privacy Statement.

Red briefing graphic
Research Briefing

Learning How to Test and Learn

We’ve identified three principles around which test-and-learn capabilities can be built to counter pervasive uncertainties in business and technology.
Abstract

To counter pervasive uncertainties in the business and technology environment and prioritize the most strategic innovation projects from myriad options, companies can take a test-and-learn approach to innovation investments. From our analysis of top-performing firms, we have identified three principles around which test-and-learn capabilities can be built: articulate a vision that focuses and guides innovation efforts; foster experimentation throughout the organization; and deepen engagement to coordinate interdependencies. This briefing describes each principle and discusses how one large firm, Deutsche Telekom, has applied these principles to increase the strategic impact of its digital innovations.

Access More Research!

Any visitor to the website can read many MIT CISR Research Briefings in the webpage. But site users who have signed up on the site and are logged in can download all available briefings, plus get access to additional content. Even more content is available to members of MIT CISR Patron and Sponsor organizations.

Digital technologies are presenting established companies with seemingly infinite opportunities to innovate. Opportunities are wide ranging, including operational improvements focused on increased profitability, enhancements to employee and customer experience, the development of new products and services, and even the creation of new business models.[foot]Learn about distinct types of digital innovation we identified and how competitive companies capitalize on them in N. Fonstad, “Designing a Competitive Innovation Portfolio,” MIT Sloan CISR Research Briefing, Vol. XVII, No.7, July 2017.[/foot] With so many options, how do business leaders identify the most strategic innovation projects?

The constant procession of powerful new digital technologies and the myriad business opportunities they enable can easily overwhelm traditional innovation investment approaches. The current pace of change in technologies and business conditions makes it difficult to anticipate which innovative ideas could have a strategic impact and which will fail to generate significant returns. To counter the uncertainties in the business and technology environment, particularly for innovations whose benefits are expected in the longer term (three to five years), companies can take a test-and-learn approach to innovation investments.

Via test and learn, a company develops the capability to rapidly identify and evaluate opportunities, enabling it to take on more experiments and improve its odds of strategic impact.

A test-and-learn philosophy recognizes that the best way to understand the potential of an innovative idea is to try the idea. The company can take on more experiments and improve its odds of strategic impact if it learns how to rapidly identify and evaluate an opportunity either to deliver on it or move on to the next idea.

We have identified three principles around which test-and-learn capabilities can be built. This briefing describes each principle and discusses how Deutsche Telekom (DT)[foot]This research briefing is based on N. Fonstad, “Reorganizing to Innovate More Competitively: Key Lessons from Deutsche Telekom,” MIT Sloan CISR Working Paper No. 426, January 2018.[/foot] has applied these principles to increase the strategic impact
of its digital innovations.

PRINCIPLE 1: ARTICULATE A VISION THAT FOCUSES AND GUIDES INNOVATION EFFORTS

Top-performing companies have a clear vision of how digital innovation priorities enact digital strategy.[foot]MIT CISR Survey on Digital Innovation, September–November 2016 (N=201); respondents were CIOs and equivalents, mostly from companies based in the United States and Europe.[/foot] To strategically focus business innovations, leaders need to convert that vision into a small set of targeted innovation areas. In doing so, they establish the parameters for test and learn: Teams can formulate hypotheses about how a given innovation relates to business objectives, test the results of early experiments and prototypes, and then adjust and prioritize subsequent investments accordingly.

When Tim Höttges became chief executive officer of Deutsche Telekom AG in 2014, he initiated a corporate strategy that consisted of two business models, one focused on businesses, the other on consumers. He proposed that DT would set itself apart from competitors by reinventing its core business to provision better-than-average network quality for business customers and a richer-than-average set of digital offerings for consumers.[foot]Via its new strategy, DT intended to operate as a modular producer with business customers, providing a service that could adapt to any number of ecosystems; and as an ecosystem driver for consumers, providing an open platform with interfaces that would allow partners to integrate and increase value creation. Learn more about these and other business models in P. Weill and S.L. Woerner, “The Next Generation Enterprise: Thriving in an Increasingly Digital Ecosystem,” MIT Sloan CISR Research Briefing, Vol. XIII, No. 4, April 2013.[/foot] Höttges then hired Christian von Reventlow as chief product and innovation officer (CPIO) to re-launch innovation at Deutsche Telekom.

Previously, in its rush to become more digital, DT had over- looked the importance of orchestrating synergies across multiple innovation projects. An internal review of those innovations revealed that it was too easy to start—and too difficult to stop—an experiment. The company was investing in sixty-four different business areas, and only a few were creating competitive value. Some innovations reached the market after competing products; others never made it to market at all. Senior management and the Board of Management were distressed by the lack of transparency into innovation investments.

Von Reventlow introduced a framework called the “4+1 model” that directs innovation efforts to four areas of immediate strategic importance plus one area representing a longer-term bet. Of the first four innovation areas, two target the development of new products and services, with one each for consumers and business customers. The other two areas focus on improving internal capabilities: the company’s business processes and its own network/infrastructure. The fifth or “+1” innovation area is chosen for its potential to influence the other four areas (5G technology in 2017).

Each innovation area has a strategic objective. For example, the core objective of the consumer products area is to become the “trusted companion for the digital life of our customers.” Regarding longer-term innovations—which the company refers to as “push innovations”[foot]Push innovations are developed by DT to leverage a promising emerging technology for which a customer demand is only anticipated. They are “pushed” to the company’s regional business units. Benefits are expected in three to five years.[/foot]—DT annually tightens the focus in each area to a specific set of innovations and technologies. For example, the 2017 consumer products focus for push innovations sought to provide a superior convergent products-and-service experience at the customer’s home by enabling cross-functional use cases across IPTV/ Entertainment, Smart Home, and Smart Voice Hub.

Initially, test and learn leads to better results on individual projects. Ultimately, a test-and- learn culture makes a company adaptable.

Such honed focus has enabled the company to make much faster progress on its most strategic concerns. Some innovation projects have been sold (e.g., tolino eReader), and others discontinued (e.g., payments). DT reallocates resources to target innovation areas, and even more strategically, to innovations that cross target areas.

PRINCIPLE 2: FOSTER EXPERIMENTATION THROUGHOUT THE ORGANIZATION

The rapid arrival of new and enhanced digital technologies makes it difficult to anticipate customer needs and how to address those needs. Thus, leaders must continuously reassess and make sense of their opportunities. To accelerate the sensemaking process, companies need constant, widespread experimentation that helps employees both identify opportunities and learn how to act on them.

In a test-and-learn organization, employees recognize a strategic objective, hypothesize how to achieve that objective, develop an experiment to test that hypothesis, measure the results of the experiment, and then adapt the experiment or discard it and start anew.

To foster experimentation, DT made two changes to funding of push innovations, both inspired by venture capital practices.[foot]Microsoft has been developing a test-and-learn environment by building data communities and linking them with non-data communities (see I.A. Someh and B.H. Wixom, “Data-Driven Transformation at Microsoft,” MIT Sloan CISR Research Briefing, Vol. XVII, No. 8, August 2017). Spotify has embraced test and learn in its agile teams (see A. Baiyere, J.W. Ross, and I.M. Sebastian, “Designing for Digital: Lessons from Spotify,” MIT Sloan CISR Research Briefing, Vol. XVII, No. 12, December 2017).[/foot] First, the company created an alternative to its traditional funding track. The traditional track funds innovations once a year based on detailed business cases, with funding available only to innovation-oriented units. Via the alternative flexible funding track, any team in the company can pitch a project during a monthly meeting of DT’s Portfolio and Innovation Board. This board, co-chaired by the CPIO and the senior vice president for strategy, assesses pitches in relation to areas of innovation linked to the company’s strategy.

Second, rather than receiving funding all at once, a project approved by either funding track now receives funding in stages. The first stage of funding is comparable to a venture capital Series A round, after which a cross-functional investment committee assesses whether the project has achieved pre-determined tangible, customer-centric milestones. Tangible results might be a minimum viable product, a prototype, use cases presented at a trade fair, a live demo running in network, a proof of concept for a new technology, or a pilot successfully launched in one country.

If a team has produced such results, it qualifies for the next round of funding, a quasi-Series B round. If not, the project is stopped. This approach has allowed funding of riskier projects by making it easier to “fail fast and cheap” while generating insights from failed experiments.

In the subsequent year, based on lessons learned, modifications are made to the specific objectives and areas of technologies that push innovations explore.

PRINCIPLE 3: DEEPEN ENGAGEMENT TO COORDINATE INTERDEPENDENCIES

One of the toughest challenges for today’s companies is to coordinate interactions across their growing and increasingly diverse portfolio of innovations. When companies do not coordinate these interactions, they put their operations and customer experience at risk. By coordinating interactions well, companies achieve significant synergies; the total impact of the company’s innovations when coordinated is greater than their individual returns summed.

Companies learning to realize synergies are involving in the innovation process stakeholder groups that will be affected by an innovation and its interactions with other innovations, both existing and anticipated.

Deep engagement ensures that those with a stake in the outcome are involved throughout the innovation process and that they share responsibilities for both the success of individual innovations and for synergies across them. With the right stakeholders actively collaborating throughout the innovation process, the team can more efficiently explore more facets of a problem and solutions, including interdependencies within and across innovations. Consequently, those involved are more capable of selecting and developing innovations with greater relevance and impact.

DT has introduced a number of changes to broaden and deepen engagement of business leaders, board members, and partners in the company’s innovation efforts. For instance, the company has mandated that innovation projects be led by a cross-functional team comprising three senior-level managers, one from a central corporate unit responsible for evangelizing push innovations and two from a business segment responsible for market aspects. All three are accountable for developing insights from experiments and recommending next steps based on prior results.

In another example, company executives and board mem- bers have become more involved in managing DT’s portfolio of innovation projects. In January 2017, a Board of Management department called Technology and Innovation was created—overseen by board member Claudia Nemat—that incorporated the central unit responsible for push innovations. The department united those working on technology, IT, networks, and innovation in one group. One executive stated that this has resulted in fewer silos and more synergies based on common targets. In addition, throughout 2017 Nemat served as a permanent host on the Investment Committee, deepening the role of the Board of Management in innovation at DT and ensuring that the board would help to create better conditions for developing 4+1 projects.

TEST AND LEARN TO ENHANCE COMPETITIVENESS

The new innovation approach at DT has assured leaders of their ability to drive and track strategic benefits from digital innovation. As a result, by the end of 2017 the Board of Management had planned to increase the budget for its 4+1 innovation areas by 40%.

Figure 1 summarizes these three principles for building test- and-learn capabilities in support of a strategic digital innovation portfolio. Initially, test and learn leads to better results on individual projects. Ultimately, a test-and-learn culture makes a company adaptable. We expect that test and learn will distinguish those companies consistently able to see and respond to the business opportunities and challenges that rapidly changing digital technologies present.
 

Figure 1: Three Principles Around Which Test-and-Learn Capabilities Can Be Built
Principle Purpose Practice
Focused Vision Describes why and what to test and learn To realize the vision, link it to a small set of innovation areas. This establishes the parameters for test and learn.
Ubiquitous Experimentation Describes when and how to test and learn To encourage experimentation throughout the company, fund the idea, not the structure.
Deep Engagement Describes who tests and learns Insist that those with a stake in the innovation share responsibilities for both the success of individual innovations and for synergies across them.
 

© 2018 MIT Sloan Center for Information Systems Research, Fonstad and Ross. MIT CISR Research Briefings are published monthly to update the center's patrons and sponsors on current research projects.
 

About the Authors

Profile picture for user nilsfonstad@mit.edu

Nils O. Fonstad, Research Scientist, MIT Sloan Center for Information Systems Research (CISR)

E94-15th floor
(617) 253-2348
Profile picture for user jross@mit.edu

Jeanne W. Ross, Principal Research Scientist, MIT Sloan Center for Information Systems Research (CISR)

MIT SLOAN CENTER FOR INFORMATION SYSTEMS RESEARCH 

Founded in 1974 and grounded in the MIT tradition of rigorous field-based research, MIT CISR helps executives meet the challenge of leading dynamic, global, and information-intensive organizations. Through research, teaching, and events, the center stimulates interaction among scholars, students, and practitioners. More than ninety firms sponsor our work and participate in our consortium. 

MIT CISR Patrons
AlixPartners
Avanade
Axway, Inc.
Cognizant
Microsoft Corporation
Pegasystems Inc.
PricewaterhouseCoopers
Standard Bank Group (South Africa)
The Ogilvy Group
MIT CISR Sponsors
Allstate Insurance Company
Amcor
ANZ Banking Group (Australia)
Australian Taxation Office
AustralianSuper
Banco Azteca (Mexico)
Banco Bradesco S.A. (Brazil)
Banco do Brasil S.A.
Bank of Queensland (Australia)
BNP Paribas (France)
Bristol-Myers Squibb
Cabot Corporation
CarMax
Caterpillar, Inc.
CEMEX (Mexico)
Charles River Laboratories, Inc.
CIBC (Canada)
Cochlear Limited (Australia)
Commonwealth Superannuation Corp. (Australia)
Credit Suisse (Switzerland)
Cuscal Limited (Australia)
CVS Health
DBS Bank Ltd. (Singapore)
Doosan Corporation (Korea)
Fidelity Investments
Fomento Economico Mexicano, S.A.B., de C.V.
Fortum (Finland)
General Mills, Inc.
General Motors Corporation
Henkel AG & Co. (Germany)
Hitachi, Ltd. (Japan)
HSBC Technology & Services (USA) Inc.
Johnson & Johnson (J&J)
Kaiser Permanente
King & Wood Mallesons (Australia)
Koç Holding (Turkey)
Markel Corporation
Mercer
National Australia Bank Ltd.
Nomura Holdings, Inc. (Japan)
Nomura Research Institute, Ltd. Systems Consulting Division (Japan)
OECD
Pacific Life Insurance Company
Pioneer Natural Resources USA Inc.
Posten Norge AS (Norway)
Principal Financial Group
Procter & Gamble
QBE
Raytheon Technologies
Reserve Bank of Australia
Royal Philips (Netherlands)
Santander UK/Grupo Santander
SC Global Tubular Solutions
Scentre Group Limited (Australia)
Schneider Electric Industries SAS (France)
Scotts Miracle-Gro
SIGMAXYZ Inc. (Japan)
State Street Corp.
Stockland (Australia)
Suncorp Group (Australia)
Teck Resources Limited (Canada)
Tetra Pak (Sweden)
Trinity Health
Truist Financial Corporation
UniSuper Management Pty Ltd (Australia)
USAA
Webster Bank, N.A.
Westpac Banking Corporation (Australia)
WestRock Company
Wolters Kluwer
Zoetis Services LLC
Find Us
Center for Information Systems Research
Massachusetts Institute of Technology
Sloan School of Management
245 First Street, E94-15th Floor
Cambridge, MA 02142
 
617-253-2348