Learning How to Test and Learn

A test-and-learn philosophy recognizes that the best way to understand the potential of an innovative idea is to try the idea. The company can take on more experiments and improve its odds of strategic impact if it learns how to rapidly identify and evaluate an opportunity either to deliver on it or move on to the next idea.

We have identified three principles around which test-and-learn capabilities can be built. This briefing describes each principle and discusses how Deutsche Telekom (DT)[foot]This research briefing is based on N. Fonstad, “Reorganizing to Innovate More Competitively: Key Lessons from Deutsche Telekom,” MIT Sloan CISR Working Paper No. 426, January 2018.[/foot] has applied these principles to increase the strategic impact
of its digital innovations.

PRINCIPLE 1: ARTICULATE A VISION THAT FOCUSES AND GUIDES INNOVATION EFFORTS

Top-performing companies have a clear vision of how digital innovation priorities enact digital strategy.[foot]MIT CISR Survey on Digital Innovation, September–November 2016 (N=201); respondents were CIOs and equivalents, mostly from companies based in the United States and Europe.[/foot] To strategically focus business innovations, leaders need to convert that vision into a small set of targeted innovation areas. In doing so, they establish the parameters for test and learn: Teams can formulate hypotheses about how a given innovation relates to business objectives, test the results of early experiments and prototypes, and then adjust and prioritize subsequent investments accordingly.

When Tim Höttges became chief executive officer of Deutsche Telekom AG in 2014, he initiated a corporate strategy that consisted of two business models, one focused on businesses, the other on consumers. He proposed that DT would set itself apart from competitors by reinventing its core business to provision better-than-average network quality for business customers and a richer-than-average set of digital offerings for consumers.[foot]Via its new strategy, DT intended to operate as a modular producer with business customers, providing a service that could adapt to any number of ecosystems; and as an ecosystem driver for consumers, providing an open platform with interfaces that would allow partners to integrate and increase value creation. Learn more about these and other business models in P. Weill and S.L. Woerner, “The Next Generation Enterprise: Thriving in an Increasingly Digital Ecosystem,” MIT Sloan CISR Research Briefing, Vol. XIII, No. 4, April 2013.[/foot] Höttges then hired Christian von Reventlow as chief product and innovation officer (CPIO) to re-launch innovation at Deutsche Telekom.

Previously, in its rush to become more digital, DT had over- looked the importance of orchestrating synergies across multiple innovation projects. An internal review of those innovations revealed that it was too easy to start—and too difficult to stop—an experiment. The company was investing in sixty-four different business areas, and only a few were creating competitive value. Some innovations reached the market after competing products; others never made it to market at all. Senior management and the Board of Management were distressed by the lack of transparency into innovation investments.

Such honed focus has enabled the company to make much faster progress on its most strategic concerns. Some innovation projects have been sold (e.g., tolino eReader), and others discontinued (e.g., payments). DT reallocates resources to target innovation areas, and even more strategically, to innovations that cross target areas.

PRINCIPLE 2: FOSTER EXPERIMENTATION THROUGHOUT THE ORGANIZATION

The rapid arrival of new and enhanced digital technologies makes it difficult to anticipate customer needs and how to address those needs. Thus, leaders must continuously reassess and make sense of their opportunities. To accelerate the sensemaking process, companies need constant, widespread experimentation that helps employees both identify opportunities and learn how to act on them.

In a test-and-learn organization, employees recognize a strategic objective, hypothesize how to achieve that objective, develop an experiment to test that hypothesis, measure the results of the experiment, and then adapt the experiment or discard it and start anew.

To foster experimentation, DT made two changes to funding of push innovations, both inspired by venture capital practices.[foot]Microsoft has been developing a test-and-learn environment by building data communities and linking them with non-data communities (see I.A. Someh and B.H. Wixom, “Data-Driven Transformation at Microsoft,” MIT Sloan CISR Research Briefing, Vol. XVII, No. 8, August 2017). Spotify has embraced test and learn in its agile teams (see A. Baiyere, J.W. Ross, and I.M. Sebastian, “Designing for Digital: Lessons from Spotify,” MIT Sloan CISR Research Briefing, Vol. XVII, No. 12, December 2017).[/foot] First, the company created an alternative to its traditional funding track. The traditional track funds innovations once a year based on detailed business cases, with funding available only to innovation-oriented units. Via the alternative flexible funding track, any team in the company can pitch a project during a monthly meeting of DT’s Portfolio and Innovation Board. This board, co-chaired by the CPIO and the senior vice president for strategy, assesses pitches in relation to areas of innovation linked to the company’s strategy.

Second, rather than receiving funding all at once, a project approved by either funding track now receives funding in stages. The first stage of funding is comparable to a venture capital Series A round, after which a cross-functional investment committee assesses whether the project has achieved pre-determined tangible, customer-centric milestones. Tangible results might be a minimum viable product, a prototype, use cases presented at a trade fair, a live demo running in network, a proof of concept for a new technology, or a pilot successfully launched in one country.

If a team has produced such results, it qualifies for the next round of funding, a quasi-Series B round. If not, the project is stopped. This approach has allowed funding of riskier projects by making it easier to “fail fast and cheap” while generating insights from failed experiments.

In the subsequent year, based on lessons learned, modifications are made to the specific objectives and areas of technologies that push innovations explore.

PRINCIPLE 3: DEEPEN ENGAGEMENT TO COORDINATE INTERDEPENDENCIES

One of the toughest challenges for today’s companies is to coordinate interactions across their growing and increasingly diverse portfolio of innovations. When companies do not coordinate these interactions, they put their operations and customer experience at risk. By coordinating interactions well, companies achieve significant synergies; the total impact of the company’s innovations when coordinated is greater than their individual returns summed.

Companies learning to realize synergies are involving in the innovation process stakeholder groups that will be affected by an innovation and its interactions with other innovations, both existing and anticipated.

Deep engagement ensures that those with a stake in the outcome are involved throughout the innovation process and that they share responsibilities for both the success of individual innovations and for synergies across them. With the right stakeholders actively collaborating throughout the innovation process, the team can more efficiently explore more facets of a problem and solutions, including interdependencies within and across innovations. Consequently, those involved are more capable of selecting and developing innovations with greater relevance and impact.

DT has introduced a number of changes to broaden and deepen engagement of business leaders, board members, and partners in the company’s innovation efforts. For instance, the company has mandated that innovation projects be led by a cross-functional team comprising three senior-level managers, one from a central corporate unit responsible for evangelizing push innovations and two from a business segment responsible for market aspects. All three are accountable for developing insights from experiments and recommending next steps based on prior results.

In another example, company executives and board mem- bers have become more involved in managing DT’s portfolio of innovation projects. In January 2017, a Board of Management department called Technology and Innovation was created—overseen by board member Claudia Nemat—that incorporated the central unit responsible for push innovations. The department united those working on technology, IT, networks, and innovation in one group. One executive stated that this has resulted in fewer silos and more synergies based on common targets. In addition, throughout 2017 Nemat served as a permanent host on the Investment Committee, deepening the role of the Board of Management in innovation at DT and ensuring that the board would help to create better conditions for developing 4+1 projects.

TEST AND LEARN TO ENHANCE COMPETITIVENESS

The new innovation approach at DT has assured leaders of their ability to drive and track strategic benefits from digital innovation. As a result, by the end of 2017 the Board of Management had planned to increase the budget for its 4+1 innovation areas by 40%.

Figure 1 summarizes these three principles for building test- and-learn capabilities in support of a strategic digital innovation portfolio. Initially, test and learn leads to better results on individual projects. Ultimately, a test-and-learn culture makes a company adaptable. We expect that test and learn will distinguish those companies consistently able to see and respond to the business opportunities and challenges that rapidly changing digital technologies present.
 

© 2018 MIT Sloan Center for Information Systems Research, Fonstad and Ross. MIT CISR Research Briefings are published monthly to update the center's patrons and sponsors on current research projects.