Beware the AI Experimentation Trap

The MIT Media Lab/Project NANDA released a new report that found that 95% of investments in gen AI have produced zero returns. The headline is just the latest to feed a wave of skepticism that AI will produce results at scale.

Headlines like these worry leaders. If 95% of the tens of billions invested in experimentation has failed to produce value, is the effort to experiment with AI a complete waste? On the other hand, how will companies learn how to use these tools without running experiments? How are leaders supposed to interpret these results?

As researchers who study AI and teach about AI transformation and technology, we believe that many leaders are making the same mistake they made a decade earlier with digital transformation: encouraging experimentation, which is good, but falling into the trap of letting experimentation run wild, which is counterproductive. For context, in the previous wave of digital transformation, when many leaders felt confused about digital transformation and the path forward, they embraced innovation and experimentation. Leaders embraced a “let 10,000 flowers bloom” approach, hoping that a few experiments produced unicorn-level returns.

While experimentation is good, without a connection to the true business opportunity, experiments inevitably fall short of hopes and expectations. It sounds obvious, but by framing AI as radical and disruptive we often lose sight of the connection to the most fundamental objective of business: to solve problems for customers.

UNDERSTAND AI IN THE LARGER ARC OF TRANSFORMATION

Although the world is talking about AI right now, pull back the frame to recall that AI is a recent conversation that is part of a larger shift. The true change we are all wrestling with is a fundamental shift from digital technology operating at the periphery of organizations (e.g., IT was about laptops, Wi-Fi, printing, and IT databases for registry of core activities) to digital at the very core of organizations (e.g., an organization built around digital workflows and customer journeys rather than its own production activities). Said differently, in many senses, every company is becoming a technology company Rather than people performing tasks based on human judgment and intuition, we are moving to a world of data- and AI-driven decisions, overseen by humans but not necessarily with people as the core engine of the activity.

FOCUS ON SERVING CUSTOMERS BETTER

Recognizing that the point of AI is simply to serve customers better can help leaders focus on what they want to achieve through experimentation. The future is fundamentally uncertain. No one knows exactly where we’re headed and no one can give you a playbook (despite what they may claim). As such, experimentation is essential to figuring out how to use AI to transform the company to serve customers better.

In these moments of strategic uncertainty, where many lack focus and direction, it is tempting to adopt a 10,000 flowers approach. But this scatted approach doesn’t usually address the core need. We see so many leaders swamped by headlines about the miracles of gen AI miss the true potential of AI transformation in exactly the same way they missed the point of digital transformation. The headlines about disruption and revolution create panic and distraction.

Your company isn’t going to become Meta, Amazon or Google, and that shouldn’t be your goal anyway. Quit stressing about some company you read about that supposedly fired 80% of its workforce to become AI-ready. The real opportunity — the one that will actually generate returns — is to look carefully at your internal operations and the external customer journey and start with how you can create real value, in the near term, using AI tools.

Use this lens to focus your experiments. Start with your strategy, and the value-creating opportunities to either transform a core activity or serve new customers better. Then explore the experiments that will prove out the best way to do this. As it turns out, this is one of the key, if underreported findings of the MIT Media Lab findings: that leaders are pursuing more cosmetic applications in marketing while avoiding transforming the core.

RUN EXPERIMENTS TO PROVE VALUE WITH AN EYE TO FUTURE SCALING

How should experiments proceed? They should check three boxes: 1) connected to true value creation, 2) as low cost as is possible so that you have multiple cycles to learn and improve, and 3) designed with a connection to how they could eventually scale to create value. This sounds simple but is hard to do in practice. At one end of the spectrum, many leaders charge ahead with experiments without thinking about how they could scale. At the other end are leaders getting bogged down, obsessing about being enterprise-ready from Day One.

The magic is to strike a balance: Is this a big enough problem to solve? And, if solved, what would it take to scale the solution? To help you assess which problems to solve among the many that might be proposed, we often use our IFD framework: intensity (how intense is the problem), frequency (how frequently does it occur), and density (how many users or instances of the problem occur). This gives you a good method to weigh the value of ideas that solve different problems.

SCALE UP WITH A NINJA TEAM

Once you prove out the value of the experiment, then you can scale it up, but doing so requires careful attention to scaling. Moving something from experiment to scale almost always comes with unexpected new challenges. Someone with power to create change needs to own the initiative, leading a team with time and resources to make change happen. These “ninja” scaling teams — we have interviewed teams at Amazon, Qualtrics, 7-Eleven and elsewhere — have the support from the senior leadership, companywide connections to get resources and focus to scale.

As you reach the appropriate scale for the project, repeat the process. Of course, there is more to this to transform at enterprise scale, which includes laying the right digital foundations — a transformation the vast majority of companies have not truly completed. But this process outlines the core machine of the transformation that can help companies avoid the waste and tap into the real value of this fundamental transformation.

+++

As we enter the post-enthusiasm wave of AI, many leaders are in danger of misinterpreting the challenges implementing AI as a signal that AI cannot be used to create value. They run the risk of falling behind on their AI transformation, just as so many leaders fell behind on their digital transformation (and are now playing catch up).