Digital Economy Dispatch #201 -- Why AI Fails

Digital Economy Dispatch #201 — Why AI Fails
15th September 2024

Managing technology change in large complex organizations has always been fraught with difficulty. The introduction of digital technology is no different. Take a look at news stories and commentaries of large-scale IT systems delivery over the past 50 years and it’s depressingly easy to find examples of IT system delays, cost overruns, and poor performance.

While there are many factors that contribute to these failures, one of the most fundamental issues is bridging the gap between the excitement and expectation created by a technology-driven change and the sober reality of delivering business-focus results.  A PWC Pulse survey of over 500 executives in 2023 reported that almost nine out of 10 respondents admit they struggle to measure the return on investment in new technology. It concluded that executives tend to “lose sight of the ultimate business outcome objective, and end up tempted by the latest bright, shiny objects”.

It’s a challenge that seems to be re-emerging with the adoption of AI. When measuring project success, many AI initiatives overemphasize technological performance metrics and underplay business impact. Hence, while they assiduously report on issues such as the size and complexity of AI models and model accuracy, they are unable to evaluate the direct business impact of AI applications on their company’s key performance metrics.

What are the key measures of success in AI-at-Scale? What are factors are getting the way of improving AI delivery success?

Measuring Enterprise AI Success

As AI tools and technologies become more widely available, we are seeing them being adopted in enterprise settings across many domains. In some cases, much of what is taking place can be seen as simply an extension of existing projects and programmes to introduce more digital technologies into the workplace to drive efficiencies and improve service delivery. From this perspective, success with AI adoption is intimately tied to traditional IT delivery methods and practices. Seen in this light, perhaps traditional success measures for IT projects are sufficient for evaluating your AI-at-Scale efforts.

However, reviewing the nature of AI-driven solutions highlights additional aspects that require consideration. In many situations, delivering successful AI-at-Scale brings a new set of pressures on organizations. They now need to cope with different project characteristics, such as scarce and expensive labour costs, intensive capital equipment requirements, significant reinvestment in new governance structures and legal frameworks, and complex maintenance needs due to reliance on extensive data sources and high algorithm complexity. This combination undoubtedly adds cost and risk beyond on-going digital change programmes and traditional information system upgrades. To understand the scale of the challenges this brings, take a look at MIT’s AI risk repository. It identifies and tracks over 700 categories of risk with AI systems.

Given this context, it is perhaps no surprise that the business results from AI are being brought into question. Indeed, Iavor Bojinov, a Fellow at Harvard Business School, goes as far as to suggest that most AI projects fail. He claims that the failure rate can be as high as 80%—almost double the rate of corporate IT project failures a decade ago. While such anecdotal figures can be questioned, it is critical that we spend some time to explore experiences in enterprise Ai adoption to understand more about how to succeed with AI-at-Scale.

The 5 AI Failure Patterns

Fortunately, several initiatives are now examining AI projects across different domains to understand more about why they often stumble and the factors that contribute to their success or failure. In one of the most rigorous of these reviews, RAND researchers have delved into AI failures by interviewing a wide range of experts from both industry and academia. Through this work, five key reasons for AI project failure emerged.

Misunderstanding or Miscommunicating the Problem

Often, the first stumbling block occurs before a single line of code is written. Stakeholders may not clearly define or communicate the problem they're trying to solve with AI. Without a precise understanding of the challenge at hand, even the most sophisticated AI solution can miss the mark.

In response, the report suggests successful AI projects require a strong alignment between technical teams and business objectives. Industry leaders must ensure that technical staff fully understand the project's purpose and domain context to avoid miscommunications and project failures.

Insufficient or Inadequate Data

AI models are only as good as the data they're trained on. Many projects falter because organizations lack the necessary quantity or quality of data to train an effective model. Remember: garbage in, garbage out!

The challenge, as the report highlights, is to maintain data quality throughout the life of the AI systems. This requires significant investment in data skills, infrastructure, and governance models.

Prioritizing Cutting-Edge Tech Over Practical Solutions

It's easy to get caught up in the excitement of the latest AI breakthroughs. However, focusing on using the newest, shiniest technology rather than addressing real user needs is a recipe for failure. The most advanced AI isn't always the best solution for every problem.

As the report states, successful AI projects demand strategic focus and long-term commitment. Industry leaders should identify enduring problems that align with organizational goals and commit product teams to solving them for significant period of time.

Inadequate Infrastructure

Building an AI model is only part of the challenge. Organizations need robust infrastructure to manage their data and deploy completed models effectively. Without this backbone, even promising AI projects can stumble at the implementation stage.

Here, the authors point out that strategic investments in infrastructure can yield significant returns on AI projects. By supporting data governance and model deployment, these investments can reduce project timelines and improve data quality, ultimately leading to more valuable AI outcomes.

Tackling Problems Beyond AI's Current Capabilities

Sometimes, enthusiasm for AI leads to unrealistic expectations. Some problems are simply too complex or nuanced for current AI technology to solve effectively. It's crucial to understand the limitations of AI and choose projects accordingly.

According to the authors, the way forward is for industry leaders to invest in education at all levels to understand AI's capabilities and limitations. Furthermore, when considering a potential AI project, leaders need to include a variety of viewpoints, including technical experts, to assess the project's feasibility and ongoing value.

Lessons From Failure

The adoption of AI in large organizations is often fraught with challenges, mirroring the historical difficulties faced with large-scale IT system implementations. While there are numerous factors contributing to these failures, a fundamental issue lies in bridging the gap between the excitement surrounding AI and the practical realities of delivering business-focused results.

Despite the growing prevalence of AI tools and technologies, many organizations struggle to measure the return on investment in these initiatives. Success in AI adoption is closely tied to traditional IT delivery methods, but the unique characteristics of AI-driven solutions introduce additional complexities and risks.

Research indicates that a significant portion of AI projects fail due to various factors, including misaligned goals, insufficient data, overemphasis on technology, inadequate infrastructure, and tackling problems beyond AI's current capabilities. To increase the likelihood of success, organizations must prioritize clear communication, invest in data quality and infrastructure, focus on practical applications, and understand the limitations of AI technology. By addressing these challenges, businesses can harness the potential of AI to drive innovation and achieve tangible business outcomes today…and into the future.