Digital Economy Dispatch #286 -- Seven Hard Lessons from One of AI's Toughest Operating Environments

Last week I wrote about Katrina Manson's Project Maven, the inside story of how a small team in a windowless Pentagon room set out in 2017 to put AI at the heart of how America fights its wars. It is a serious piece of reporting, drawn from more than two hundred interviews, and reviewers from The Economist to the New York Times have rightly placed it among the most important recent books on AI and conflict. But the more I consider it, the more I am struck by how the experiences described reveal important success factors for every leader as they seek to deliver AI-driven transformation.

Strip out the descriptions of targeting algorithms, drone footage, and rooms full of intelligence analysts, and what remains is a case study every digital leader will recognise. A small team trying to drag a vast organisation into a different way of working. A technology that promised more than it could initially deliver. Bureaucracy that fought back. Talent that came and went. A long argument about ethics that no one wanted to have but everyone had to. The names and the stakes differ from anything most of us work on, but the patterns will feel familiar to anyone who has tried to push a serious AI programme through a complex organisation.

There are seven lessons I think are worth pulling out.

1. Leadership sometimes means defying the system you serve

Manson's protagonist, Marine Corps Colonel Drew Cukor, was given a mandate by Deputy Defense Secretary Bob Work in the April 2017 memo establishing the Algorithmic Warfare Cross-Functional Team, but very little authority to deliver it. What Cukor and his small team had instead was focus, energy, and a refusal to accept that the Pentagon's procurement cycle, security culture, and turf wars were immovable. At several points their actions bordered on insubordination. The book makes clear this was not heroism for its own sake. It was the only way to get the work done in the timescales that mattered.

The lesson for digital leaders is not "break the rules". It is that real AI delivery requires people willing to apply judgement against the grain of the organisation, and must include senior-level cover for such actions when they do. Most large enterprises and public bodies have no shortage of governance. What they lack is the small number of leaders prepared to push, with purpose, against settled assumptions.

2. The data problem is rarely the data problem you expect

The single biggest constraint on Project Maven was not algorithms. It was data: too little of it, badly labelled, locked up by classification rules, and contested between agencies. Manson told NPR that the early Project Maven models had been trained on images of wedding cakes, bridal veils, and grooms' suits before being repurposed for the battlefield, where they confused trees for people and a cloud for a school bus. Cukor himself said his AI was, in those early days, "just a bag of potato chips" to operators.

Anyone who has run an enterprise AI programme will recognise this. The hard work is not picking a model. It is finding data that is current, representative, and legally usable, then labelling it properly, then negotiating who is allowed to share what with whom. Privacy, ownership, and rights are not back-office issues that the lawyers will sort out later. They are first-order design constraints. Treating them as such early saves months of rework, and a great deal of avoidable embarrassment, later.

3. Experiment at the scale the problem requires

One detail in the book has stayed with me. The Project Maven team would, at peak, test more than 1,500 algorithms in order to deploy fewer than a dozen of them. That ratio is worth considering. It is not the typical organisational picture of AI: pick a model, prove a pilot, scale it up. It is closer to drug discovery. Most things you try will not work. Some will work, but not well enough. A small number will earn their place in production.

Most organisations are nowhere near set up for this. They run a handful of carefully curated pilots, almost all of which are declared successful, and then wonder why so little reaches the front line. The infrastructure question for the next phase of enterprise AI is not "do we have a platform". It is whether the organisation can afford, financially and culturally, to throw away ninety-nine per cent of what it builds. Without that capacity, pilot purgatory is more or less guaranteed.

4. The model is not the system

Perhaps the most important sentence in Manson's account of why Project Maven eventually started to work is the one that names three things, not one: the quality of the underlying data, the system in which the algorithm sat, and the smoothness of the workflow the operators could build around it. None of those was sufficient on its own. All three had to come together.

This matters because too many AI conversations in business and government still reduce to a debate about models. Which foundation model? Which vendor? Which open-source variant? Those choices matter, but they matter least. What changes outcomes is whether the model is embedded in a system that fits the workflow of the people using it, fed by data they can trust, in a form they can act on. AI value is a property of systems, not algorithms.

5. The vendor relationship is itself a strategic risk

A second-order story runs through Manson's book alongside the Pentagon's internal one: how a handful of private companies, most prominently Palantir and Amazon Web Services, alongside Microsoft, Anduril and others, became indispensable to a national security capability. Project Maven did not just buy algorithms from these firms. It built workflows around them, trained its people on their interfaces, and made operational decisions inside their environments. Palantir's growth in particular was supercharged by the engagement, and the company's commercial posture was anything but passive.

That dependency carries three risks every digital leader will recognise. The first is alignment: how tightly do you want your strategy and your data coupled to a single supplier? The second is commercial: aggressive vendors will press their advantage at renewal. The third, easily missed, is bias, not only in the algorithms but in the framing each vendor brings to what AI is for and how it should be used. Buying from a vendor is also buying into their worldview.

This is where "consolidate demand, diversify supply" earns its place. Concentrating buying power gives an organisation leverage. Concentrating supply takes it away. The lesson from Project Maven is not to avoid commercial partners, which is neither possible nor desirable. It is to design the relationship deliberately, with exit-by-design, multiple credible sources, and a clear-eyed view of who is the buyer and who is the seller.

6. Persistence is a core competency

Project Maven survived a presidential transition, a public revolt by Google employees that ended the company's involvement in 2018, multiple changes of leadership, sustained turf wars between agencies, and several rounds of internal political opposition. It also survived a great deal of personal animosity inside its own team. None of that is unusual. What is unusual is that the project kept going long enough to deliver anything.

Persistence is unfashionable as a leadership virtue. It is harder to put on a CV than "transformation", and harder to talk about on stage than "vision". But Manson's account is, more than anything else, the story of a programme that refused to die. The implication for digital leaders is uncomfortable. Many of the most important AI initiatives in your organisation will not succeed in their first form, their first business case, or under their first sponsor. The question is whether they have the kind of patient backing that lets them get to the second, third, and beyond.

7. "Human in the loop" can quietly become "human on the loop"

The most uncomfortable thread in Manson's book is the one that runs through its final chapters. Project Maven has always insisted that it does not pull the trigger. Its job is to identify possible targets in surveillance footage, sort them, rank them, and pass them to a human operator who decides what happens next. That distinction, between an AI that recommends and a human who decides, has been the public ethical bedrock of the programme since its founding.

But Manson's reporting makes that line look much less stable than it sounds. When the system delivers a prioritised list of targets with precise coordinates, ready to be fed into a weapons system, and when the operator works under time pressure to act on what the system surfaces, what kind of decision is the human actually making? At what point does "human in the loop", actively choosing, become "human on the loop", rubber-stamping what the machine has already framed?

This is not a question confined to military AI. Every digital leader running a system that ranks candidates for hiring, flags transactions for review, or prioritises cases for casework will recognise the same tension. The system does not decide. The system suggests. But if the human accepts the suggestion in case after case, who is really in charge? Project Maven forces the question into the open in the most uncomfortable possible setting. The work for the rest of us is to ask it of our own systems before someone else does.

Where this leaves us

Project Maven continues to evolve, now housed at the National Geospatial-Intelligence Agency. And the ethical questions Manson's book raises about lethal autonomy are real and unresolved. But for digital leaders trying to make AI work in their own organisations, the more useful reading is as a case study in delivery: how a small team, with weak formal authority, built something the wider system did not know how to build for itself, and kept building it long after the initial enthusiasm had faded.