Why do AI projects fail?
The demo worked. The meeting went well. Six months later the system is still in staging and nobody wants to be the person who says it should be switched off. That path has repeated causes, and none of them is the model.
Key takeaways
MIT measures that 95% of AI pilots return nothing you can put a number on, across 52 executive interviews, 153 surveys and 300 public deployments. The industry problem is not adopting AI: it is sustaining it.
RAND breaks the failure down, and the uncomfortable number is the second one: 34% are abandoned before production, but another 28% do get built and still fail to deliver the expected value. That second group is the one running a live system that disappoints.
The most frequent cause is not technical: nobody defined what a correct answer was before starting. Without that criterion, "it works" is an opinion and the discussion never closes.
The second is cost per query discovered too late. A pilot at a hundred queries a day costs nothing; the same system at a hundred thousand changes the conversation with finance.
These numbers are not ours
They come from the market, each with a study behind it, and they explain why most implementations disappoint.
Seven ways to lose the project
- Nobody defined what a correct answer isWithout a set of real questions and their expected answers, every review is a matter of taste. The project does not die: it stalls in an endless round of tweaks.
- Cost per query showed up at the endThe pilot cost pennies. In production, with full context and ten times the volume, the monthly model bill exceeds whatever headcount it saved.
- Retrieval was bad and the model got blamedIf the passage that mattered never reached the prompt, no model can answer well. It is the most common failure and the easiest to mistake for hallucination.
- The system never learned to say it does not knowAn assistant that always answers with the same confidence means the first serious error destroys months of accumulated trust.
- There was no owner after handoverThe vendor delivered, the internal team had other priorities, and the system aged on its own. The document base changed and retrieval degraded with nobody watching.
- A process nobody had organised got automatedAI on top of a chaotic process produces chaos faster. If the rules are not clear to a person, the model will not infer them.
- A demo got bought instead of a systemBuilding the demo is a weekend. Sustaining it takes error handling, spend limits, traces, permissions and somebody on call. That is what is almost never in the quote.
The model is almost never the problem
When a system answers badly, the reflex is to swap models. Sometimes it helps; it usually masks the symptom. In most cases that reach a diagnostic, the model was receiving incomplete, contradictory or plainly wrong information, and answered with what it had.
Separating the two is cheap and takes an afternoon. Take thirty real questions, check by hand whether the correct passage appeared in what was retrieved, and the failure point becomes obvious. If retrieval misses, the work is in chunking, reranking and hybrid search. If retrieval hits and the answer is still wrong, then it is the prompt or the model.
Skipping that step is expensive: teams have switched model providers three times chasing a problem that lived in how the documents were split.
Six questions that reveal whether a project is on track
- Is there a test set of questions with their correct answers? How many?
- What is retrieval recall@k today, and what was it three months ago?
- What does one query cost, and what would it cost at ten times current volume?
- What does the system do when it cannot find the answer? Does it say so, or improvise?
- Who reviews, monthly, the questions the assistant could not answer?
- How long does a prompt change take to reach production, and who approves it?
A serious diagnostic takes a week and ends with concrete findings, each with its fix and its estimated effort. Not with a forty-slide deck.
If the diagnostic does not produce at least three actionable findings, we do not charge for it.
Frequently asked questions
What percentage of AI projects fail?
MIT measures that 95% of AI pilots produce no measurable return ("The GenAI Divide", 2025), and RAND that AI projects fail at roughly twice the rate of ordinary software projects (2024). RAND's breakdown is more useful than the headline: 34% are abandoned before production, 28% get built and never deliver the expected value, and 18% deliver something but not enough to justify the cost. The bottleneck sits between pilot and operation, not at initial adoption.
What is the most common reason a chatbot answers badly?
Retrieval, not the model. If the passage holding the answer never reached the prompt, the model answers with what it has and that reads as a hallucination. Before swapping models, review thirty real questions by hand and check whether the correct content was among what was retrieved.
Can a failed AI project be rescued?
In most cases yes, and it costs less than starting over. What gets salvaged is usually the integration, the permissions and the domain knowledge, which is where the time went. What gets rebuilt is usually document chunking, the retrieval strategy and error handling. The prior diagnostic is what separates a rescue from a rewrite in disguise.
How long does it take to move a pilot into production?
Four to eight weeks when the pilot was built well, and that work is almost never about the model: it is the evaluation set, per-user permissions, error handling, cost control, traces and an operating plan. When the pilot was built without any of those pieces, the route to production looks more like building it again with what the first attempt taught.
How do I stop it happening again?
Define the acceptance criterion before starting. Thirty to fifty real questions with their correct answers, written by someone who knows the business, and two target numbers: how often the system finds the information that mattered, and how often the answer stays inside that information. With that, any change can be evaluated in hours instead of argued about in meetings.