Why does my AI chatbot answer wrong?
The model gets blamed almost every time, and it is almost never the model. Before switching providers, work out which of the two steps broke the answer, because the fix is completely different.
Key takeaways
A bad answer breaks in one of two places: the system never found the information (retrieval failure) or it found it and answered beyond it (faithfulness failure). The fix differs in each case.
The test costs an afternoon: thirty real questions, checked by hand for whether the correct passage appeared among what was retrieved. If it did not, the model is not your problem.
The retrieval metric is recall@k; the answer metric is groundedness. Without those two numbers, quality arguments get settled by whoever talks loudest.
The most frequent cause of bad retrieval is chunking: passages split by character count rather than by meaning, leaving the question in one chunk and the answer in another.
Four steps to locate the failure
- Collect thirty real questionsFrom conversations that already happened, not invented ones. With the correct answer written by someone who knows the business.
- Look at what the system retrievedFor each question, check whether the passage containing the answer appeared in what was handed to the model. That is recall@k by hand.
- Split them into two pilesQuestions where the correct passage never arrived: retrieval failure. Questions where it arrived and the answer was still wrong: generation failure.
- Fix the big pile firstIn most systems that reach a diagnostic, the retrieval pile is the big one. Swapping models does not move that pile by a single point.
Seven concrete failures and what gives them away
| Symptom | Likely cause | What to do |
|---|---|---|
| Answers with another product or customer data | Fixed-size chunking split the answer across two passages | Rechunk by document structure, with overlap |
| Finds the obvious and misses synonyms | Vector search only, no keyword matching | Hybrid search: vectors plus BM25 |
| Retrieves related passages but not the right one | No reranking after search | Add a reranker over the top results |
| Invents data with total confidence | The prompt does not require grounding or allow refusal | Explicit refusal instruction and source citation |
| Answered well, then quietly got worse | New documents landed and nobody re-measured | Scheduled re-run of the evaluation set |
| Different answers to the same question | High temperature, or carried-over conversation context | Lower temperature, bound the history sent |
| Slows down or falls over at peak | No rate limits, no cache, no retry with backoff | Cache frequent queries and control concurrency |
Splitting documents by size breaks answers
The fastest way to stand up a retrieval system is to split every document into thousand-character chunks. It works in the demo and fails in production for a simple reason: real documents have structure, and that structure does not fall every thousand characters.
A pricing table split in half leaves the headers in one chunk and the figures in another. A policy clause gets separated from its exception. When a user asks about the exception, the system retrieves the clause and answers exactly the opposite of what applies.
Chunking by structure (sections, clauses, whole table rows) with some overlap between passages usually moves recall more than any model change. It is boring work and it is where the return is.
Five things a production assistant must have
- When it cannot find something, it says so and escalates. A system that always answers is inventing in some share of cases.
- Every answer should be able to show which document it came from. That is what makes auditing possible without reading code.
- It is the highest-value deliverable of monthly operation: it says what the knowledge base is missing and what customers are actually asking.
- When it hands off to a person, that person receives the whole conversation, not a notification.
- Without that number you cannot decide whether a cheaper model would hold, or when volume stops being profitable.
If what was delivered is a configuration inside a vendor platform, with no access to prompts or logic, there is little to diagnose: what you can change is whatever that platform lets you change.
Also when the problem is the source. If the documents the system answers from are out of date or contradict each other, no technical adjustment fixes it. That work is editorial, not engineering.
Frequently asked questions
Why does my chatbot make things up?
For two reasons that get confused with each other. The first is that it never received the right information: the passage that mattered never reached the prompt and the model answered with what it had. The second is that it did receive it, but the prompt neither requires grounding nor allows the model to say it does not know. The first is fixed in retrieval; the second with an explicit refusal instruction and source citation.
What is recall@k and why does it matter?
It measures how often the passage containing the correct answer appears among the k the system retrieved. If recall@5 is 0.6, then in four out of ten questions the model never saw the information it needed. No change of model, prompt or temperature improves that number: it is a search problem.
Will a better model fix bad answers?
Only if the failure is in generation. When the problem is retrieval, a better model produces better-written answers that are just as wrong, and sometimes worse: a more capable model fills gaps more elegantly. That is why you split the two piles before touching anything.
What does it cost to diagnose a chatbot already in production?
A serious diagnostic is five days of work and ends with concrete findings, each with its fix and estimated effort. At Quarl, if the diagnostic does not produce at least three actionable findings, it is not charged.
How often should it be re-measured?
Monthly if the document base changes often, quarterly if it is stable. Degradation gives no warning: new documents land, the distribution of what is indexed shifts, and recall drops without anyone noticing until a customer complains. Re-running the evaluation set turns that surprise into a number you can see coming.