What are AI hallucinations, and what risks do they pose?
An AI hallucination occurs when a system presents a false or unsupported answer with confidence. It might invent a procedure, change a condition, or attribute a claim to a document that doesn't contain it.
The Junta de Andalucía's guide to generative AI includes examples of incorrect deadlines and nonexistent legal references. It also warns that configuration and tuning don't eliminate the risk entirely.
The damage depends on where the answer goes. An internal draft can be corrected before it goes out. An answer sent to a customer could promise a nonexistent discount. An automated action could enter incorrect data into a system.
An answer can sound reasonable and still be wrong. That's why we review both the content and the source that should support it. When AI works with business figures, separate this risk from the calculation problems we explain in why AI gets numbers wrong.
How can you reduce AI hallucinations in business?
Controls start before you deploy the system. This process lets you compare changes instead of relying on a general impression.
- Choose the highest-risk cases. Identify answers that could affect money, commercial terms, or decisions about people. Define what the system can answer and what needs review. If it can take actions, decide which ones need approval.
- Prepare reference questions and answers. Collect real questions from customers or employees. Include questions with clear answers, questions with no answer in the sources, conflicting documents, and ambiguous requests. For each case, record the expected answer or abstention and the approved source.
- Measure the baseline. Run the questions through the current configuration and save the answers. Classify each case as a correct answer, an error, a justified abstention, or an unnecessary abstention. An incomplete answer or an invented condition counts as an error.
- Change the controls and test again. You can improve the documents, adjust the search, or require the system to abstain when it lacks support. Repeat the test with the same questions and conditions. Set aside questions you haven't used to tune the system, and test those before deployment.
- Compare and decide. For each category, divide the number of cases by the total evaluated. Compare all four measures before and after the change. Review the highest-risk cases separately. Decide in advance which results the responsible team will accept and which errors prevent deployment.
How do you verify sources before answering?
Retrieval-augmented generation, or RAG, searches for information before drafting an answer. Google Cloud's RAG documentation suggests evaluating both the retrieved passages and the generated text. Having documents available doesn't guarantee that the system will find the right passage or interpret it correctly.
First, we check whether the search found a current, approved source. Then we check that every condition in the answer appears in that source. A visible citation helps with review, but it doesn't prove the cited text supports the answer.
Microsoft's evaluation library recommends checking figures, dates, names, and conditions against the source. It also warns that a plausible sentence can add information the document doesn't support.
In an illustrative case, a customer asks about a discount on an overdue invoice. The system searches the approved policy but finds documents with conflicting conditions. Instead of promising the discount, it says it can't confirm it and routes the question to the collections team.
The expected result is a referral with the question and the documents found, not an invented answer. If a current, clear policy exists, the system can propose an answer supported by the relevant passage. The team decides whether it needs review before sending.
Which option should you use to control responses?
The choice depends on the task and the controls your tool allows.
| Option | When it makes sense | What to review |
|---|---|---|
| Approved answer without AI | Questions have fixed answers. | Content owner and whether the content is current. |
| Feature in your existing tool | You already search documents from the platform you use. | Supported sources, citations, and evaluation options. |
| Automation platform | You need to connect questions to a review workflow. | Permissions, routing, and error logs. |
| Custom development | You need specific controls or complex connections. | Testing, maintenance, and limits on actions. |
Cost and timing depend on source quality, required connections, and the level of review. Permissions and evaluation work also affect them. If a tool doesn't let you test or limit what it does, consider another option.
What controls do you need after deployment?
The initial evaluation gives you a baseline. Documents and questions change, so the team needs to keep reviewing.
- Assigned source owners: assign someone to approve and update each document. Remove expired versions from search.
- Reviewable logs: retain the question, answer, and sources used. Limit access to people who need to review them. Avoid storing personal data you don't need.
- Abstention routing: decide which team receives questions with no supporting source or conflicting sources. Tell the customer that the answer is pending.
- Limited actions: separate proposing an action from executing it. For sensitive changes, require approval and verify the data in the relevant system.
- Regular testing: add detected errors to the evaluation and repeat it when the model, documents, or system connections change. If the system fails, suspend the affected function. Fall back to an approved answer or human review.