Guide · Bots & AI
AI hallucinations and business risk: keeping assistants honest
Language models generate fluent text by predicting likely words. Most of the time, the result is useful. Sometimes it is confidently wrong — an invented price, a non-existent policy, a promise the business never made. This is often called hallucination. For a customer-facing assistant, a single invented commitment can create a dispute, a complaint or a legal problem. This guide explains the risk and the controls that reduce it.
[ 01 ]
In short
In short
- 01
Hallucinations cannot be eliminated completely, but they can be reduced substantially and their impact limited.
- 02
Ground answers in your own sources (retrieval-augmented generation) and instruct the model to answer only from them.
- 03
Make "I do not know — let me connect you with a person" an acceptable and expected answer.
- 04
Prevent commitments: prices, discounts, deadlines and legal statements only from approved data.
- 05
Test systematically before launch and review conversations after it.
[ 02 ]
Guide
Why it happens
A language model does not look up facts; it produces text that fits the context. When the context lacks the answer, it may still produce a plausible one. Outdated training data, ambiguous questions and pressure to be helpful make this more likely. See LLM.
Business risks
- Wrong information: prices, availability, delivery terms, opening hours.
- Unauthorised commitments: discounts, refunds, guarantees, deadlines.
- Legal and medical statements the business cannot make.
- Reputational damage when screenshots of wrong answers circulate.
- Data exposure if the assistant is tricked into revealing information it should not.
Controls that work
Grounding
The assistant retrieves relevant passages from your knowledge base and answers from them. The model is instructed not to use outside knowledge for business facts. See RAG and preparing a knowledge base.
Clear refusals
Instructions and examples teach the assistant to say when the sources do not contain the answer, and to offer a human. Measure how often it does this; an assistant that never says "I do not know" is probably guessing.
Structured data for facts
Prices, stock, order status and schedules come from systems through lookups, not from the model's text generation. The model presents the data but does not invent it.
Restricted topics
Topics outside the business, legal and medical advice, and negotiations are declined or handed over.
Citations
Answers include links to the source page or document, so customers and staff can verify.
Guardrails against manipulation
Users may try to make the assistant ignore its instructions ("prompt injection"). Limit what the assistant can access and do, validate outputs for sensitive actions, and never let the model alone execute payments, refunds or data changes without checks.
Human review
For high-stakes situations — complaints, large orders, legal questions — the assistant drafts and a person approves, or the conversation goes directly to a person.
Testing
Before launch, build a test set of real questions, including tricky ones: questions without answers in the sources, ambiguous questions, attempts to get discounts, questions in each language. Check answers for accuracy and refusals. Repeat after every significant change to sources, instructions or model.
After launch
- Review a sample of conversations regularly.
- Track unanswered questions and fill knowledge gaps.
- Provide an easy way for users to report wrong answers.
- Keep logs, with personal data handled according to your privacy policy.
Transparency
Tell users they are talking to an AI assistant, what it can help with and how to reach a person. Clear expectations reduce frustration and risk.
[ 03 ]
FAQ
Frequently asked questions
Can a newer model solve hallucinations?
Newer models are generally more accurate, but no model is fully reliable without grounding and controls. Design for the possibility of errors.
Should we avoid AI for customer service?
Not necessarily. For structured tasks, scripted flows are safer; for open questions, AI with the controls above can help many customers quickly. See AI chatbot vs scripted bot.
Who is responsible for what the assistant says?
The business that deploys it. That is why commitments must come from approved data and why sensitive cases go to people.