AI chatbots make things up because inventing and answering are, inside the model, the same operation. A language model predicts likely continuations of text. Ask it something well covered by its training data and the likely continuation is usually true. Ask it about your returns policy, which it has never seen, and the likely continuation is a plausible returns policy: fluent, specific, formatted like the real thing, and fictional.
The industry calls this hallucination, which is a gentler word than it deserves. Nothing inside the model separates recall from invention, so it cannot notice the difference and warn you. It has no gap to feel. That is why a wrong answer arrives in exactly the same tone as a right one.
The confident tone is the dangerous part
If wrong answers sounded wrong, this would be a small problem. Instead, fluency is unrelated to knowledge. The sentences are well formed whether or not the content behind them exists, and customers read fluency as authority — reasonably, because that is how it works with people.
Confidence scores do not rescue this. A model’s own estimate of its certainty is produced by the same machinery as the answer, and a percentage tells a customer nothing they can act on. We have written elsewhere about why source links beat confidence scores, and the argument comes down to this: a number invites you to trust, a document lets you check.
Three ways a support bot ends up wrong
Not all wrong answers have the same cause, and the fixes differ.
The first is the one everyone means by hallucination: the bot has no source material for the question and writes something anyway. This is the failure of an ungrounded bot, and it is the most damaging, because the invented answer often sounds more polished than the true one.
The second is a retrieval miss. The bot has your documents, searches them, and pulls the wrong passages — usually because the customer asked in vocabulary your content does not use. The answer is then confidently based on the wrong page.
The third is drift. The right passages are in front of the model and it still writes past them, adding a detail that felt likely. This is the quiet one. It survives most grounding claims, because the passages were correct and only the last step went wrong.
What it costs when it happens
In 2024 a Canadian tribunal ruled that Air Canada had to honour a bereavement discount its website chatbot had described to a customer, although the airline’s published policy said otherwise. The airline argued the chatbot was responsible for its own words. The tribunal disagreed, which is the sensible reading: a bot on your site speaks for you.
Most small businesses will never see a tribunal. They see the cheaper version instead — a refund honoured because the bot promised it, a customer who repeats the wrong answer to five other people, and the slow discovery that the support inbox is now cleaning up after the tool that was supposed to reduce it.
What actually reduces it
Grounding is the first and largest fix: restrict the bot to a defined set of your documents so that the raw material for invention is gone. It removes the first failure entirely and shrinks the third.
Verification after the answer catches most of the rest. Serochat asks the model to name the document behind its reply, then checks that name against the passages actually retrieved; if it names something that was never there, the answer is thrown away and the visitor is offered a human. A second check compares wording, and an answer that has wandered too far from its source is replaced with the handoff message.
An honest escalation route makes the remaining failures survivable. A bot that can say it does not know, and hand over the conversation with its context attached, turns a wrong answer into a slightly slower right one.
And better documents fix the retrieval misses that no vendor can fix for you. When a bot hands off, the question it could not answer is worth reading; it is usually pointing at a page you have not written.
What none of this makes go away
Grounding does not end wrong answers, and any vendor who tells you it does has just made the first hallucination of your relationship. A grounded bot repeats a stale policy page with complete conviction. Retrieval still misses. Verification catches inventions, not inaccuracies faithfully copied from your own content.
What changes is the shape of the failure. An ungrounded bot fails invisibly, in ways nobody can trace. A grounded one fails in a way you can see, name, and fix at the source — and, more often than not, declines to fail at all and asks for a human instead.
That is the trade we made deliberately. We would rather the bot say less than it could than say more than it knows.