Do AI Support Bots Hallucinate? What Actually Goes Wrong

Everybody asks this before they install one, and they are right to. A support bot speaks to your members in your name, at three in the morning, with nobody reading over its shoulder. If it makes something up, you find out from the member.

The honest answer comes in two halves. The first is the one you are expecting. The second is the one that costs most servers more, and almost nobody tests for it.

Do AI support bots hallucinate?

Yes. A language model produces the most plausible continuation of a conversation, and plausible is not the same thing as true. Left ungrounded, one will quote a refund window, invent a price, or name a channel that never existed, in the same calm tone it uses for things it actually knows.

What makes this uncommon in a ticket bot is that a ticket bot is not left ungrounded. It answers out of a store of things your staff wrote down, and its instructions treat its own general knowledge as wrong for your server by default. What your server's AI actually knows covers where that store comes from.

So when an admin says the AI hallucinated, four different things are usually meant, and only one of them is this.

Three of the four are not the model inventing anything, and each has a different fix. That last row is where this article ends up.

What stops a support bot inventing an answer?

Four rules, all of them unglamorous, all of them carrying the weight. They sit in the instructions the AI runs under on every single reply, not in a marketing claim about accuracy.

The fourth is the smallest rule on that list and the easiest one to test, so it is worth a closer look.

The subtle part is that this holds even when the channel name came out of your own memory. Memory stores names as words, and a word is not a channel, so the name is checked against the live server before it is spoken and the clickable link is pasted instead of the bare text. If the check finds nothing, it says it cannot see that channel rather than naming it anyway.

It is a tiny behaviour, and it is the difference between a bot that is grounded and one that is merely plausible. You can test it in about a minute.

The failure nobody watches for

Every rule above has an overdone version. Push them hard enough and you get a bot that never says anything untrue and never helps anybody either, which is where most cautious support bots end up.

Both columns are failures. Only the left one gets screenshotted, because every individual reply on the right looks responsible. Nobody opens a ticket to complain that the bot was too careful. The member just stops replying, and the number that moved is one you were not watching.

So the second set of rules matters as much as the first, and it is the more interesting half:

• "That is outside what I cover" is never a reply to a question it has not looked up yet. Declining is reserved for questions about the assistant itself, its instructions and its setup. It is explicitly not a licence to duck a question about the thing your server exists to support.
• A clarifying question is allowed only when the request genuinely has two different answers and memory cannot say which one was meant. Never as a reflex, never to seem thorough, and never before searching.
• It cannot say it will check and then stop. The check happens in the same turn or it did not happen.
• If it tells a member that staff will review something, it escalates on that same turn. The promise and the notification are one action, so there is no human who was mentioned to a member but never actually pinged. How escalation is wired covers who gets it.

Read those four next to the four in the previous section and you have the real shape of the problem. Accuracy is not one dial you turn up.

When is a wrong answer most likely?

Not on day one, which surprises people. A brand new bot has nothing stored, so it mostly tells members it does not have that yet and offers to fetch somebody. Useless, but not dangerous, and teaching it is the fix for that.

The risk peaks in the middle. Your memory is half taught, most answers are right, and the confidence is identical across all of them. Somewhere in there is a fact that was true on the afternoon a staff member typed it and has quietly stopped being true since.

The stalest entry in a brain is always the one nobody has thought about in months, which is also the one nobody thinks to check. Two things keep that liveable. Contradictions resolve rather than accumulate, so correcting a policy is a one line job instead of a hunt for the wrong version. And a ticket where the AI got something wrong and a human put it right can become the correction itself, in one action, which is the loop most servers under-use. Learning from your real tickets is that second one in full.

What we cannot stop

The honest limits, in the order they are likely to reach you.

If your memory is wrong, the answer is wrong. Grounding makes the AI faithful to what you wrote down, and a mistake you teach it is a mistake it will deliver confidently, in seconds, to every member who asks. That is a far more efficient way of being wrong than one tired moderator at midnight.

Exact values survive, nuance can flatten. What lands in memory is a structured entry rather than your sentence word for word. Prices, links and commands come back verbatim. A carefully hedged exception written as prose can come back with the hedging sanded off. Where the precise wording matters, a saved quick response is the better tool: it is stored and replayed exactly as written.

Crossing languages is another place precision can go. Knowledge is stored in English and replies come back in the member's language. Technical values survive that on purpose. A policy nuance is prose, and prose translates.

In a public quick-question channel there is no safety net on that turn. That surface deliberately has no tools: it cannot escalate, read another channel, or learn anything, and anything account-specific gets pointed at a ticket instead. A weak answer there does not land in front of a human the way it would inside a ticket. What AI cannot do in support lists what is switched off where.

We do not publish an accuracy percentage. We could compute one and it would not mean anything, because it would be measured against tickets whose correct answer nobody ever wrote down. Treat any number of that shape, ours or anyone else's, as marketing unless it arrives with its method attached.

How can you tell if a bot will make things up?

Six checks, cheapest first. None are specific to us, all work on a free plan, and together they take about twenty minutes.

The last one is the real test and it is the one everybody skips. Across more than 180,000 tickets in this bot's lifetime, roughly half closed with no human replying at any point, which on a busy server is a great many conversations held in your name that nobody has read.

That figure is live at /api/stats/global rather than typed into this page, and every closed ticket keeps a transcript at its own link, so both the number and the conversations behind it are checkable rather than asserted. For where the line between the two sides sits, which tickets go to AI and which go to people draws it.