Why models make things up

Hallucination is the name for an AI answer that sounds convincing but has no basis: an article number that does not exist, a book title that was never published, a figure nobody ever measured. The word sounds like an occasional glitch. In fact, hallucination follows directly from how the model works.

Recall the lesson on the text predictor. When answering, the model builds text one token at a time (a token is a small piece of text, roughly part of a word). At every step there are many plausible continuations, and the model picks one of the possible continuations. There is no separate step inside it that asks "is this true?" or "do I actually know this?". When the patterns from model training are strong, say for a fact written thousands of times in many places, the chosen continuation is usually right. When the patterns are weak, the model writes anyway, because writing is the only thing it does.

There is one more push. After pre-training, the model is trained further to become a helpful assistant. A complete, confident answer tends to be rated more helpful than "I don't know". As a result, unless asked, a model rarely admits it does not know.

The example we use throughout this course: someone asks which article governs maternity leave and how long it is. The model answers with an article number from the 2003 Manpower Law, a leave of one and a half months before and one and a half months after birth, adds the condition "only for permanent employees", then calls it all "the rule in force today". Part of that answer really is in the old text. The permanent-employee condition is not in that text: it is invented. And since 2024 there is a newer law on the welfare of mothers and children that sets maternity leave for working mothers at no less than three months, and up to six months in specific conditions. The model did not lie, because lying takes intent. The old rule was written about and discussed for some twenty years, so its pattern is far stronger in the training data than the new rule, which may even have been issued after the model's knowledge cutoff. It wrote the shape of a legal answer it knows very well, and filled the parts it did not know with the closest-looking pattern.

That is why fluency is not a sign of truth. A made-up answer and a correct one are written with the same tone and the same polish. You cannot tell them apart by style, only by source.

Not every question is equally risky. Made-up answers turn up most often for six kinds of question.

First, very specific details: article numbers, case numbers, statistics, exact dates, prices. Their shape is easy to imitate and their content is hard to recall. Second, names of people, small organisations or rarely discussed products. The model has few examples about them, so it mixes in traits from similar names. Third, quotes and references: "who once said ...", "list five studies on ...". The model can produce titles, authors and years that sound very academic. Fourth, recent events. The model's knowledge stops at its knowledge cutoff, so regulations, prices or news after that are unknown to it. Fifth, internal matters: your office's policy, your client's name, last week's meeting decision. None of it was ever in its training data. Sixth, questions with a false premise, such as "why was that rule revoked last year?" when it never was. The model tends to follow the direction of the question and explain a revocation that never happened.

By contrast, the risk of made-up content is smallest when the model works on material in front of it: summarising text you paste, changing tone, reorganising, translating. Not zero, because it can still misread or add something, but you have material to match it against.

The example shows one check that is often skipped. The question is not only "does this article exist?" but also "is this still the latest rule on the matter?". A correct quote from an old law can still mislead if a newer rule now covers the same thing, and a model may well quote the old rule because its pattern is strongest in the training data.

Three simple steps:

1. Take the text from the official regulation database, not from the model's memory or a blog summary.
2. Look up the regulation's status there. Official databases usually record whether a regulation has been amended or revoked, and by which regulation.
3. Also search for newer regulations on the same topic. A new rule can govern the same matter without revoking the old one, so an "in force" status on the old rule does not settle everything. In the maternity-leave example, the 2024 law sets a different length of leave from the 2003 text.

How two rules are read together for one employee's case is a question for the HR team or legal counsel, not for the model and not for this course. Your job as an AI user is simpler: do not stop at the first answer, and do not pass a rule on to anyone before you know it is still the latest.