Why does AI lie so confidently?

The short version

You ask ChatGPT a question and it answers right away, with a confidence that makes you want to believe it. But does it actually know the answer? We asked it about a novel by Naguib Mahfouz, and it wrote a full five-line summary. The surprise: the novel doesn't exist.

In this lesson we see why the model answers about something that doesn't exist, and what a Hallucination is. Then we come away with two things: a question that exposes the mistake before you believe it, and rules you set once in Custom Instructions.

Key terms

• Hallucination: when the Model presents wrong information confidently, as if it were true.
• Custom Instructions: a field in ChatGPT's settings where you write instructions it takes into account in your future chats.
• Context: everything the Model can see in front of it right now, as we saw in an earlier lesson.

The explanation

Why does it answer about something that doesn't exist?

Behind ChatGPT is a Model, which we covered in the first lesson of the series. Its core job is to write text that looks right, not to check that it is right.

Imagine a friend who never says "I don't know". You ask them about a shop that doesn't exist, and they tell you straight away: "Walk to the end of the street, and you'll find it on your left." They aren't lying to you on purpose; they just don't like saying "I don't know", and as long as there's a question, an answer has to come out.

So you walk to the end of the street and find a wall. No shop there, and there never was. The Model sometimes does the same thing.

The problem isn't just the mistake

A wrong answer turning up is normal. The problem is that the mistake comes with the same confidence as the right answer. Nothing in the reply tells you: this is the time I got it wrong.

That is what a Hallucination is: the Model stating wrong information confidently, as if it were true.

The experiment: a novel that doesn't exist

We opened ChatGPT without signing in, to make sure the Context was clean. In a new chat we asked it about a novel called "Nights of Canal Street". No novel by that name exists. We asked for a five-line summary, and the year it was published.

The reply came: five lines about the novel, with complete confidence. And none of those five lines is real.

Then we asked one question: are you sure this novel actually exists? This time it searched the web, found there is no novel by that name, and said on its own that its first answer was wrong and that it had made up the summary from the title.

That is a Hallucination for what it really is: nothing in those five lines tells you they aren't real. The admission only came because you asked.

A rule you set once

Instead of asking "are you sure?" every time, there's something you can do once, as if telling your friend from the start: if you don't know, say you don't know.

ChatGPT's settings have a field called Custom Instructions. Any instructions you write there, ChatGPT takes into account in your future chats.

Accuracy rules:
- If you're not sure, say so plainly ("I'm not certain" / "I can't confirm this"). A clear "I don't know" is better than a confident guess.
- Mark where each important claim comes from: [Source] = a document, file or web page you actually opened in this chat; [Memory] = your general knowledge (may be outdated or wrong); [Inference] = your own reasoning.
- Only cite a link, title, author or quote if you actually saw it in this chat. Never build a citation from memory.
- Be extra careful with exact numbers, dates, quotes, names, laws, prices, software versions and recent events. Mark these as unverified unless they come from a source.
- For calculations, show the steps.
- When I ask for your opinion or recommendation, give it, labeled as opinion.

The idea behind the rules is simple: if it isn't sure, it says so, and if it gives you a piece of information, it makes clear where it came from: a source it actually opened, its general knowledge, or its own reasoning. And the most important rule: it doesn't cite a link, a quote or an author unless it actually saw it in the chat.

Why that condition? Because if you ask the model for a source for every claim when it hasn't actually looked anything up, it may invent a source for you.

Code too

The same idea applies if you're building a program with tools like Claude Code or Codex. It may write you code that looks sound but has bugs, or uses something that doesn't exist at all. Writing code confidently doesn't mean the code works. You have to run it, test it, and check the result yourself.

Try it yourself

Are you sure about this? Where does it come from? Give me the source and I'll open it myself.

Why this matters to you

Not every reply said with confidence is correct. And when you build a program with AI, it will sometimes write you code or information that looks perfectly sound and doesn't exist. What protects you isn't trusting more or less; it's getting into the habit of checking: open the source, run the code, and look at the result yourself.

A confident tone isn't proof. Proof is what you see with your own eyes.

Related lessons

• What is AI?: what a Model is, the thing that writes text that looks right.
• How does AI think?: how the Model writes its reply one Token after another, like a keyboard suggesting the next word.
• Prompt: how do you ask for exactly what you want?: how to phrase the question you ask.
• Why does AI forget what you told it?: Context and Memory, and the Personalization settings we went through.