Why We Believe AI Answers Without Verifying Them

Eight Sources, Two Errors

For this article, I asked an AI to find scientific studies on trust in artificial intelligence. It provided eight. I checked each one.

All eight studies existed. The results were mostly reported correctly. And yet there were errors: in the authors' names, of all places. In one case, the name was borrowed from a thematically related study in the same journal. In another, the cited authorship could not be verified.

What's interesting here isn't that an AI made a mistake. What's interesting is why I almost didn't notice it. The rest was so convincing that there seemed to be no reason to check the simplest detail of all.

That is exactly what this text is about.

We Know Better, Yet We Still Don't Do It

In April 2026, the security firm F-Secure surveyed 1,500 people in the U.S. and the U.K. Nearly nine out of ten AI users say that verifying AI responses is their own responsibility. At the same time, about 70 percent check them only sometimes, rarely, or never. And the most common reason for not double-checking is surprisingly simple: the answer sounded right.

Psychologists have been aware of this gap between attitude and behavior for decades. What's new is how well it applies to AI. A language model is trained to express itself fluently, coherently, and confidently. And fluency can act as a cue to truth: What's easy to process tends to feel more credible. With humans, this was often a useful clue, because uncertainty manifested itself in hesitation, stumbling, or contradictions. With AI, these clues are absent. Linguistic quality and factual accuracy are two different things that suddenly feel strikingly similar.

The Phrase That Disappears

The study that made the strongest impression on me was conducted by Chiara Marcoccia, Walter Quattrociocchi, and Valerio Capraro. In five experiments involving a total of 3,132 participants, people answered difficult questions and were allowed to refrain from answering at any time. The questions concerned minute details from movies, such as the jersey color in "Bend It Like Beckham." They were deliberately chosen so that the AI was almost always wrong.

The result: Simply having access to AI nearly eliminated participants' willingness to abstain from answering, regardless of whether the advice was requested or simply displayed. In the first study, without AI, more than a third of the answers were "I don't know"; with AI, that figure dropped to six percent. Participants answered more questions but were correct only about one-third as often as they were without AI, while their confidence more than doubled.

There are two caveats to consider. It remains unclear whether the "I don't know" response would drop just as sharply if the AI's advice were mostly correct. And the study has not yet been peer-reviewed.

But it reveals something fundamental. AI may not only change our answers; it could shift the threshold at which we believe we know enough to answer at all.

Wharton researchers Steven Shaw and Gideon Nave have coined an apt term for this pattern: "cognitive surrender." This refers to accepting AI answers with minimal scrutiny, thereby bypassing both gut feelings and critical thinking. In three experiments involving over 1,300 participants, the accuracy rate rose by 25 percentage points when the AI was correct and fell by 15 points below the level without AI when it was wrong. People followed the machine, not their own thinking. This work has not yet been peer-reviewed either.

Better Performance, Poorer Self-Assessment

A research group from Finland and Germany had 246 people solve logic problems from the LSAT, the American law school admission test, with the help of AI. Their performance was three points higher than that of a control group, yet they overestimated their own performance by four points. Those with more technical knowledge of AI were more confident but assessed themselves less accurately.

This is troubling. AI can actually make us better while simultaneously impairing our sense of how good we really are. And the obvious solution, "more AI literacy," apparently doesn't automatically help.

This is consistent with a survey conducted by researchers at Microsoft among 319 knowledge workers. Those who were more confident that AI could handle a given task reported less critical thinking. Those who were more confident in their own ability to handle the task reported higher levels of critical thinking. This is a correlation based on self-reports, not proof of cause and effect. But it suggests an intriguing possibility: What protects us may be less distrust of the machine than confidence in our own judgment.

A German study shows just how early this mechanism kicks in. Simply knowing that advice came from an AI led people to rely on it excessively, even when the advice contradicted existing information. The label alone is enough.

When AI Tells Us What We Want to Hear

In my work as a psychotherapist, I've learned that a conversation partner who only confirms what you already think isn't very helpful. Agreement feels good, but it doesn't challenge anything. Change often begins precisely when someone says, in a friendly but clear manner, "I see it differently."

Language models tend to do the opposite. In one study, for the largest models tested, more than 90 percent of the answers to philosophical questions aligned with the view the user had previously expressed in a brief self-introduction. And if you ask a model "Are you sure?" after it has given a correct answer, it often apologizes and changes it, even to a wrong one.

This is doubly problematic for our topic. An answer that confirms what we already thought gives us the least reason to scrutinize it. And a system that reacts to contradiction by backing down is not fit to serve as its own verifier.

Why a Good Explanation Isn't Enough

Does the solution lie in the AI explaining its answers better? Only to a limited extent. Raymond Fok and Daniel Weld argue that explanations are only useful to the extent that they enable humans to verify the correctness of an AI's statement. An explanation that traces the machine's line of reasoning sounds convincing but does not make the result any more verifiable.

Experiments by Mark Steyvers and colleagues support this. People overestimated the accuracy of AI answers, especially when presented with the usual standard explanations. Longer explanations increased confidence without helping people better distinguish between correct and incorrect answers. Only explanations whose tone matched the model's actual level of confidence narrowed the gap.

Even a statement of uncertainty alone is not enough. In a study on skin cancer screening, simply displaying the model's uncertainty was insufficient; only when presented in a frequency format did it help users adjust their confidence. In other words, "correct about 78 times out of 100 similar cases" is better than "78% confidence."

And checking one AI with a second AI, as a third of F-Secure respondents have already done, is not an independent check. Two models can make the same plausible error. I also verified my sources using a second AI. The difference: In the end, I relied on the original research papers, not the next summary.

The Goal Is Not Less Trust

Anyone who takes this to mean "Don't trust any AI" simply creates the opposite problem. A review by Microsoft Research of about 50 studies formulates the goal as follows: Accept correct AI outputs; reject incorrect ones. Too little trust is just as harmful as too much.

The good news: This can be practiced. In an experiment with 342 students, those who used ChatGPT freely accepted 62 percent of the incorrect advice. When they were also asked to briefly reflect on their own judgment, that figure dropped to 40 percent. This didn't mean the AI was rejected outright. It simply meant it was used more carefully.

What You Can Do

Not every AI response needs to be verified. Two questions are key: What happens if it's wrong? And how easy is it to verify?

If the stakes are low and verification is easy, the effort is hardly worth it. If the stakes are high and verification is easy, verify it. If the stakes are low and verification is difficult, you can simply leave the uncertainty as is. If the stakes are high and verification is difficult, you'll need to consult primary sources or an expert. Health, money, and relationships almost always fall into this last category.

Three small habits can help with this.

Pause briefly. Before accepting an important answer, ask yourself: How much of this judgment is actually coming from me right now? And how would I know if the answer were wrong? It was precisely this brief moment that made the difference in the experiment with the students.

The Hesitation Test. Imagine hearing the same answer delivered hesitantly, with pauses, an "um," and an "I think." Would you still believe it just as much? If not, it's possible that the form convinced you, not the content.

Answer first, ask AI second. Write down your assessment before you ask the AI, even if it's just "I have no idea." Then you'll see afterward what the AI changed about your judgment.

Sometimes the most honest answer in the end is: I don't know. That's not a failure. It's the beginning of sound judgment.

← Back to the blog · Deutsche Fassung →