More trust is not automatically better
Trust is useful only when it is calibrated to what the system can actually do.
A person can over-rely on a wrong recommendation because the AI sounds confident. They can also ignore a correct recommendation because prior failures made them too skeptical.
Both are failures of the human-AI system.
A bounded experiment shows the failure mode
In one preregistered 2026 experiment, 295 adults judged 80 faces while receiving guidance that was correct only 50% of the time. Participants who reported always using the guidance were 8 percentage points less accurate than those who did not use it and 5 points less accurate than those who used it selectively. Within the AI-guidance group, more positive attitudes toward AI also predicted poorer discrimination (b = -0.19, p = .008). Source: Scientific Reports
That is a narrow face-classification task, not proof that positive attitudes toward AI are generally harmful. But it directly supports the skeptical claim we are making: indiscriminate reliance can reduce decision quality when guidance is unreliable.
Explanations can help, and they can persuade
An explanation can help a person reason. It can also make a weak recommendation more persuasive.
Research on cognitive forcing found that interaction designs requiring users to engage their own reasoning can reduce overreliance compared with passive advice presentation.
Other work has found a gap between what a model knows and what people think it knows, and that uncertainty language can sometimes narrow that gap.
The design problem is not simply how much explanation to show. It is whether the interface helps the person make a better decision.
Confidence changes how people allocate attention
A CHI study surveyed 319 knowledge workers about 936 first-hand GenAI work examples. Higher confidence in GenAI was associated with less reported critical-thinking effort, while the work itself shifted toward verification, integration, and stewardship. The study was observational, so it does not show that AI causes cognitive decline—but it quantifies the scale of the evidence behind the concern. Source: Microsoft Research
That makes calibrated reliance a design problem, not just a user-training problem.
Design for judgment, not confidence
A better system should make it easier to answer a few practical questions:
- How strong is the evidence for this answer?
- What is the model uncertain about?
- Is this a domain where the system has demonstrated reliability?
- What would independently verify the recommendation?
- How costly is it if I accept the answer and it is wrong?
- Am I being asked to think, or merely being persuaded?
Put friction where the consequence justifies it
Constant warnings teach people to ignore warnings. Constant approval requests turn safety into bureaucracy.
Useful friction should appear where the consequence or uncertainty justifies it. A low-risk brainstorming task does not need the same interface as a consequential recommendation.
Low-risk work should stay low-friction. Higher-consequence decisions deserve a pause only when the pause improves judgment.
Do not ask whether you trust AI in general.
Do not ask whether you trust AI in general.
Ask whether this system, on this task, with this evidence, deserves your reliance.
The system should make the evidence, uncertainty, and consequence visible enough that the user can decide whether reliance is justified.
Sources
- To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI — Interaction designs and overreliance.
- Examining human reliance on artificial intelligence in decision making — Positive AI attitudes and poorer discrimination under unreliable guidance in a bounded experiment.
- What large language models know and what people think they know — Calibration gap between model knowledge and human belief.
- The Impact of Generative AI on Critical Thinking — Confidence and self-reported critical-thinking effort.