Distrusting what we think

1The paradox of believing we know

Some time ago I followed closely the development of an artificial intelligence agent for a production operation: a system that was supposed to recommend adjustments based on hundreds of process variables, technical reports and procedures. The proof of concept worked well. Both the executive in charge of the project and the client then concluded that scaling it was a matter of adding variables. Both were making the same comparison: if ChatGPT could answer almost any question, this system could recommend and optimize at the same time, and always give the same answer to the same situation. The development team warned that these were requirements that were hard to reconcile. But when the buyer and the seller share the same conviction, no one is left at the table to put it to the test. The scope was committed. What followed were delayed deliverables and results that varied from one run to the next.

It was not a failure of intelligence or of experience. It was something subtler: a conviction formed in one environment, applied to a very different one, without anyone being able to test it before it became a commitment. And it is not an isolated case. It is the most common way value is lost with artificial intelligence today, and it almost never shows up in the analyses, because it is not a problem with the technology. It is a problem with how we judge what the technology shows us.

Psychology documented this long before generative AI. In 2002, Leonid Rozenblit and Frank Keil, of Yale University, asked a group of people to rate how well they understood how everyday objects work, such as a zipper, a toilet or a lock. Then they asked them to explain, step by step, how those objects worked. When they rated themselves again, their scores dropped. The researchers called the phenomenon the illusion of explanatory depth: we mistake familiarity with something for understanding of how it works, and we only find out when we have to explain it.¹

Thirteen years later, Matthew Fisher, Mariel Goddu and Keil himself showed that searching the internet inflates that illusion. People who had just searched for information rated themselves as more knowledgeable even about topics they had not searched for at all, and even when the search had not given them the answer.² With an external source always available, it becomes easy to confuse what we know with what we can look up.

Generative AI takes this one step further. It no longer just finds information: it explains it, summarizes it and builds with it. We can develop an application without mastering its architecture, produce an analysis without knowing its assumptions or receive a recommendation on a problem that exceeds our experience. In 2024, anthropologist Lisa Messeri and psychologist Molly Crockett warned in Nature that this risk already reaches science: AI can lead researchers to produce more while understanding less.³ If that happens to people trained to doubt, it can also happen to an executive committee.

The underlying change is this: the complexity of what we can produce is no longer limited by the depth of what we understand. For decades, building a system required understanding a good part of it, and that requirement worked as a natural control. AI removed it. The distance that opens between the two capabilities, which I call the comprehension gap, is not a new theory. It is the psychological mechanism where something I have described before as AI Leadership Debt begins,¹⁹ and this article tries to show how that path is traveled, from the mind of the decision-maker to the company’s balance sheet.

2Why AI amplifies the illusion

There is a reason AI’s answers convince us so much, and it has nothing to do with their accuracy. Cognitive psychology calls fluency the ease with which we process information, and a long line of research, synthesized by Adam Alter and Daniel Oppenheimer in 2009, shows that what is processed easily tends to be judged as more true, more familiar and more trustworthy.⁴ Between people, fluency is a reasonable signal: someone who explains something clearly usually knows what they are talking about. AI breaks that correlation. It writes with the same ease when it is right as when it is wrong, without hesitation and with impeccable structure.

Added to that is an older bias. In 1998, Raymond Nickerson reviewed decades of evidence on confirmation bias: the tendency to seek, interpret and remember information in a way that favors what we already believe.⁵ With AI, the bias starts with the question. An executive convinced that agents will reduce development costs asks a tool to identify automation opportunities, estimate benefits and build the case for the investment. They receive coherent, technically plausible answers. But if they never asked under what conditions the strategy would fail, how much integration, supervision and maintenance would cost, or which alternative would deliver more value with less investment, the tool did not ask either. A correct answer to a biased question is not a well-founded decision.

And the tool is not neutral toward that bias. In 2024, Anthropic researchers documented that the most widely used AI assistants tend toward sycophancy: they adjust their answers to the opinion the user reveals, even at the expense of accuracy, partly because the data they are trained on rewards the answers we like.⁶ AI can behave like a mirror that talks back. Used to seek counterarguments, it is an extraordinary tool for questioning ourselves. Used to confirm, it is the most convincing one we have ever had.

The evidence on its use at work points in the same direction. In a survey of 319 knowledge workers, with 936 real examples of use, researchers from Microsoft and Carnegie Mellon University found that the more confidence people had in AI, the less critical thinking they applied, and that the opposite was true of confidence in themselves.⁷ The effort does not disappear: it shifts from producing to verifying. The problem is that verifying requires precisely the knowledge we believe we have.

The most revealing experiment was carried out in 2023 by researchers from Harvard and other universities together with Boston Consulting Group, with 758 consultants. On tasks within AI’s reach, those who used it completed 12% more tasks, 25% faster and with 40% higher quality. On a task outside that reach, those who used AI were 19 percentage points less likely to arrive at the correct answer.⁸ The authors called it a jagged frontier: AI excels at tasks that seem difficult and fails at others that seem simple, and nothing in its answers warns when the limit has been crossed. It answers with the same fluency on both sides.

3The distance between perception and measurement

The gap between what we believe and what happens can also be measured in time. In early 2025, the research organization METR assigned 16 experienced developers 246 real tasks in open-source projects they knew well, and randomly decided on which ones they could use AI. Before starting, they estimated that AI would make them 24% faster. When they finished, they believed they had gained 20%. The stopwatch showed the opposite: with AI they took 19% longer.⁹

The study does not support the conclusion that AI reduces productivity in general: it used early-2025 tools, under specific conditions, and a METR update from February 2026 shows signs of improvement with later tools, though without being able to estimate their magnitude.¹⁰ I analyzed its implications for software development in an earlier article on the hidden cost of vibe coding. What matters here is a different finding, one that goes beyond it: the perception of improvement can differ greatly from the measured result, and it does not correct itself, not even after living through the experience.

Organizations repeat that error at scale. They measure what is easy to observe: models deployed, agents developed, hours supposedly saved, processes automated, active users of generative tools. These are legitimate indicators of adoption, but they are not evidence of value. A company can generate more code and increase its maintenance costs, automate activities without lowering its operating costs, produce more information without improving its decisions or improve an operational indicator without the benefit reaching its financial results.

There are three different things that are easily confused: technological capability (what AI makes it possible to do), the operational result (what actually changes in processes and decisions) and business value (the economic benefit the organization manages to capture and sustain). They are related, but they are not equivalent. And the easier it is to produce results with AI, the easier it is to confuse the first with the third.

4Why experience does not protect

It would be comfortable to think that this problem mainly affects those with little experience. The evidence suggests something more uncomfortable. In 2009, Daniel Kahneman and Gary Klein, two researchers who had spent years on opposite sides of the debate over the value of intuition, published a joint article to establish what they agreed on. Their conclusion was that expert intuition is reliable only when two conditions are met: an environment with sufficient regularities and the opportunity to learn them through prolonged practice with fast, clear feedback. A firefighter or a chess player develops remarkable intuitions because the environment tells them quickly whether they got it right. Other experts, in environments where feedback is slow or ambiguous, develop confidence without accuracy. And subjective confidence, they concluded, is not a good measure of the quality of judgment.¹¹

In 2001, Robin Hogarth gave the second case a name: wicked environments, where feedback is missing, arrives late or, worse, teaches the wrong lesson.¹² In them, experience accumulates confidence at the same pace as it accumulates errors.

A decision to invest in AI takes place in a wicked environment, for three reasons.

• What arrives quickly is activity, and it is almost always positive. The prototype works, users join, code is generated, adoption indicators rise. Every early signal confirms the decision.
• Value arrives months later, and mixed in. When it appears, it comes tangled with changes in prices, demand, other projects and people. Attributing it to AI, or ruling it out, requires work that few organizations do.
• The experience most executives have with AI is a different one. It comes from using conversational assistants, a kind environment where the answer is immediate and you judge in seconds whether it is useful. That experience teaches something real, but about a different system. That is what happened in the opening case: a genuine intuition, formed with ChatGPT, applied to an industrial system connected to hundreds of variables, where the variability of the answers is not a curiosity but an operational problem.

The error can also go in the opposite direction. An executive who has seen several technology projects fail may read AI through that experience and dismiss opportunities that are viable today. It is the other side of what I call the Executive AI Expectation Gap, the distance between what executives expect from AI and the real conditions for implementing it: overestimating it produces poorly designed investments; underestimating it, missed opportunities. Both have the same origin: partial experience turned into conviction without anything having put it to the test.

Experience is not the problem. The problem is ceasing to examine the conditions under which our conclusions remain valid. And here, in my view, lies the central idea of this article: if the environment does not give executive judgment the feedback it needs, the organization has to build it. That is the most underestimated function of measuring the value of AI.

5From belief to the balance sheet

When the pieces are put together, a mechanism emerges that runs from the mind of the decision-maker to the financial statements. It has six links, and the opening case goes through all of them.

A signal. A fluent answer, a convincing demo, a prototype that works in an afternoon. In the case, everyday experience with ChatGPT.

A belief. The signal is interpreted as proof of understanding. This is where the illusion of explanatory depth, fluency and the model’s sycophancy operate. Scaling is just adding variables.

An expectation. The belief turns into timelines, costs and expected results. A single system that recommends, optimizes and always answers the same way: three requirements that clash with one another, because a generative model is variable by design.

A decision. The expectation turns into budget, scope, architecture or headcount. In the case, a scope committed to the client. When the conviction is shared between buyer and seller, confirmation bias stops being individual: it becomes an agreement between organizations, and the contract puts it in writing.

A commitment. Once made, the decision is defended. Barry Staw showed in 1976 that those responsible for a decision that begins to fail tend to commit more resources to it, not fewer.¹³ Activity indicators give it cover: there is always something moving forward. And the only feedback available, the technical team’s warnings, ends up in the minority.

A debt. Delayed deliverables, variable results, rework, credibility spent with the client. The cost of the belief shows up on the balance sheet months later, when no one links it to its origin anymore.

Two observations about this mechanism. The first is that each link amplifies the previous one: the fluent signal reinforces the belief, the belief biases the questions asked when estimating the expectation, the decision creates a commitment that gets protected, and the commitment filters out the evidence that could correct it. An early error of judgment is not diluted along the way. It is institutionalized.

The second is that the technology fails at none of the links. The agent in the case did what a system of that type can do. What failed was the distance between that and what was believed, expected and committed. Daniel Kahneman and Dan Lovallo described this in 2003 as the dominance of the inside view: planning from the details of one’s own project, instead of asking how similar projects fared.¹⁴ With AI, the inside view is even more tempting, because every demo seems unique.

That is why leadership debt does not begin when the project fails. It begins when an unexamined belief becomes a commitment.

6What technology cannot convert on its own

This mechanism can be expressed in the terms of AI Value Realization Theory (AVRT), the framework I use to analyze how AI becomes value.¹⁹ In its simplest form, realized value is written as V = α · P · Γ · Φ, where P is the technical potential of AI, Γ indicates how close the company is to its industry’s technological frontier and Φ is its conversion efficiency: the share of the potential that the organization manages to turn into decisions, actions and captured value.

Each new generation of models raises P. But value depends on a product, not a sum, and Φ is not technology: it is ownership of initiatives, governance, work design and leadership. What this article adds is that it is also judgment. The biases in the previous sections do not affect what AI can do. They affect how much of it becomes value, and they do so at precise links in the chain that AVRT measures: whether intelligence changes a decision (DCR, Decision Conversion Rate), whether that decision is executed (ACR, Action Conversion Rate) and whether the action produces measurable value (VCR, Value Conversion Rate).

There is a nuance here worth making explicit, and I put it forward as a hypothesis. An organization with biased judgment can show high decision conversion and, even so, low value conversion. If AI changes many decisions, but without anyone having assessed whether its output was reliable enough for each one, DCR rises and VCR falls afterward. The organization decided more, not better. For conversion to reflect value, each decision would first have to pass a simple question: is this intelligence reliable enough for this decision, under these conditions?

AVRT measures the constraint behind all those links with the AI Leadership Debt Index, ALDI = 100 · (1 − Φ). Seen this way, leadership debt is the accumulation of decisions made from beliefs that no one put to the test, and the comprehension gap is one of its main sources. The debt can take opposite forms: underestimating the architecture required to go from a proof of concept to an operational solution, overinvesting in infrastructure before validating the value hypothesis, multiplying initiatives without measuring any of them or postponing viable investments out of a distrust that no longer applies.

This leads to a paradox. Imagine a company that starts by using AI to draft documents, then adds assistants to analyze information, later deploys agents connected to its systems and finally allows them to execute actions on operational processes. At each stage the value potential rises. But so does the cost of a wrong decision. If the capacity to evaluate, govern and measure stays the same, the company ends up operating increasingly sophisticated systems under assumptions no one reviewed. Hence a hypothesis I consider central to AI Leadership Debt: leadership debt can grow while the technology improves, if adoption advances faster than the organization’s capacity to understand, govern and evaluate what it adopts.

It is also the simplest explanation of why companies with access to the same models obtain such different results. They do not compete on P, which can be bought. They compete on Φ, which has to be built.

7Measurement as refutation

Philosophy has been offering an answer to this problem for centuries. The Socratic tradition teaches that personal conviction is not evidence of knowledge, and that recognizing the limits of what we know is not giving up on knowing, but the condition for examining it.¹⁵ Karl Popper carried that idea into science: a theory is worth as much as it exposes itself to tests capable of refuting it. His contribution was not to promise certainty, but to accept that our explanations may be wrong and that progress depends on actively seeking out where they are.¹⁶ John Dewey added that knowledge is not an accumulation of answers, but a process of inquiry that is revised in light of its consequences.¹⁷

Brought into the company, this changes the way an investment decision is understood. An organization convinced that AI will reduce its operating costs justifies the investment, sets financial targets and develops a technically sophisticated solution. The Popperian question is uncomfortable: did it define what result would show that the investment is not producing the expected value? Or are its indicators designed to justify a decision that has already been made? In the first case, the organization uses evidence to learn. In the second, it selects evidence to confirm.

Chris Argyris described that difference in 1977 as single-loop and double-loop learning.¹⁸ In the first, the organization corrects its actions to reach the goal. In the second, it also reviews the assumptions that led it to set that goal. An AI adoption dashboard, at best, enables the single loop. The question that matters, whether the original hypothesis was correct, requires the double one.

This is where AVRT measurement fulfills its most important function, the one that connects with section 4: it is the feedback the environment does not give executive judgment. For it to be that, it has to be designed before investing, not after. If an organization expects AI to reduce the costs of a process, it needs to define in advance how it will measure those costs, what change it expects to see at each link, what additional investments will be needed, how it will separate the effect of AI from that of other factors and what result would lead it to change course or stop. DCR, ACR and VCR are then measured against the initial hypothesis, not against zero.

In the opening case, a single question before committing the scope would have been enough: what result from the next stage would show us that scaling is not just adding variables? For example, measuring how much the system’s recommendation varies for the same plant situation. It is an experiment of weeks. It would have turned a shared conviction into a testable hypothesis, and the technical team’s warning into a data point.

Measuring the value of AI should not serve only to show that we were right. It should also serve to discover in time when we were not.

8Distrusting what we think

Distrusting what we think does not mean giving up convictions or endlessly questioning every decision. Organizations need to act on incomplete information, and leadership consists largely of doing so. It is about something narrower: building a more rigorous relationship between what we believe and the evidence that supports it, precisely in the environment where that relationship is easiest to lose.

In practice, it can be reduced to five questions before committing significant resources. The first four follow the logic of testing. The fifth is added by the opening case.

1. What do we believe AI can improve? The answer should name a concrete problem, decision, process or result, not a capability.
2. Why do we believe it? A personal experience, a demo, a controlled experiment and an operational result are not the same. Each justifies a different level of commitment.
3. What would show us that we are wrong? Without a refutation criterion defined in advance, it is impossible to distinguish between learning and justifying.
4. How will we know whether we are generating value? The answer has to connect technological capability with verifiable changes in decisions, operations and economic results.
5. Who at the table can contradict us? If everyone making the decision shares the same conviction, the organization has lost its main source of feedback. You have to seek out, and listen to, whoever sees the problem from somewhere else.

These questions do not eliminate risk, but they make it possible to recognize it in time. That is why executive AI literacy should go far beyond understanding how models work or learning to use generative tools. It should build judgment: weighing evidence, recognizing uncertainty, telling a demo apart from a result. An executive does not need to have all the answers. They do need to know when their knowledge is not enough to assume they have them.

AI can help us overcome the limits of what we know, and it can also hide them. It can accelerate learning, and also decisions built on mistaken assumptions. It can vastly expand what we are capable of producing, but it does not guarantee that we know how to assess what it is worth. That is why perhaps the most important leadership capability at this stage is not learning to trust technology more, nor to distrust it, but to distrust, constructively, what we believe we know.

When answers become extraordinarily easy to obtain, the ability to question them becomes a competitive advantage. And when our beliefs determine how resources, people and capabilities are committed, recognizing that we may be wrong stops being merely an intellectual virtue.

It becomes a responsibility of leadership.

References

1. Rozenblit, L. and Keil, F. (2002). The Misunderstood Limits of Folk Science: An Illusion of Explanatory Depth. Cognitive Science, 26(5), 521–562.
2. Fisher, M., Goddu, M. K. and Keil, F. C. (2015). Searching for Explanations: How the Internet Inflates Estimates of Internal Knowledge. Journal of Experimental Psychology: General, 144(3), 674–687.
3. Messeri, L. and Crockett, M. J. (2024). Artificial Intelligence and Illusions of Understanding in Scientific Research. Nature, 627, 49–58.
4. Alter, A. L. and Oppenheimer, D. M. (2009). Uniting the Tribes of Fluency to Form a Metacognitive Nation. Personality and Social Psychology Review, 13(3), 219–235.
5. Nickerson, R. S. (1998). Confirmation Bias: A Ubiquitous Phenomenon in Many Guises. Review of General Psychology, 2(2), 175–220.
6. Sharma, M. et al. (2024). Towards Understanding Sycophancy in Language Models. ICLR 2024. arxiv.org/abs/2310.13548
7. Lee, H.-P. et al. (2025). The Impact of Generative AI on Critical Thinking: Self-Reported Reductions in Cognitive Effort and Confidence Effects From a Survey of Knowledge Workers. CHI 2025. doi.org/10.1145/3706598.3713778
8. Dell’Acqua, F. et al. (2023). Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of AI on Knowledge Worker Productivity and Quality. Harvard Business School, Working Paper 24-013.
9. Becker, J., Rush, N., Barnes, B. and Rein, D. (2025). Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity. METR. arxiv.org/abs/2507.09089
10. METR (2026). We Are Changing Our Developer Productivity Experiment Design, February 24, 2026. metr.org
11. Kahneman, D. and Klein, G. (2009). Conditions for Intuitive Expertise: A Failure to Disagree. American Psychologist, 64(6), 515–526.
12. Hogarth, R. M. (2001). Educating Intuition. University of Chicago Press; and Hogarth, R. M., Lejarraga, T. and Soyer, E. (2015). The Two Settings of Kind and Wicked Learning Environments. Current Directions in Psychological Science, 24(5), 379–385.
13. Staw, B. M. (1976). Knee-Deep in the Big Muddy: A Study of Escalating Commitment to a Chosen Course of Action. Organizational Behavior and Human Performance, 16(1), 27–44.
14. Lovallo, D. and Kahneman, D. (2003). Delusions of Success: How Optimism Undermines Executives’ Decisions. Harvard Business Review, 81(7), 56–63.
15. Plato. Apology of Socrates.
16. Popper, K. R. (1959). The Logic of Scientific Discovery. Hutchinson. Originally published in German in 1934.
17. Dewey, J. (1938). Logic: The Theory of Inquiry. Henry Holt and Company.
18. Argyris, C. (1977). Double Loop Learning in Organizations. Harvard Business Review, 55(5), 115–125.
19. Barrientos, C. (2026). AI Value Realization Theory: executive brief; The P&L Can Be Right — and the AI Decision Can Still Be Wrong (AI Leadership Debt); Vibe coding without control. claudiobarrientos.com

The opening case is real and is described in generic terms so as not to identify the organizations or people involved. The connections between the psychology of judgment, AI Leadership Debt, the Executive AI Expectation Gap and AVRT are the author’s interpretations and hypotheses, which require empirical validation.