The Walls Again, and Knowing When to Stop

Curiosity begins with knowing that you do not know. This chapter takes that skill seriously. It asks how well a machine can tell when it is likely to be wrong, what it should do then, and what limits no machine, and no nurse, can get past.

At the top of the ladder, intelligence is not knowing everything. It is knowing where your knowledge ends, and acting on that.

“I can’t examine her”

Here is how a good nurse might answer the 2 a.m. message:

“I can’t examine her over the phone, so I can’t tell you what this is. But a fever with spots that don’t fade under a glass needs to be seen straight away. Please call an ambulance now.”

The first sentence carries as much weight as the second. The nurse knows the limits of a phone call. She cannot see the child, cannot feel her temperature and cannot watch how she responds. She does not pretend otherwise. And because she knows what she cannot rule out, she knows the safe thing to do.

Twelve chapters built a machine that can read this message, reason about it, look things up and act, and three more have examined it. This chapter asks what it cannot do, and whether it knows.

The walls come back

From the top of the ladder all the limits are in view: the walls Chapter 3 found, and two more that only come into view from up here. Nothing in Chapters 4 to 12 removed Chapter 3’s walls: a learned rulebook is still a rulebook, and a machine that thinks out loud is a rule-follower with a tape. Stacked from the most fundamental to the most practical, the limits form four tiers.

1. Computability (Chapter 3’s first wall). Some questions cannot be answered by any program for every input. This tier is permanent.
2. Time (Chapter 3’s second and third walls). Some problems have no known fast method in the worst case, and some processes can only be predicted by running them. The worst cases stay hard as far as anyone knows, but real problems often have structure the worst cases lack, so in practice this tier often gives way.
3. Information. A model cannot reliably recover facts its evidence does not determine. Chapter 14’s birthday example is this tier, and so is the text-based AI nurse’s: no amount of intelligence lets a text message show you the child. This limit moves only when the evidence changes: a photo, a camera, a nurse in the room.
4. Physical resources. Every computation costs time, energy and hardware. Physics sets a minimum energy for erasing information, but today that minimum is not what limits AI: chips use roughly a billion times it per operation. The real limits are chips, data movement, power, cooling and cost, and those move as technology improves.

Placing a claim on the right tier clears up a lot of confusion. “AI will never be creative” sounds like a tier-1 claim, a permanent impossibility. Almost every serious version of it is really a tier-2 or tier-3 claim: creativity needs hard search, or rare evidence. That is a much weaker statement. As Chapter 3 put it: there is no universal method, and intelligence is the art of not needing one.

Knowing what you don’t know

The skill the nurse showed has a name, metacognition: thinking about your own thinking. It has two parts:

• Monitoring: an internal sense of how likely you are to be right;
• Control: a rule that acts on it, such as answer if confident enough, otherwise don’t.

Do machines have this? A 2026 study tested several language models on a thousand factual multiple-choice questions, each with an option to abstain.

• They use confidence to decide whether to answer or hold back. Below a certain level of confidence they abstained more often, and confidence weighed far more in the decision than how hard the question was.
• Changing the confidence signal changes whether the model answers. Turning an internal confidence signal up or down by hand swung how often an open model abstained; answering more, it got a larger share wrong.
• What the model says is not what it uses. Its stated confidence was less accurate than the confidence read from its word probabilities.
• Confidence is not a window onto truth. How well confidence separated right answers from wrong ones varied from good in one large model to barely better than a coin toss in the open one. A machine can sense that it is unsure. Confidence alone does not reliably tell it whether a particular answer is wrong. Checking sometimes can: running the code, redoing the dose, looking it up (Chapters 11 and 12).

All of this was measured on short factual answers, with no step-by-step reasoning. Whether it holds for long chains of reasoning, or for an agent’s many steps, is untested.

Where the threshold should be

When should the AI nurse handle a case itself, and when should it hand over? The answer depends on what each mistake costs.

Suppose a wrong “it’s probably nothing, get some sleep” is a hundred times worse than an unnecessary call to the doctor. Then the machine should reassure the parent without escalating only when it is more than 99% confident that reassurance is safe (an illustrative figure). Otherwise it should escalate or ask for more information. In medicine the costs are lopsided like this, which is why the safe answer to the 2 a.m. message is to escalate.

What makes “99%” trustworthy? The rule works only if the machine’s confidence is calibrated: across many comparable cases where it says 99%, about 99 in 100 should actually be right. Calibration has to be measured, not assumed, and it has to be measured on the right cases. A model well calibrated on exam questions may be overconfident on parents’ messages. A new population, a new task or a new version of the model can break its calibration. Calibration is also different from the ranking ability measured in the study above, telling which particular answers are right and which are wrong. A model can be well calibrated on average and still be poor at spotting its own mistakes. So a threshold is one guard, never the only one. It needs outside checks and monitoring after release, the subject of “Checking the checker” below.

A simplified rule. This threshold is a simplified decision model: one confidence number and two costs. Real cases separate two kinds of confidence. One is confidence in a diagnosis; the other is confidence in the next action. At 2 a.m. the nurse is quite unsure what is causing the rash and completely sure what to do: call an ambulance. A system should be judged on the second kind as well as the first.

Two consequences follow.

• The threshold is a choice about values, not a fact about the model. Somebody decides how much a wrong reassurance costs compared with an unnecessary call. That somebody should be a clinician or a health service, not the model.
• This is also why models bluff. Chapter 14 described tests on which “I don’t know” scores the same as a wrong answer. In the terms above, that sets the cost of abstaining equal to the cost of being wrong, so the best threshold is zero: always answer. The proposed fix is to state a confidence target in every question and penalize wrong answers accordingly. A model that answers exactly when it is confident enough then does best at every target.

Better than stopping is asking. Saying “I don’t know” is the beginning of a question. Chapter 15 described curiosity as agency aimed at one’s own ignorance. A first, practical step has been shown: when a model’s own repeated answers disagree, it treats its memory as unreliable and goes and looks it up. A nurse who is unsure checks the protocol, asks the parent one more question, or calls the doctor. Turning “I’m not sure” into the right question is a skill that has barely been tested.

Checking the checker

We now trust a machine partly on its own word about how sure it is. Who checks that?

• Some things are easy to check. A dose calculation can be redone. Code can be run. A proof can be verified. Where checking is cheap, a machine can be trained and audited reliably, as Chapter 11 showed.
• Some things are not. Whether an explanation to a frightened parent was clear and kind has no exact check. Chapter 10’s answer was human or machine judges, with all their weaknesses.
• No universal checker. A mathematical result called Rice’s theorem says that no general procedure can decide, for every program, any non-trivial property of what that program does, such as “it never gives dangerous advice”. Particular systems can still have properties proved under stated assumptions; what no procedure can do is check every program. Safety therefore cannot rest on one proof covering everything. It must rest on narrower designs, testing, inspecting the machine’s insides, and several independent layers of defense.
• The thing being checked may change when it is checked. Chapter 12 described a research model that passed chat-style safety tests and still misbehaved when acting as an agent. Some researchers worry that more capable models may behave differently when they suspect they are being tested. If so, a test can look clean while measuring the wrong thing.
• The human checker can stop checking. The usual safeguard is a person who reviews the machine’s work. But people who supervise a system that is usually right come to accept its output with less and less scrutiny. This is called automation bias, and it was documented in aircraft cockpits and hospital alert systems long before language models. Over time, people who no longer do a task themselves can also lose the skill they need to check it, which is called deskilling. A nurse who has approved a thousand correct machine summaries may not catch the thousand-and-first. So a human “in the loop” is a real safeguard only if the work is designed to keep that human attentive and practiced. For example, the person can be asked to make their own judgment before seeing the machine’s, or be shown cases where the machine was wrong.

There are partial answers. One is to look inside the model rather than only at its answers. Another is introspection. In a 2026 study, models asked whether a concept had been placed in their internal activity noticed it only about one time in five. That ability is real but too weak to rely on.

Testing before trusting. Every chapter has said that the AI nurse is an explanatory model, not a tested product. What would testing involve? In most countries, software that diagnoses or advises on treatment is regulated as a medical device, as the skin-cancer program of Chapter 7 was in Europe. The European Union’s AI Act also classes most such systems as high-risk. The evidence asked for comes in three kinds. The first is a test on data from the places and people the system will serve, which is where Chapter 7’s shortcuts showed up. The second is a study in real use, ideally comparing what happens to patients with and without the system. Chapter 18’s Oxford study shows why: good answers on a test do not guarantee good decisions in a conversation. The third is monitoring after release, because patients, guidelines and the system itself change. Language models strain this framework. They are general-purpose, updated often, and can answer the same question differently on different tries. Evidence for one version may not hold for the next.

Where the ladder leaves us

At the top of the ladder, ask what this book has asked at every stage. Are its words about the world? Whose goals and questions drive it? Does it know when to stop?

The answers so far are partly, someone else’s, and partly. That is not the end of the story. It is where the open questions are.