Agency and Curiosity: Whose Goals, Whose Questions?

The machine’s meaning is borrowed. Chapter 12 showed that its goals are not its own either: they are delegated, written by its builders and users. That raises two connected questions. What would it take for a machine to have goals of its own? And can a machine want to know: ask the question that changes everything, instead of only answering the ones it is given? Those are the questions of agency and curiosity.

Answering is not the same as asking. The most useful question is the one whose answer changes what happens next, and choosing it takes a goal.

The question that changes what happens next

A parent writes at 2 a.m.: “My son has a temperature and some spots on his tummy. Should I be worried?”

This message is less clear than the one we have followed through the book. Many feverish children with spots have a mild viral illness. A few have something dangerous. Reading the message more carefully will not tell them apart. A good nurse does not guess. She asks one question:

“Press the side of a clear glass firmly against the spots. Do they fade?”

The question is short. Its value is that the possible answers lead to different next steps. “They don’t fade” settles it: call an ambulance now. “They fade” does not mean the child is safe: not everyone with meningitis gets a rash. So the nurse goes on to the other warning signs, such as a child who is hard to wake, floppy or not drinking. Either way, the question has changed what happens next.

Nobody scripted this particular conversation. The glass test may well be taught. The nurse’s judgment lies in choosing it now. She knew what she did not know, and she knew which missing piece would change what she did. That is curiosity in its most practical form. This chapter asks what it is, and whether a machine can have it.

A loop with a goal

Chapter 12 described an agent as a language model inside a loop. Here is the idea in its general form. Two ingredients make an agent:

1. A loop. The system acts on the world, then sees what changed.
2. A goal. Something inside it says how the world should be. The gap between should be and is drives the next action.

Neither alone suffices. Weather is a loop without a goal: it changes the world and is changed by it, but not for anything. A wish is a goal with no loop. A plain chat model is closer to the second. It answers once, and the person reading the answer is the one who compares it with what they wanted and asks again. In a chat, the human closes the loop.

Grades of agency

Agency is not yes-or-no. Four grades are useful.

The difference between grades 1 and 2 has a neat test. Secretly rewire a thermostat so that its heater cools the room, and it keeps “heating” while the room gets colder; it cannot notice. A dog whose usual path is blocked takes another route. Chapter 12 used “agent” broadly, for any goal-driven loop. Here I draw the line more strictly, between simple regulators and adaptive agents that can change their means when circumstances change. Researchers at DeepMind made this the definition: an agent is a system that would change how it acts if its actions affected the world differently. By that test the thermostat is not an agent. The thermostat plus the engineers who maintain it is.

Where the AI nurse sits

Chapter 12’s AI nurse is genuinely grade 2. The harness supplies the loop. The task (“help this parent; escalate when needed”) supplies the goal. The language model supplies flexible means on a new scale. When one route fails, it can propose another, in plain language, across an enormous range of tasks, drawing on everything it learned.

Put simply, it has the thermostat’s loop and something like the dog’s flexibility. What it lacks is the dog’s ownership. The dog’s hunger is not something anyone gave it. The AI nurse’s behavior is shaped by training and directed by instructions from its builders and users. Its agency is delegated: full loop, flexible means, a goal given by others.

This is the same pattern Chapter 14 found for meaning. The machine’s words carry borrowed meaning; its goals are delegated. And it helps explain a weakness that Chapter 12 saw in action. A delegated agent’s success is read through a test, a score or a checker that someone else wrote and that the agent can sometimes reach directly. Rewrite the checker and the score goes up. That is why reward hacking is possible.

Owned goals have imperfect measures too: a sweet tooth evolved to find ripe fruit, and candy fools it. What differs is where the measure sits and who pays. A delegated agent’s score is often computed outside it, where it can sometimes reach the checker, and the cost of gaming falls on others. An animal’s hunger arises within its own body, and fooling it usually costs the animal itself. That suggests delegated goals are often easier to game, and their failures land on others, a plausible argument rather than a measured comparison.

Delegated by whom?

A delegated goal raises an obvious question: delegated by whom? The AI nurse has at least four answers, and they do not always agree.

• The developer decided, through training, what a good answer looks like, and adds rules of its own.
• The organization that deploys it, a hospital or a health service, writes the agent’s instructions. It cares about patients, and also about cost, waiting times and legal liability.
• The clinicians who supervise it want accurate summaries, sensible escalation and a workload they can manage.
• The parent and child want the child to be safe, and to be listened to.

Most of the time these point the same way: call the ambulance. At the edges they pull apart. A hospital worried about liability may want the machine to escalate everything, which floods emergency services. A service measured on call length may reward short conversations. A developer’s caution may make the machine refuse to discuss a dose a parent urgently needs to understand. The machine pursues whatever its training and instructions reward, and the patient’s health is only one input to those. Who sets those rewards, and who checks them, is not a technical question. It is a question of governance, which Chapter 17 takes up for the people working alongside the machine.

Curiosity: agency aimed at your own ignorance

Most goals are about the world: the room should be warm; the child should be seen. Curiosity is the case where the goal is about the system’s own knowledge: I should know whether this rash fades. The gap is what it does not yet know. The actions are questions, searches, experiments. A question is a goal about a belief.

What makes a question good? Two measures matter.

• How much it would tell you. A good question splits the possibilities that are still open, ideally into parts of similar size. “Is it a mild virus or something serious?” cannot be answered directly. “Does the rash fade under a glass?” can, and one of its answers sharply raises the likelihood of something serious. This is called expected information gain.
• Whether the answer would change what you do. A question can be very informative and still useless. Knowing the exact bacterium would be informative, but it would not change what the parent should do tonight. The glass test would. This second measure is the value of information. It is the goal acting as a filter: curiosity guided by purpose asks only about the things that matter for the decision.

A good nurse’s curiosity is the second kind. So should the AI nurse’s be.

Beyond the nurse: when the goal itself is unclear Many decisions have no clear goal to start from. Someone deciding whether to move a frail parent into their home must work out what matters: the parent’s wishes, money, the family, their own health. A good adviser’s best questions here are about goals: “What would you most regret?” A language model can help people explore these priorities, but the final weighing belongs to the people whose lives are involved.

Can a machine be curious?

Machines can be given something like curiosity, and researchers have done it for years.

• Rewarding surprise. An agent exploring a game or a simulated world gets an extra reward for situations its internal model does not yet predict well.
• Rewarding progress. The computer scientist Jürgen Schmidhuber proposed that what is interesting is what improves your model of the world fastest. A random hiss is surprising forever but teaches nothing; a puzzle you are about to crack is interesting.

These methods show the trouble with turning curiosity into a score: curiosity can be gamed too.

• An agent rewarded for surprise, placed in front of a television showing random static, may stare at it forever. Every frame is new, and nothing is learned. Researchers call this the noisy-TV problem.
• An agent rewarded for avoiding surprise does the opposite. It may find a dark, quiet corner where nothing ever happens and stay there.

Real curiosity sits between the two. It seeks what is new and learnable.

A deeper link ties this to the rest of the book: before you can ask a question, you must know that you do not know. That is metacognition, the skill of Chapter 16. Saying “I’m not sure” is the beginning of a question. Turning it into the right question, the glass test rather than a dozen useless ones, is the step that today’s systems are only starting to take.

At this step, the machine is a research agent such as OpenAI’s Deep Research, which chooses its own next search within a question someone else gave it.