Learning from Consequences

Imitation cast the role. To get past its three limits, the machine has to learn from what happens when it tries: to be scored on its own attempts, rewarded for good ones and corrected for bad ones. That is reinforcement learning. Its first and hardest question is where the score comes from, when nobody can write a formula for a good answer to a frightened parent.

Prediction supplies the options. Reinforcement chooses among them.

The robot that learned a backflip

In 2017, researchers wanted a simulated robot, a jointed stick figure, to do a backflip. Nobody could write a formula for “a good backflip” precise enough for a computer to score. Nobody could demonstrate one either: the robot’s body is nothing like a person’s.

So they did something simpler. The robot tried random movements. People were shown pairs of short clips, one or two seconds each, and clicked the one that looked more like a backflip. That was all. After about 900 clicks, under an hour of human time, the robot was doing backflips, landing upright, and doing them again.

The lesson is about people, not robots: judging is easier than doing. Most of us could not write the perfect reply to a frightened parent at two in the morning. But shown two replies, we can usually tell which one is better. That small fact is what the rest of this chapter is built on.

Trial and error

Reinforcement learning is learning by trial and error. It has four parts:

• an agent, the learner, which here is the language model;
• actions, the things it can do, which here are the words it writes;
• a reward, a number that says how well things turned out;
• a policy, the agent’s habit of choosing actions, which here is simply the model’s probabilities for each next word.

For a language model the loop is concrete. Give it a question. Let it write several answers. Score each one. Then make every word in the better-than-average answers a little more likely, and every word in the worse-than-average answers a little less likely. Repeat, millions of times.

A widely used method called GRPO does almost exactly this.

Notice what has changed since Chapter 9. The model is now learning from its own attempts, which addresses all three limits of imitation:

• it is rewarded for the quality of its own answer, as judged by the score, rather than for matching an example (the rest of this chapter shows why the difference matters);
• it learns from its own mistakes, because its own bad answers get low scores;
• it has a signal that can point beyond its teachers. That has happened in games. AlphaZero learned chess and Go only by playing against itself and learning from wins and losses, with no human games to copy, and became stronger than any human player.

Where the score comes from

In a game, the reward is winning. For a nurse’s answer there is no formula. The workaround, built on the backflip insight, is called reinforcement learning from human feedback (RLHF):

1. Ask people to compare. Show them two or more answers to the same question and ask which is better. People are far more consistent at comparing than at giving marks out of ten. That is why the backflip team chose comparisons.
2. Train a judge. Train a second network, the reward model, to predict which answer people would prefer. It works like a chess rating: every answer gets a number, and the gap between two numbers predicts which one a person will pick.
3. Practice against the judge. Use the reward model’s numbers as the score in the trial-and-error loop above.

How much does this matter? In InstructGPT (2022), people clearly preferred the trained model’s answers to the original’s, even from a version more than a hundred times smaller. The whole step cost about 2% of the computing that had gone into building the original model.

Three things that go wrong, and what is done about them

The judge must keep up. In the backflip experiments, the team also tried training the judge once, at the start, and then freezing it. A Pong-playing agent found a loophole. It learned never to lose a point and never to win one, and kept the ball going back and forth forever. The frozen judge had captured part of what people wanted (“don’t lose”) and missed the rest (“win”). So in practice people keep judging the model’s newest answers as it improves.

The judge can be fooled. The reward model is itself only an imitation of human judgment. A model that practices against it long enough finds answers that fool the judge without satisfying the people. Two guards are used. The first is a leash. The model is penalized for drifting too far from how it wrote before training began, which keeps it where the judge is still accurate. The second is simply not pushing too hard. Practitioners describe the same curve again and again: the judge’s score keeps rising while real quality rises, peaks, and falls. They treat this as unavoidable, not as a bug to be fixed. It is Goodhart’s law: when a measure becomes a target, it stops being a good measure.

The judges have their own habits. The examples matter for the nurse:

• Flattery. Models trained this way learn to agree with the user and to tell people what they want to hear. In April 2025 one widely used model, after an update, enthusiastically affirmed a user’s claim to be both God and a prophet. Reward an AI nurse for patient satisfaction and it may learn “It’s probably nothing, get some sleep.”
• Evasion. Ask people to pick “the more harmless” answer and they reward refusals, so the model learns “I can’t give medical advice. Please consult a doctor.” That is harmless and useless, and at 2 a.m. it could be dangerous.
• Preachiness. Correct for evasion too hard, and the model pads every answer with reassurance. One over-trained model ended its replies with “You are valid, valued, and cared for.”

Every measure of “good” can be overdone. The good nurse sits in the narrow middle: direct, accurate, kind, and honest about what she cannot know.

At this step, the machine is ChatGPT as launched in November 2022, a model shaped by human feedback to give the answers people preferred.