Part VI

Living with It

The machine joins the team. Will it replace the nurse, or relieve her?

Part VI · Chapter 17

The Machine on the Ward: Working in a Team

The machine never works alone. This chapter follows the 2 a.m. case past the first conversation, into the team of people who must read, check and act on what the machine writes.

On a ward the machine joins a team. It helps when someone filters what it writes and someone signs for what is done; without that, it floods the people it was meant to help.

The case does not end at 2 a.m.

Chapter 12 followed one conversation. The machine read the parent’s message and told the parent to call an ambulance. Then it looked up the child’s record and alerted the doctor on call. Then the conversation ended. The case did not.

Follow the child. The ambulance crew reads a summary on the way. A triage nurse in the emergency department takes over, then a children’s doctor. By morning the child is on the children’s ward, and the night nurse hands over to the day team at eight. A discharge letter goes to the family doctor. The call-back the machine booked comes the next morning, and a week later someone checks on the child again. A dozen people touch the case. The machine touched it first, and may touch it again at every step: a summary here, a draft letter there, an alert in between.

The machine of Chapter 12 served one parent. This chapter asks what changes when it serves a team.

One case, many hands Six members of the team handle the 2 a.m. case in turn: the AI nurse, the ambulance crew, the emergency department, the children's ward, the morning handover and the family doctor. All of them read and write one shared record. The machine writes fast and people read slowly, so a nurse in charge filters what reaches whom. One case, many hands Everyone reads the shared record and writes to it. The machine writes fastest; someone must filter. AI nurse 2 a.m. Ambulance 2:20 a.m. Emergency 3 a.m. Ward night Handover 8 a.m. Family doctor next week The shared record notes, summaries, alerts, letters: one memory for the whole team One error, many copies A wrong entry is read as fact and repeated in every document. The flood The machine writes faster than people can read and check. The nurse in charge filters what reaches whom, checks facts at their source; a clinician signs.

Figure 17.1. One case, many hands. Everyone on the team reads the shared record and writes to it, and the machine writes faster than any of them. Someone has to filter what reaches whom.

Memory across days

Chapter 12 named three kinds of memory: the weights, the context and the notes. The weights usually stay fixed while the machine works. The context empties when the conversation ends. Tonight’s case survives only in what is written down: saved notes and the record.

A team already has a written memory: the patient record. On the ward the machine becomes one of its writers. That carries a duty. What the machine writes at 2 a.m., the day team reads at eight as fact. Chapter 11 noted that a model reads an early mistake as faithfully as a correct step. The record does the same for the whole team: a wrong dose entered at night is given again in the morning.

One story, many documents

One case produces many documents: the triage note, the ambulance handover, the emergency note, the ward plan, the discharge letter, the letter to the family doctor and the message to the parent. People write them at different times, in a hurry, and they drift apart. One says the fever started at midnight, another at ten.

A machine that drafts them all can keep them consistent: the same times, the same doses, the same allergy. It can also copy one error into every document. Consistency is not correctness, a distinction Chapter 14 drew for maps. Five documents that agree may all repeat one wrong entry. A good team checks a fact at its source, not in the last document that repeated it.

A good team also corrects at the source. Suppose the parent says the fever began at ten, not midnight. The nurse corrects the entry. Later drafts use the correction, and the people who read the earlier version are told what changed. The record keeps the correction and where it came from. Like the agent of Chapter 12, the team needs a loop that catches mistakes and acts on them.

The flood

A machine writes faster than people read. On a ward that can move the work from writing to reading and checking. Studies of alerts and drafting show why more output does not necessarily mean less work.

  • Alerts. Hospital software warns prescribers about drug interactions, doses and allergies. A review of 23 studies found that prescribers override between 46% and 96% of these alerts. Many overrides were justified. That is the problem: a warning that is usually unnecessary teaches people to dismiss it, including the rare one that matters. This is called alert fatigue.
  • Warnings that flood and miss. A sepsis-warning model used in hundreds of US hospitals was tested independently at one of them in 2021. It flagged 18% of all hospitalized patients, yet missed two-thirds of those who developed sepsis. To find one case, staff had to assess about eight flagged patients.
  • Drafts. Messages from patients to their clinicians grew 157% during the pandemic. At Stanford in 2023, a language model drafted replies for 162 clinicians to edit. They used about one draft in five. Their scores for burden and burnout fell, but they spent no less time. Saving effort and saving minutes are different benefits.

The first two studies concern older software, not language models. Their lesson carries over: whatever produces more text than people can check will, in time, go unchecked.

Chapter 16 warned that people who supervise a machine can stop checking because it is usually right. This is the reverse: people who cannot keep up with what it produces. Both end the same way. The human check fails.

When teams do worse

It seems obvious that a person and a machine together should beat either alone. Often they do not. A 2024 meta-analysis pooled over a hundred experiments. On average, the pair did better than the person alone, but worse than the better of the two working alone. Losses came mainly in tasks that ended in a decision; tasks that created content, such as drafting text, showed gains. When the person alone outperformed the machine alone, the pair tended to gain. When the machine was the stronger, the pair tended to lose.

These were experiments, not wards. They show why the benefits of working together must be tested rather than assumed. Adding a machine to a team does not add its skill to the team’s. The design decides: who drafts, who decides and who checks what. The same problem appears in a law firm, a school or a software team: helping each person produce more does not by itself help the group work better.

The nurse in charge

Every ward already has a person who filters. At night the nurse in charge decides what reaches the doctor and what can wait until morning. She turns twelve bleeps into three calls.

Filtering is not enough if she must first read everything the machine writes. The better fix starts at the source. At handover, the machine produces one short update: what has changed, what has been checked and what still needs a decision. Routine copies stay in the record instead of becoming new messages. Important claims point back to their source, so the nurse can check them without rebuilding the whole case.

The machine changes this job without removing it. On a ward that uses one well, someone:

  • decides which alerts reach whom;
  • reviews the alerts people keep ignoring and removes or redesigns those that add nothing;
  • condenses the machine’s drafts before they reach busy colleagues;
  • checks a fact at its source before it spreads through the record;
  • signs for what is done.

The last point matters most. A machine can draft the discharge letter, but a clinician signs it. A clinician is responsible for what they approve. That approval does not remove the responsibilities of the hospital or of the system’s developer. Chapter 15 showed that the developer, the hospital, the clinicians and the patients each shape what the machine is for. On the ward those interests meet in one place: the team’s daily choices about what the machine may write, to whom, and with whose signature.

Back to the parent

What does the parent see? At best, one story told the same way by everyone they meet: the ambulance crew already knows about the penicillin allergy, the ward nurse already knows when the fever started, and the family doctor’s letter matches what the parent was told. The machine can make that possible. Only the team can make it true.

Notes and references

Notes

  • The alert-override range (46.2% to 96.2% across 23 studies) and the most common reasons given (“will monitor”, “patients have tolerated before”) are from Poly et al. (2020).
  • The sepsis-model figures (alerts for 18% of 38,455 hospitalizations; 67% of sepsis cases not flagged; about eight patients assessed per case when each patient is counted at first alert) are from Wong et al. (2021).
  • The inbox figures (a 157% rise in patient messages; 162 clinicians; mean draft use 20%; lower task-load and work-exhaustion scores; no change in time) are from Garcia et al. (2024), a five-week, single-site study without a control group.
  • The meta-analysis pooled 106 experiments and 370 effect sizes in its published version (Vaccaro, Almaatouq & Malone 2024).

References

  • Garcia, P., Ma, S. P., Shah, S., et al. (2024). Artificial intelligence–generated draft replies to patient inbox messages. JAMA Network Open, 7(3), e243201. https://doi.org/10.1001/jamanetworkopen.2024.3201
  • Goddard, K., Roudsari, A., & Wyatt, J. C. (2012). Automation bias: A systematic review of frequency, effect mediators, and mitigators. Journal of the American Medical Informatics Association, 19(1), 121–127.
  • Poly, T. N., Islam, M. M., Yang, H.-C., & Li, Y.-C. (2020). Appropriateness of overridden alerts in computerized physician order entry: Systematic review. JMIR Medical Informatics, 8(7), e15653. https://doi.org/10.2196/15653
  • Vaccaro, M., Almaatouq, A., & Malone, T. (2024). When combinations of humans and AI are useful: A systematic review and meta-analysis. Nature Human Behaviour, 8, 2293–2303. https://arxiv.org/abs/2405.06087
  • Wong, A., Otles, E., Donnelly, J. P., et al. (2021). External validation of a widely implemented proprietary sepsis prediction model in hospitalized patients. JAMA Internal Medicine, 181(8), 1065–1070. https://doi.org/10.1001/jamainternmed.2021.2626