Ussama Assad Start a project

Domain 5 of 5 · 15% of the exam

Context Management & Reliability

What it covers

The context budget, escalation, errors across agents, degradation in long sessions, human review and confidence calibration, provenance. About 15% of scored items, by the official exam guide.

Know cold

One idea per card. Open the ones you aren't sure of.

One budget

System prompt, history, tool results and the response all share the context window. When answers get shallow, add up what is in there. The usual culprit is a large tool result; trim it before it enters context.

Lost in the middle

Models use information at the start and end of a long context better than information in the middle (Liu et al., 2023). Anthropic's guidance is to put long documents at the top and the question at the end.

/compact is lossy

It replaces the history with a summary, and summaries drop exact numbers, IDs and decisions. Keep those in a scratchpad file, and tell /compact what to keep. The project-root CLAUDE.md is re-read after compaction; path-scoped rules come back only when a matching file is touched again.

Subagents for noisy reading

Let a subagent read the 40 files and return a summary, so the main context stays clean.

Escalate on explicit conditions

The customer asked for a person, the case is outside policy, or the agent is stuck after its allowed attempts. Not on sentiment or the model's own confidence score. When a lookup returns several matches, ask for another identifier rather than guessing.

"Failed" is not "found nothing"

A subagent that times out returns a structured failure with what it tried and any partial results. Handle what you can locally, escalate the rest, and mark the gap in the final output.

Aggregates hide segments

Before you cut human review, check accuracy by document type and field, calibrate the confidence threshold on labelled data, and keep sampling after launch.

Provenance per claim

Keep each claim tied to its source, date and method. When sources conflict, show both. Never average them or drop one quietly.

A wrong number has three possible causes

Say a weekly report says 48 orders shipped and the orders table shows 51. Someone may have counted from memory instead of querying. Or the wrong rows got into the table, for example a filter that quietly drops orders with no postcode. Or the table is right and the report copied it wrong. Each needs a different fix, and fixing one leaves the other two open. Practice scenarios about counts, totals and accuracy often hinge on which of the three you are looking at.

Traps

  • Trusting the aggregate. 96% overall can hide one document type at 70%. Look for the option that breaks the number down.
  • Escalating on mood or self-rated confidence. The exam guide calls both unreliable proxies. Escalate on conditions you can state and check.
  • Averaging two sources. An average invents a value neither source states. Show both, each with its source and date.
  • Trusting /compact with exact values. Summaries drop numbers and IDs. Keep them in a file.
  • Reporting a failure as "found nothing". A timed-out subagent returns a structured failure, and the gap is marked in the output.

Practice: 4 questions

Q17 · Domain 5 · Context Management & Reliability

A billing support agent can hand a conversation to a human. Which two are sound triggers for that handoff? (Select 2)

  1. A. The customer explicitly asks to speak to a person.
  2. B. A sentiment score on the customer's last message drops below a set threshold.
  3. C. The request falls outside documented policy, or the agent hasn't resolved it within its allowed attempts.
  4. D. The model rates its own confidence in its answer below 70%.
  5. E. The customer's message runs longer than 150 words.
Answer and explanation

Answer: A and C. Escalate on conditions you can state and check: the customer asked, the case isn't covered, or the agent is stuck. The public exam guide calls sentiment and self-reported confidence unreliable proxies for how complex a case is. Message length tells you nothing about whether a human is needed.

  • A. Right. An explicit request for a person is a condition you can state and check.
  • B. The public exam guide calls sentiment an unreliable proxy for how complex a case is.
  • C. Right. Outside documented policy, or stuck after the allowed attempts, are both conditions you can state and check.
  • D. Self-reported confidence is the other proxy the exam guide calls unreliable.
  • E. Message length tells you nothing about whether a human is needed.

Q18 · Domain 5 · Context Management & Reliability

Two sources disagree on when a supplier was founded. A company-registry record dated 2024 says 2011; a 2019 news article says 2009. Your research agent is writing the supplier profile. What should it do?

  1. A. Use 2010, the midpoint of the two, and mark the year as approximate in the profile.
  2. B. Report both years with their source and date, and flag the conflict for the reader.
  3. C. Keep the registry's year and drop the news article from the profile without comment.
  4. D. Leave the founding year out of the profile entirely, since the sources can't agree.
Answer and explanation

Answer: B. Keep each claim tied to its source and date, and show conflicts. Averaging invents a year neither source states. Dropping one silently hides evidence the reader may need. Leaving the field out loses both. Saying the registry is probably more reliable is fine, as long as both values stay visible.

  • A. Averaging invents a year neither source states.
  • B. Right. Each claim stays tied to its source and date, and the reader sees the conflict. Saying the registry is probably more reliable is fine, as long as both values stay visible.
  • C. Dropping one source silently hides evidence the reader may need.
  • D. Leaving the field out loses both pieces of evidence.

Q19 · Domain 5 · Context Management & Reliability

Three hours into a Claude Code debugging session you run /compact. Afterwards Claude has lost the exact error codes and the config values you agreed on, and its advice turns generic. What prevents this next time?

  1. A. Switch to a model with a larger context window so that compaction happens less often.
  2. B. Never compact at all, and start a brand-new session whenever the context fills up.
  3. C. Tell Claude at the start of the session to remember every number exactly as given.
  4. D. Keep exact values in a scratchpad file as you go, and tell /compact what must survive.
Answer and explanation

Answer: D. Compaction replaces the history with a summary, and summaries drop exact detail. A file survives compaction, and /compact accepts instructions about what to preserve. A bigger window only delays the same loss, a fresh session throws everything away, and an instruction to remember is advice.

  • A. A bigger window only delays the same loss. Compaction still replaces the history with a summary.
  • B. A fresh session throws everything away, including what you wanted to keep.
  • C. An instruction to remember is advice, and the summary can still drop the numbers.
  • D. Right. A file survives compaction, and /compact accepts instructions about what to preserve.

Q20 · Domain 5 · Context Management & Reliability

An invoice-extraction pipeline scores 96% accuracy overall. The team wants to stop human review of high-confidence outputs. What should happen first?

  1. A. Wait until the 96% figure has held steady for a full quarter, then switch review off.
  2. B. Hire enough extra reviewers that every single output can still be checked by hand before it ships.
  3. C. Check accuracy by document type and field, calibrate on labelled data, keep sampling after launch.
  4. D. Lower the confidence threshold so that a larger share of the outputs qualify as high-confidence.
Answer and explanation

Answer: C. An overall figure can look fine while one document type or field fails badly, because the large good segments dilute the small bad one. Check by segment, make sure the confidence score means something against labelled data, and keep a random sample under review after launch to catch new failure patterns. A stable aggregate is still an aggregate, more reviewers means no automation, and a lower threshold widens the risk.

  • A. A stable aggregate is still an aggregate. One weak document type can hide inside it the whole time.
  • B. Checking every output by hand means no automation at all.
  • C. Right. An overall figure can look fine while one document type or field fails badly. Check by segment, make sure the confidence score means something against labelled data, and keep sampling after launch.
  • D. A lower threshold sends more outputs past review and widens the risk.