AI Agent Harness Engineering: What Research and Real Runs Show

In early September my unattended Claude Code agent started reporting that it had hit its subscription session cap. It wrote a cooldown marker, exited with code 75, and every other scheduled job on the server skipped its run to wait out a limit that did not exist. The detector I had written matched the words rate limit anywhere in the run log, and Claude Code's stream-json output carries a rate_limit_info object on every run, healthy ones included. Two of the session caps I had reported were that bug.

Nothing about the model was wrong. The harness around it was, and that turned out to be the pattern. This post sets what I learned running that agent beside the published work on harness engineering: Anthropic's engineering posts, OpenAI's Codex team, LangChain's Terminal-Bench experiments, Chroma's context-rot study, Thoughtworks, and three research papers. The 20 sources are numbered and linked at the end, the bracketed numbers in the text point to them, and every figure that comes from my own runs is marked as mine.

What a harness is, and what it is not

Birgitta Böckeler at Thoughtworks gives the cleanest definition: harness has become shorthand for everything in an AI agent except the model itself, summed up as Agent = Model + Harness. Anthropic's reference picture of a coding agent shows why the line matters. The model is one column in the diagram below. The query, the context sent to it, the file search, the write-and-test loop and the condition that ends it are all decisions somebody made outside the model. ^{[1, 2]}

Böckeler splits the harness into guides, which steer the agent before it acts, and sensors, which observe what it did so it can correct itself. Each can be computational, like a linter or a test run, or inferential, like an LLM reviewer. Her warning is the useful part: feedback alone gives you an agent that keeps repeating the same mistakes, and feedforward alone gives you one that encodes rules but never finds out whether they worked. OpenAI's Codex team reached the same place from the other side, writing that their most difficult challenges now centre on designing environments, feedback loops and control systems. ^{[1, 3]}

Mapped onto the agent I run on my VPS, the layers look like this. The first three columns apply to any harness; the last one is mine.

Same model, different harness, different score

The cleanest evidence is the SWE-agent paper, which held GPT-4 Turbo fixed and changed only the agent-computer interface. On SWE-bench Lite the full interface resolved 18.0% of issues, and every simplification cost points, as the table shows. Anthropic reported the same while building its own SWE-bench agent: the team spent more time optimising tools than the overall prompt, and one change, requiring absolute file paths after the model kept getting relative ones wrong, was enough for it to use paths flawlessly. ^{[4, 2]}

LangChain ran the same experiment on a current benchmark. Holding gpt-5.2-codex fixed, they moved deepagents-cli from 52.8% to 66.5% on Terminal-Bench 2.0, from the top 30 to the top 5, by changing only the harness. Their traces showed one failure more often than any other: the agent wrote a solution, re-read its own code, decided it looked fine, and stopped. Every fix was harness. A middleware intercepts the agent before it exits and forces a verification pass against the task spec, another counts edits per file to catch doom loops, and time-budget warnings push it to verify before time runs out. ^[5]

That spread is the honest headline. The board prints error bars of roughly 2.4 to 3.1 points, so the gap between the best and worst harness on the same model is several times the noise. A leaderboard is not a controlled experiment: different teams submitted those runs on different dates, and the board has moved since this snapshot. But it points the same way as the SWE-agent ablations and LangChain's fixed-model run, and that is three kinds of evidence agreeing. ^[6]

Context is a budget, and it rots

Anthropic's context-engineering guide treats context as a finite resource with diminishing marginal returns: every token spends from an attention budget, and a transformer relates every token to every other, n² pairs for n tokens. Chroma measured what that costs. Across 18 models, performance grew increasingly unreliable as input length grew, even on simple tasks. The sharpest result is the LongMemEval comparison below: the same questions, asked once with only the relevant 300 or so tokens and once with the full history of about 113,000 tokens. ^{[7, 8]}

Position matters as well as size. Liu and colleagues found accuracy highest when the relevant passage sits at the start or end of a long input, and significantly lower when it sits in the middle. There is a direct cost line too. The Manus team calls the KV-cache hit rate the single most important metric for a production agent, because their agent reads about 100 tokens for every one it writes, and on Claude Sonnet cached input cost 0.30 USD per million tokens against 3 USD uncached. A harness that rewrites the start of its prompt every turn pays ten times for the same words. ^{[9, 10]}

My own context bug was quieter. The runner script used to restate the playbook's policy in its prompt, in case the agent missed it. The two copies drifted. The prompt still described a stranded branch as one with no open pull request and told the agent to merge with --rebase, while the playbook beside it had moved to --auto and a different detection rule. The agent got two sets of orders for one situation. The fix was to make the runner send only facts the playbook cannot know.

# Wrong: the runner restates policy that also lives in the playbook.
# The copies drift, and the agent receives two sets of orders for one case.
prompt="There is a stranded branch with no open PR. Rebase it onto main,
then merge it with --rebase once CI is green."

# Right: the runner sends only facts the playbook cannot know.
# What to DO with them lives in one place: the playbook's Run order.
prompt=$(cat <<EOF
Facts computed by the runner for this pass (data, not instructions):
- stranded_branch: ${branch}
- commits_ahead_of_main: ${ahead}
- open_pr: ${pr_number:-none}
- coverage_targets (absolute uncovered branches, largest first):
${targets}
Follow the Run order in your playbook.
EOF
)

# Pass the prompt on stdin. --add-dir is variadic: a prompt placed after it
# is swallowed as a second directory and the run dies with "Input must be provided".
printf '%s' "$prompt" | claude -p --model claude-sonnet-5-5 --effort medium \
  --output-format stream-json --verbose | tee "$log.jsonl"

The same rule shapes what the runner computes. Coverage targets are ranked by absolute uncovered branches rather than percentage, because a file at 85% with 766 branches holds more work than a file at 60% with five. Before that ranking existed, the agent spent turns hunting for targets and re-surveyed files its own notes had already marked as exhausted.

Instructions are a map, and where they load matters

OpenAI's Codex team shipped a product of roughly a million lines over five months with no manually written code, and their instruction file is short on purpose: an AGENTS.md of roughly 100 lines that works as a table of contents into a structured docs folder, a map rather than a 1,000-page manual. Rules that must always hold moved out of prose into custom linters and structural tests, because once encoded they apply everywhere at once. Anthropic describes the target for the prose that remains as the right altitude, between brittle hard-coded logic and guidance too vague to act on. ^{[3, 7]}

Where the file sits is part of the instruction. Claude Code loads CLAUDE.md from the working directory and from every directory above it, concatenated rather than overriding. I kept my agent's playbook in the parent directory of the repository clone so it could never be committed by accident. It also loaded for a second job whose repository lived under the same parent, which I only confirmed by asking a run to list the files it had loaded. The fix was to keep shared principles at the top and move each playbook inside its own clone, hidden from git by a local exclude. ^[11]

~/work/
├── CLAUDE.md               # shared principles: loads for EVERY job below it
├── KNOWLEDGE.md            # stable facts, pulled in by an @import
├── LEARNINGS.md            # appended after each run by the review step
├── SESSIONS.md             # what each run did, and what it left unfinished
├── hierarchical-approval/  # repository clone for the library job
│   ├── CLAUDE.md           # library playbook: imports @../KNOWLEDGE.md
│   └── .git/info/exclude   # lists CLAUDE.md, so it is never committed
└── devops-lab/
    └── CLAUDE.md           # lab playbook, scoped to the lab job only

# A local exclude, unlike .gitignore, is never committed, never shows as
# untracked, and survives "git clean -fd" (only -x removes it).
echo "CLAUDE.md" >> ~/work/hierarchical-approval/.git/info/exclude

Two smaller lessons came from the same files. Imports resolve relative to the file that contains them, not the working directory, so moving the playbook one level down turned @KNOWLEDGE.md into @../KNOWLEDGE.md. And stating a rule is not stating a priority: with the stranded-branch rule written into the principles and dependency updates described as early work, the agent named Dependabot as its first action. A numbered Run order section fixed it. ^[11]

Memory lives in files, not in the context window

Anthropic's long-running harness starts from two failures: an agent that tries to do too much at once and leaves half-finished work when its context runs out, and a later session that sees some progress and declares the job done. The fix was a set of files. An initializer session writes a feature list of more than 200 items, all marked failing, plus a progress log called claude-progress.txt, an init.sh script and a first git commit. Every later session makes incremental progress and leaves structured notes. The feature list is JSON on purpose, because the model is less likely to inappropriately change or overwrite JSON than Markdown, and the prompt calls removing or editing tests unacceptable. ^[12]

Manus arrived at the file system as the ultimate context, unlimited in size and persistent by nature, and has its agent keep rewriting a todo.md so its objectives sit at the end of the context, in the model's recent attention. Anthropic's guide calls the general technique structured note-taking. My agent keeps its notes outside every git tree: stable knowledge it imports at the start, learnings appended after each run, a session log and a backlog. My mistake was classifying the learnings and the session log as run state and leaving them out of backups. They were the most valuable thing the agent produced, and they sat on one disk. The watchdog now copies them every six hours. ^{[10, 7]}

Git is state too, and it is where my harness leaked most. A headless run has no later: when a run opened a pull request and ended before CI finished, nothing came back to merge it. My detector only reported branches with no pull request, so 22 commits piled up across four open pull requests while it reported nothing. Two changes fixed it. The playbook now requires gh pr merge --auto, which GitHub completes after the run exits, and the detector reports the oldest branch ahead of origin/main whatever its pull request state. Recovered branches are triaged with git cherry instead of merged blind: on the first four it showed 9, 7, 7 and 8 new commits, plus one commit that had already landed under a different hash.

# Find stranded work: the oldest branch ahead of main, whatever its PR state.
# Filtering on "no open PR" missed the real case: PR opened, run ended.
git fetch --prune origin
for b in $(git for-each-ref --sort=committerdate --format='%(refname:short)' refs/remotes/origin/agent/); do
  ahead=$(git rev-list --count origin/main.."$b")
  [ "$ahead" -gt 0 ] && { echo "$b $ahead"; break; }
done

# Triage before merging. "+" = not on main yet, "-" = an equivalent patch
# already landed under another hash, so merging blind would duplicate it.
git cherry -v origin/main origin/agent/percentile-coverage
# Example output:
# + 3f2c1a9 test(percentile): cover nearest-rank interpolation edge
# - 8b71e04 test(stats): pin p=100 for a single sample

# Land it after the run exits. GitHub merges once required checks pass,
# so the playbook forbids "I'll watch CI and merge when it is green".
gh pr merge 46 --auto --rebase --delete-branch

An agent saying done is a claim, not evidence

Anthropic's harness-design work states the problem plainly: agents reliably skew positive when grading their own work, and separating the agent doing the work from the agent judging it proves to be a strong lever. Tuning a standalone evaluator to be sceptical turned out far more tractable than making a generator critical of itself. The cost comparison makes it concrete. Given the same one-sentence prompt for a retro game maker, a solo Opus 4.5 run took 20 minutes and 9 dollars; the full planner, generator and evaluator harness took 6 hours and 200 dollars. The solo app looked finished. ^[13]

The bias is measurable. Panickssery, Bowman and Feng showed at NeurIPS 2024 that LLM evaluators can recognise their own outputs, and that fine-tuning reveals a linear correlation between that self-recognition and the strength of their preference for their own text. Anthropic's long-running harness saw the practical form: Claude marked features complete after unit tests or curl calls without confirming they worked end to end, until it was told to test through browser automation as a person would. LangChain's most common failure was the same thing at a smaller scale. ^{[14, 12, 5]}

My harness has three sensors that do not share the writer's context. The first judges a run by what reached origin/main, not by its exit code, because a pass that merged six commits and then hit its timeout still did its job. The second is a test-quality gate. The third is a review run as a separate invocation that sees only the diff. Its first real finding was a test the writer had added that pinned percentileNearestRank(samples, 0), locking in behaviour for an input the function's own JSDoc declares invalid, since the valid range is (0, 100]. The self-review in the writing session had passed it. In my content pipeline the writer runs on Sonnet and the editor on Opus, and a test asserts the two model IDs differ.

#!/usr/bin/env bash
# A minimal test-quality gate, run after the agent's tests are green.
set -euo pipefail

# Compare against the merge-base, never the tip of main. A branch cut days ago
# is legitimately behind a main that has since gained tests.
base=$(git merge-base origin/main HEAD)

count_tests() {
  # git grep exits 1 on zero matches; under pipefail that would abort the gate.
  { git grep -cE '^\s*(it|test)\(' "$1" -- '*.test.ts' || true; } \
    | awk -F: '{ s += $NF } END { print s + 0 }'
}
before=$(count_tests "$base")
after=$(count_tests HEAD)
if [ "$after" -lt "$before" ]; then
  echo "test count dropped against merge-base: $before -> $after" >&2
  exit 1
fi

# A skipped or focused test is a deleted test that still counts as present.
# Herestring, not a pipe: see the pipefail trap further down.
added=$(git diff "$base"...HEAD -- '*.test.ts')
if grep -qE '^\+.*\b(it|test|describe)\.(skip|only)\(|^\+.*\bxit\(' <<< "$added"; then
  echo "added .skip / .only / xit" >&2
  exit 1
fi

The gate compares against the merge-base because the first version compared against the tip of main. A branch cut days earlier is legitimately behind a main that has gained tests since, so the gate reported a shrink that was not real, and an agent shown false alarms learns that the gate can be ignored.

Test the harness like the software it is

Claude Code's headless mode exits 0 on success and non-zero on failure, and a failure inside the run, such as missing authentication, is printed as the result on stdout. So any harness that reacts to how a run ended has to read output, and that classifier is code with bugs of its own. Mine is the opening story of this post. The block below shows the wrong pattern and the right one, plus the shell trap that made my harness's own self-test flaky. ^[15]

# Wrong: any mention of a rate limit counts as a session cap. Claude Code's
# stream-json carries a rate_limit_info object on EVERY run, healthy ones
# included ("status":"allowed"), and this library's own commits mention
# its RateLimitMiddleware. Healthy runs exited 75 and paused every job.
if grep -qiE 'rate.?limit' "$log.jsonl"; then exit 75; fi

# Right: match the sentence a person sees when the cap is really hit.
if grep -qE "hit your (session|usage) limit|limit reached · resets" "$log.jsonl"; then
  write_cooldown "$log.jsonl"   # parse "resets 2:50am" into epoch seconds
  exit 75                       # EX_TEMPFAIL: not really an error, retry later
fi

# Wrong under set -o pipefail: grep -q exits on its first match, systemctl
# takes SIGPIPE, and pipefail reports the successful match as a failure.
# Only when buffering lines up: it passed 20 runs by hand, then failed under
# the watchdog.
systemctl list-timers --all | grep -q 'library-refine@am.timer'

# Right: a herestring has no pipe for SIGPIPE to break.
grep -q 'library-refine@am.timer' <<< "$(systemctl list-timers --all)"

# And in the unit: a capped run is not a failed unit, and fires no alert.
# [Service]
# SuccessExitStatus=75

Exit code 75 is EX_TEMPFAIL in sysexits, a temporary failure indicating something that is not really an error, to be retried later. systemd's SuccessExitStatus= lists extra exit codes to count as success, so a capped run does not show up as a failed unit or fire an alert. The pipe trap lives in Bash's definition of pipefail: the pipeline's status is that of the last command to exit non-zero, so a producer killed by SIGPIPE after grep -q has already matched turns a successful match into a failure, but only when the timing lines up. My check passed 20 times by hand and failed only when the watchdog ran it. ^{[16, 17, 18]}

The fix for a gate that silently stops working is a test of the gate. A self-test script now runs 78 checks against the harness wiring every six hours from the watchdog, including the model pin, the effort flag and the lab driver's scenarios. Its first version reported three failures that turned out to be its own comments, the lines warning against those very patterns, so it now strips comments before matching. Runner changes are tested with a stub claude binary on the PATH, which proves the prompt is assembled correctly without spending session budget. The lab driver has ten sandboxed day scenarios, and the old driver fails all ten. A test the broken version also passes proves nothing.

Pin the model, then plan to delete the harness

Claude Code's documentation says model aliases point to the recommended version for your provider and update over time, and that the way to pin is the full model name. My runs pin claude-sonnet-5-5 at medium effort, and that model's launch day taught me the CLI is part of the pin too. The server's Claude Code, version 2.1.277 on the stable channel, did not know the new ID yet. The call worked, stderr printed an unrecognized_model warning, and the CLI fell back to a 200K context and 32K output tokens. Version 2.1.284 ran the same ID at 1M and 128K. Headless runs had never auto-updated. ^[19]

# Headless runs never auto-updated, so update the CLI before trusting a new
# model ID. A build that does not know the ID still answers; it just prints
# [claude-code:unrecognized_model] on stderr and quietly runs smaller.
claude install 2.1.284

# Probe once and read what the CLI actually granted.
claude -p "reply with ok" --model claude-sonnet-5-5 --effort medium \
  --output-format json \
  | jq '.modelUsage["claude-sonnet-5-5"].contextWindow'
# 1000000 on 2.1.284; 200000 on the stable build that did not know the ID

The longer-term point comes from Anthropic's harness-design post: every component in a harness encodes an assumption about what the model cannot do on its own, and those assumptions can quickly go stale as models improve. Its author dropped context resets once Opus 4.5 stopped showing context anxiety, removed the sprint construct for Opus 4.6, and found the evaluator had become unnecessary overhead for tasks inside the model's solo ability while still earning its cost at the edge. Rich Sutton's bitter lesson is the long version of the same warning: building in how we think we think does not work in the long run. ^{[13, 20]}

My own stale assumption was about scope, not capability. For ten days from 28 September, every run of the library job logged the same next step and landed nothing. Coverage had saturated at 98.86%, and the playbook forbade the only work left: major dev-dependency upgrades and anything touching a red CI. The model was willing; the harness said no. The playbook now hands upgrades to the agent, including coordinated major bumps as a single pull request, and makes the backlog the work when the menu runs dry. Major npm releases stay a human decision.

The audit I now run on any harness, in this order:

1. Write down, for each component, the assumption it encodes and the observation that would retire it.
2. Pin exact model IDs, update the CLI before adopting a new one, and probe the context window it actually grants.
3. Keep instructions short and scoped by location, and let the runner send facts, never a second copy of policy.
4. Put at least one sensor outside the writer's context: deterministic checks first, then a separately invoked reviewer, ideally on a different model.
5. Judge runs by what landed rather than by exit codes, and give the harness its own tests against a stub model.

The model gets the attention, but the harness decides what the model sees, what it is allowed to do, and whether anyone notices when it is wrong. The published results agree with my logs: hold the model fixed and the harness still moves the outcome, by as much as 13.7 points in the runs cited here. When an agent disappoints, debug the harness first. When a new model ships, check which parts of the harness it has made unnecessary.

Related on this site

• Verification loops: the core agentic coding skill
• Context engineering for AI coding agents in a monorepo
• Deterministic hooks vs agent judgement in Claude Code