# AI Agent Harness Engineering: What Research and Real Runs Show

> Twenty sources and one unattended Claude Code agent on what really moves agent results: the harness. Harness-only gains, context rot, self-grading bias, and the self-tests a harness needs.

URL: https://www.matthewswong.com/en/blog/ai-agent-harness-engineering-research/

- Author: Matthews Wong (https://www.matthewswong.com/en/about/)
- Published: 2026-10-18
- Updated: 2026-10-18
- Category: AI

> **Key takeaway:** An AI agent harness is everything around the model: instructions, tools, context, state files, verification and the runner. Published results show harness changes alone lifting one model 13.7 points on Terminal-Bench 2.0. Running an unattended Claude Code agent taught me the same thing: most failures I logged were harness bugs, not model mistakes.

In early September my unattended Claude Code agent started reporting that it had hit its subscription session cap. It wrote a cooldown marker, exited with code 75, and every other scheduled job on the server skipped its run to wait out a limit that did not exist. The detector I had written matched the words rate limit anywhere in the run log, and Claude Code's stream-json output carries a rate_limit_info object on every run, healthy ones included. Two of the session caps I had reported were that bug.

Nothing about the model was wrong. The harness around it was, and that turned out to be the pattern. This post sets what I learned running that agent beside the published work on harness engineering: Anthropic's engineering posts, OpenAI's Codex team, LangChain's Terminal-Bench experiments, Chroma's context-rot study, Thoughtworks, and three research papers. The 20 sources are numbered and linked at the end, the bracketed numbers in the text point to them, and every figure that comes from my own runs is marked as mine.

## What a harness is, and what it is not

Birgitta Böckeler at Thoughtworks gives the cleanest definition: harness has become shorthand for everything in an AI agent except the model itself, summed up as Agent = Model + Harness. Anthropic's reference picture of a coding agent shows why the line matters. The model is one column in the diagram below. The query, the context sent to it, the file search, the write-and-test loop and the condition that ends it are all decisions somebody made outside the model.

*Only the LLM column is the model. The loop condition, until tests pass, is a harness decision, and so is everything the interface chooses to send. Diagram:*

Böckeler splits the harness into guides, which steer the agent before it acts, and sensors, which observe what it did so it can correct itself. Each can be computational, like a linter or a test run, or inferential, like an LLM reviewer. Her warning is the useful part: feedback alone gives you an agent that keeps repeating the same mistakes, and feedforward alone gives you one that encodes rules but never finds out whether they worked. OpenAI's Codex team reached the same place from the other side, writing that their most difficult challenges now centre on designing environments, feedback loops and control systems.

Mapped onto the agent I run on my VPS, the layers look like this. The first three columns apply to any harness; the last one is mine.

Instructions

Guide

What the agent believes the job is

One playbook per repository, kept out of git

Tools and interface

Guide

Which actions are cheap and which are impossible

Headless claude -p, prompt on stdin, a pinned model

Context

Guide

What is in the window when the model decides

Facts computed by the runner, never restated policy

State between runs

Guide and sensor

What survives when the session ends

Knowledge, learnings and session logs outside the git tree

Deterministic checks

Computational sensor

Whether the work is real

CI, plus a test-count gate against the merge-base

Review

Inferential sensor

Whether the work is any good

A second invocation that sees only the diff

Runner

Plumbing

Whether any of it runs at all

systemd timers, a lock, and exit code 75 for a capped run

## Same model, different harness, different score

The cleanest evidence is the SWE-agent paper, which held GPT-4 Turbo fixed and changed only the agent-computer interface. On SWE-bench Lite the full interface resolved 18.0% of issues, and every simplification cost points, as the table shows. Anthropic reported the same while building its own SWE-bench agent: the team spent more time optimising tools than the overall prompt, and one change, requiring absolute file paths after the model kept getting relative ones wrong, was enough for it to use paths flawlessly.

Full interface: 100-line file viewer, summarised search, edit command with a lint check

18.0%

baseline

Edit command without the lint check

15.0%

-3.0 points

Viewer shows the whole file instead of 100 lines

12.7%

-5.3 points

Iterative search instead of summarised results

12.0%

-6.0 points

No edit command at all

10.3%

-7.7 points

LangChain ran the same experiment on a current benchmark. Holding gpt-5.2-codex fixed, they moved deepagents-cli from 52.8% to 66.5% on Terminal-Bench 2.0, from the top 30 to the top 5, by changing only the harness. Their traces showed one failure more often than any other: the agent wrote a solution, re-read its own code, decided it looked fine, and stopped. Every fix was harness. A middleware intercepts the agent before it exits and forces a verification pass against the task spec, another counts edits per file to catch doom loops, and time-budget warnings push it to verify before time runs out.

*Read the Model column, not the rank. GPT-5.3-Codex appears under four harnesses here, from 75.1% with Simple Codex to 64.7% with Terminus 2: a 10.4-point spread on one model. Screenshot:*

That spread is the honest headline. The board prints error bars of roughly 2.4 to 3.1 points, so the gap between the best and worst harness on the same model is several times the noise. A leaderboard is not a controlled experiment: different teams submitted those runs on different dates, and the board has moved since this snapshot. But it points the same way as the SWE-agent ablations and LangChain's fixed-model run, and that is three kinds of evidence agreeing.

> Before you switch models because an agent underperforms, change one harness component and rerun the same task set. In LangChain's run the most common failure was fixed by a check that fires when the agent tries to stop, not by a different model.

## Context is a budget, and it rots

Anthropic's context-engineering guide treats context as a finite resource with diminishing marginal returns: every token spends from an attention budget, and a transformer relates every token to every other, n² pairs for n tokens. Chroma measured what that costs. Across 18 models, performance grew increasingly unreliable as input length grew, even on simple tasks. The sharpest result is the LongMemEval comparison below: the same questions, asked once with only the relevant 300 or so tokens and once with the full history of about 113,000 tokens.

*The information needed to answer is identical in both bars. Only the amount of surrounding history changes, and Claude Opus 4 falls from 0.92 to 0.38. Chroma notes the Claude gap is driven largely by the models abstaining rather than guessing. Chart:*

Position matters as well as size. Liu and colleagues found accuracy highest when the relevant passage sits at the start or end of a long input, and significantly lower when it sits in the middle. There is a direct cost line too. The Manus team calls the KV-cache hit rate the single most important metric for a production agent, because their agent reads about 100 tokens for every one it writes, and on Claude Sonnet cached input cost 0.30 USD per million tokens against 3 USD uncached. A harness that rewrites the start of its prompt every turn pays ten times for the same words.

My own context bug was quieter. The runner script used to restate the playbook's policy in its prompt, in case the agent missed it. The two copies drifted. The prompt still described a stranded branch as one with no open pull request and told the agent to merge with --rebase, while the playbook beside it had moved to --auto and a different detection rule. The agent got two sets of orders for one situation. The fix was to make the runner send only facts the playbook cannot know.

The same rule shapes what the runner computes. Coverage targets are ranked by absolute uncovered branches rather than percentage, because a file at 85% with 766 branches holds more work than a file at 60% with five. Before that ranking existed, the agent spent turns hunting for targets and re-surveyed files its own notes had already marked as exhausted.

## Instructions are a map, and where they load matters

OpenAI's Codex team shipped a product of roughly a million lines over five months with no manually written code, and their instruction file is short on purpose: an AGENTS.md of roughly 100 lines that works as a table of contents into a structured docs folder, a map rather than a 1,000-page manual. Rules that must always hold moved out of prose into custom linters and structural tests, because once encoded they apply everywhere at once. Anthropic describes the target for the prose that remains as the right altitude, between brittle hard-coded logic and guidance too vague to act on.

*The left prompt enumerates cases until it meets the first one it missed; the right one gives the model nothing to act on. The middle names the goal, the tools and when to escalate. Diagram:*

Where the file sits is part of the instruction. Claude Code loads CLAUDE.md from the working directory and from every directory above it, concatenated rather than overriding. I kept my agent's playbook in the parent directory of the repository clone so it could never be committed by accident. It also loaded for a second job whose repository lived under the same parent, which I only confirmed by asking a run to list the files it had loaded. The fix was to keep shared principles at the top and move each playbook inside its own clone, hidden from git by a local exclude.

Two smaller lessons came from the same files. Imports resolve relative to the file that contains them, not the working directory, so moving the playbook one level down turned @KNOWLEDGE.md into @../KNOWLEDGE.md. And stating a rule is not stating a priority: with the stranded-branch rule written into the principles and dependency updates described as early work, the agent named Dependabot as its first action. A numbered Run order section fixed it.

## Memory lives in files, not in the context window

Anthropic's long-running harness starts from two failures: an agent that tries to do too much at once and leaves half-finished work when its context runs out, and a later session that sees some progress and declares the job done. The fix was a set of files. An initializer session writes a feature list of more than 200 items, all marked failing, plus a progress log called claude-progress.txt, an init.sh script and a first git commit. Every later session makes incremental progress and leaves structured notes. The feature list is JSON on purpose, because the model is less likely to inappropriately change or overwrite JSON than Markdown, and the prompt calls removing or editing tests unacceptable.

Manus arrived at the file system as the ultimate context, unlimited in size and persistent by nature, and has its agent keep rewriting a todo.md so its objectives sit at the end of the context, in the model's recent attention. Anthropic's guide calls the general technique structured note-taking. My agent keeps its notes outside every git tree: stable knowledge it imports at the start, learnings appended after each run, a session log and a backlog. My mistake was classifying the learnings and the session log as run state and leaving them out of backups. They were the most valuable thing the agent produced, and they sat on one disk. The watchdog now copies them every six hours.

Git is state too, and it is where my harness leaked most. A headless run has no later: when a run opened a pull request and ended before CI finished, nothing came back to merge it. My detector only reported branches with no pull request, so 22 commits piled up across four open pull requests while it reported nothing. Two changes fixed it. The playbook now requires gh pr merge --auto, which GitHub completes after the run exits, and the detector reports the oldest branch ahead of origin/main whatever its pull request state. Recovered branches are triaged with git cherry instead of merged blind: on the first four it showed 9, 7, 7 and 8 new commits, plus one commit that had already landed under a different hash.

## An agent saying done is a claim, not evidence

Anthropic's harness-design work states the problem plainly: agents reliably skew positive when grading their own work, and separating the agent doing the work from the agent judging it proves to be a strong lever. Tuning a standalone evaluator to be sceptical turned out far more tractable than making a generator critical of itself. The cost comparison makes it concrete. Given the same one-sentence prompt for a retro game maker, a solo Opus 4.5 run took 20 minutes and 9 dollars; the full planner, generator and evaluator harness took 6 hours and 200 dollars. The solo app looked finished.

*Every screen of the solo build rendered, and play mode did not work: the author could not play the level they had just made. Nothing in the code on its own shows that. Screenshot:*

The bias is measurable. Panickssery, Bowman and Feng showed at NeurIPS 2024 that LLM evaluators can recognise their own outputs, and that fine-tuning reveals a linear correlation between that self-recognition and the strength of their preference for their own text. Anthropic's long-running harness saw the practical form: Claude marked features complete after unit tests or curl calls without confirming they worked end to end, until it was told to test through browser automation as a person would. LangChain's most common failure was the same thing at a smaller scale.

My harness has three sensors that do not share the writer's context. The first judges a run by what reached origin/main, not by its exit code, because a pass that merged six commits and then hit its timeout still did its job. The second is a test-quality gate. The third is a review run as a separate invocation that sees only the diff. Its first real finding was a test the writer had added that pinned percentileNearestRank(samples, 0), locking in behaviour for an input the function's own JSDoc declares invalid, since the valid range is (0, 100]. The self-review in the writing session had passed it. In my content pipeline the writer runs on Sonnet and the editor on Opus, and a test asserts the two model IDs differ.

The gate compares against the merge-base because the first version compared against the tip of main. A branch cut days earlier is legitimately behind a main that has gained tests since, so the gate reported a shrink that was not real, and an agent shown false alarms learns that the gate can be ignored.

## Test the harness like the software it is

Claude Code's headless mode exits 0 on success and non-zero on failure, and a failure inside the run, such as missing authentication, is printed as the result on stdout. So any harness that reacts to how a run ended has to read output, and that classifier is code with bugs of its own. Mine is the opening story of this post. The block below shows the wrong pattern and the right one, plus the shell trap that made my harness's own self-test flaky.

Exit code 75 is EX_TEMPFAIL in sysexits, a temporary failure indicating something that is not really an error, to be retried later. systemd's SuccessExitStatus= lists extra exit codes to count as success, so a capped run does not show up as a failed unit or fire an alert. The pipe trap lives in Bash's definition of pipefail: the pipeline's status is that of the last command to exit non-zero, so a producer killed by SIGPIPE after grep -q has already matched turns a successful match into a failure, but only when the timing lines up. My check passed 20 times by hand and failed only when the watchdog ran it.

*Five of the six problems were in the harness rather than in the model's output. The review layer is the one place a model mistake surfaced, and a separate invocation caught it. The guide and sensor split behind the layers follows Böckeler.*

The fix for a gate that silently stops working is a test of the gate. A self-test script now runs 78 checks against the harness wiring every six hours from the watchdog, including the model pin, the effort flag and the lab driver's scenarios. Its first version reported three failures that turned out to be its own comments, the lines warning against those very patterns, so it now strips comments before matching. Runner changes are tested with a stub claude binary on the PATH, which proves the prompt is assembled correctly without spending session budget. The lab driver has ten sandboxed day scenarios, and the old driver fails all ten. A test the broken version also passes proves nothing.

> Do not auto-retry a run that hit a subscription cap. In my setup, a token from claude setup-token drew on the same session limit as my own interactive sessions, so a retry spends what is left of your budget on a doomed call. Write a cooldown marker with the parsed reset time, have every job check it first, and serialise jobs behind a lock.

## Pin the model, then plan to delete the harness

Claude Code's documentation says model aliases point to the recommended version for your provider and update over time, and that the way to pin is the full model name. My runs pin claude-sonnet-5-5 at medium effort, and that model's launch day taught me the CLI is part of the pin too. The server's Claude Code, version 2.1.277 on the stable channel, did not know the new ID yet. The call worked, stderr printed an unrecognized_model warning, and the CLI fell back to a 200K context and 32K output tokens. Version 2.1.284 ran the same ID at 1M and 128K. Headless runs had never auto-updated.

The longer-term point comes from Anthropic's harness-design post: every component in a harness encodes an assumption about what the model cannot do on its own, and those assumptions can quickly go stale as models improve. Its author dropped context resets once Opus 4.5 stopped showing context anxiety, removed the sprint construct for Opus 4.6, and found the evaluator had become unnecessary overhead for tasks inside the model's solo ability while still earning its cost at the edge. Rich Sutton's bitter lesson is the long version of the same warning: building in how we think we think does not work in the long run.

Solo agent, retro game maker

Opus 4.5

20 min

$9

Looked complete; the level could not be played

Full harness: planner, generator, evaluator, ten sprints

Opus 4.5

6 hr

$200

A 16-feature spec built out, playable with rough physics

Simplified harness without sprints, browser DAW

Opus 4.6

3 hr 50 min

$124.70

A different app; the builder ran coherently for over two hours unbroken

My own stale assumption was about scope, not capability. For ten days from 28 September, every run of the library job logged the same next step and landed nothing. Coverage had saturated at 98.86%, and the playbook forbade the only work left: major dev-dependency upgrades and anything touching a red CI. The model was willing; the harness said no. The playbook now hands upgrades to the agent, including coordinated major bumps as a single pull request, and makes the backlog the work when the menu runs dry. Major npm releases stay a human decision.

The audit I now run on any harness, in this order:

Write down, for each component, the assumption it encodes and the observation that would retire it.

Pin exact model IDs, update the CLI before adopting a new one, and probe the context window it actually grants.

Keep instructions short and scoped by location, and let the runner send facts, never a second copy of policy.

Put at least one sensor outside the writer's context: deterministic checks first, then a separately invoked reviewer, ideally on a different model.

Judge runs by what landed rather than by exit codes, and give the harness its own tests against a stub model.

The model gets the attention, but the harness decides what the model sees, what it is allowed to do, and whether anyone notices when it is wrong. The published results agree with my logs: hold the model fixed and the harness still moves the outcome, by as much as 13.7 points in the runs cited here. When an agent disappoints, debug the harness first. When a new model ships, check which parts of the harness it has made unnecessary.

## Sources

- Birgitta Böckeler, Harness engineering for coding agent users (martinfowler.com, 2026)
- Anthropic, Building effective agents (2024)
- OpenAI, Harness engineering: leveraging Codex in an agent-first world (2026)
- Yang et al., SWE-agent: agent-computer interfaces enable automated software engineering (arXiv 2405.15793)
- LangChain, Improving Deep Agents with harness engineering (2026)
- Terminal-Bench: the benchmark and its leaderboard
- Anthropic, Effective context engineering for AI agents (2025)
- Chroma, Context Rot: how increasing input tokens impacts LLM performance (2025)
- Liu et al., Lost in the Middle: how language models use long contexts (TACL)
- Manus, Context engineering for AI agents: lessons from building Manus (2025)
- Claude Code docs: how Claude remembers your project
- Anthropic, Effective harnesses for long-running agents (2025)
- Anthropic, Harness design for long-running application development (2026)
- Panickssery, Bowman and Feng, LLM evaluators recognize and favor their own generations (NeurIPS 2024)
- Claude Code docs: run Claude Code programmatically
- FreeBSD manual: sysexits(3)
- systemd manual: systemd.service and SuccessExitStatus=
- GNU Bash manual: pipelines and pipefail
- Claude Code docs: model configuration
- Rich Sutton, The Bitter Lesson (2019)

## Source links

- https://martinfowler.com/articles/harness-engineering.html
- https://www.anthropic.com/engineering/building-effective-agents
- https://openai.com/index/harness-engineering/
- https://arxiv.org/abs/2405.15793
- https://www.langchain.com/blog/improving-deep-agents-with-harness-engineering
- https://www.tbench.ai/
- https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents
- https://www.trychroma.com/research/context-rot
- https://arxiv.org/abs/2307.03172
- https://manus.im/blog/Context-Engineering-for-AI-Agents-Lessons-from-Building-Manus
- https://code.claude.com/docs/en/memory
- https://www.anthropic.com/engineering/effective-harnesses-for-long-running-agents
- https://www.anthropic.com/engineering/harness-design-long-running-apps
- https://arxiv.org/abs/2404.13076
- https://code.claude.com/docs/en/headless
- https://man.freebsd.org/cgi/man.cgi?query=sysexits&sektion=3
- https://www.freedesktop.org/software/systemd/man/latest/systemd.service.html
- https://www.gnu.org/software/bash/manual/html_node/Pipelines.html
- https://code.claude.com/docs/en/model-config
- http://www.incompleteideas.net/IncIdeas/BitterLesson.html

## FAQ

### What is an AI agent harness?

A harness is everything in an AI agent except the model: the instructions, the tools and their interface, how context is assembled, the state kept between sessions, the checks that verify work, and the runner that schedules it. Thoughtworks' Birgitta Böckeler sums it up as Agent = Model + Harness. Coding agents such as Claude Code ship with a built-in harness, and teams add an outer one with their own instruction files, tests, hooks and review.

### Can changing only the harness improve an AI agent's results?

Yes, by a measurable margin. LangChain moved deepagents-cli from 52.8% to 66.5% on Terminal-Bench 2.0 with gpt-5.2-codex held fixed, and the SWE-agent paper showed that removing the edit command cut SWE-bench Lite results from 18.0% to 10.3% on the same model. When an agent underperforms, change one harness component and rerun the same tasks before switching models.

### Why shouldn't an AI agent grade its own work?

Models skew positive about their own output. Anthropic found agents confidently praise their own work, and Panickssery, Bowman and Feng showed at NeurIPS 2024 that LLM evaluators recognise and favour their own generations. Use deterministic checks such as tests and test-count gates first, then a reviewer that runs in a separate context, ideally on a different model.

### How long should a CLAUDE.md or AGENTS.md file be?

Claude Code's documentation targets under 200 lines per CLAUDE.md, and OpenAI's Codex team keeps AGENTS.md at roughly 100 lines that work as a table of contents into a docs folder. Rules that must always hold belong in linters, tests or hooks rather than prose. Remember that Claude Code also loads every CLAUDE.md in the directories above the working directory, so a file in a parent folder applies to every project beneath it.

### How do I run Claude Code unattended without losing work?

Run claude -p from a scheduler as an unprivileged user, pin the exact model ID, log stream-json output, and judge each run by what reached the main branch rather than by its exit code. Use gh pr merge --auto so pull requests merge after the run ends, treat a session-cap hit as a temporary failure with exit code 75 instead of retrying it, and test runner changes against a stub claude binary.

---

- HTML version: https://www.matthewswong.com/en/blog/ai-agent-harness-engineering-research/
- Bahasa Indonesia: https://www.matthewswong.com/id/blog/ai-agent-harness-engineering-research/
- Site index for agents: https://www.matthewswong.com/llms.txt
- Sitemap: https://www.matthewswong.com/sitemap.xml
- API description (OpenAPI): https://www.matthewswong.com/openapi.json