Introduction

图 0-1｜AI Harness 六层能力模型与依赖结构

数据来源：基于本文分析绘制的示意图。

1. Background and Problem Statement

1.1. Model capability is rising, but engineering confidence is not keeping pace

Since 2023, the capability metrics of foundation models have continued to climb, with scores on various coding and reasoning benchmarks being refreshed almost every quarter. In stark contrast to the capability curve, however, is the stagnation or even decline of trust on the engineering side.

The Stack Overflow 2025 Developer Survey (published 2025-07-29; 49,000+ responses, covering 177 countries, 62 questions, 314 technologies; the 15th edition; A-level source) provides several key figures:

Meanwhile, the most striking finding in that survey is this: 66% of respondents cited "an AI solution that is almost right but not quite" as their biggest frustration, and 45.2% said debugging AI-generated code takes longer than writing it themselves.

These figures paint a clear contradiction: adoption has neared saturation, but trust has not kept pace. Enterprises have widely adopted AI coding and agent tools at scale, yet engineering predictability has not been established in tandem.

1.2. The bottleneck has moved from inside the model to outside it

Two leading labs reached the same judgment almost simultaneously in early 2026.

OpenAI writes in Harness engineering: leveraging Codex in an agent-first world (2026-02-11, A level):

"Our most difficult challenges now center on designing environments, feedback loops, and control systems."

Anthropic frames the same judgment in Harness design for long-running application development (2026, A level):

"Harness design is key to performance at the frontier of agentic coding."

The often-quoted remark given by Google DeepMind's Philipp Schmid on 2025-06-30 serves as the most concise expression of this judgment:

"Most agent failures are not model failures anymore, they are context failures."

All three statements point to the same conclusion: once the model is strong enough, what limits a system's performance is no longer the model's own intelligence but the engineering environment in which the model is placed. That environment is called an AI Harness.

1.3. A repeatedly observed phenomenon: the same model, different scores

The phenomenon that best illustrates the value of a Harness is the "same model, different scores" result that recurs in benchmarks.

Terminal-Bench (Stanford + Laude Institute, released 2025; A-level project, C-level data) states this explicitly in its methodology:

"Terminal-Bench does not score a model in isolation. Every result is a model plus an agent harness... the leaderboard is a ranking of systems, not of models."

In other words, what the leaderboard ranks are systems, not models. Below are several publicly available comparison figures, with a source grade of B/C; all of them must be marked [to be verified] before being used for decisions:

All of the above figures are marked [to be verified]. But the direction they jointly point to is clear and confirmed by A-level sources: differences in Harness design are enough to produce performance differences of several or even a dozen-plus percentage points, an order of magnitude that often exceeds the difference from switching models.

2. Definition of AI Harness

2.1. Unified definition

This project consistently uses the following definition throughout, and it must not be rewritten:

AI Harness (artificial intelligence harness layer / agent runtime framework): a complete engineering carrier layer that sits between the foundation model and the real business system. It is not responsible for improving the model's own intelligence, but rather for converting the model's uncertainty into engineering predictability.

The term "harness" is borrowed from the software-engineering Test Harness: a test harness does not produce functionality, but it makes functionality verifiable automatically, repeatably, and decisively. The same holds for an AI Harness — it makes model capabilities orchestratable, observable, constrainable, and regression-verifiable.

2.2. The four qualifying conditions of the definition

To avoid ambiguity in the documents that follow, the definition comprises four qualifying conditions that must hold simultaneously:

2.3. Four official and industry formulations of "Harness"

In addition to this project's unified definition, the following four official formulations offer multi-angle corroboration of the definition:

Formulation 1 · Anthropic (product definition)

In Effective harnesses for long-running agents (2025, A level), Anthropic directly defines the Claude Agent SDK as a harness:

"The Claude Agent SDK is a powerful, general-purpose agent harness adept at coding, as well as other tasks that require the model to use tools to gather context, plan, and execute."

This is direct evidence that the word "Harness" has been written into a leading vendor's product definition.

Formulation 2 · OpenAI (three pillars)

In Harness engineering (2026-02-11, A level), OpenAI gives the core philosophy "Human Steer, Agent Execute" and breaks Harness engineering into three pillars:

Formulation 3 · martinfowler.com (methodological framing)

In Harness engineering for coding agent users (2026, A level), Birgitta Böckeler breaks the Harness into three dimensions:

The same article also contains a line that deserves to be quoted repeatedly:

"A good harness should not necessarily aim to fully eliminate human input, but to direct it to where our input is most important."

Formulation 4 · martinfowler.com (human-in-the-loop vs. human-on-the-loop)

Humans and Agents in Software Engineering Loops (2026, A level) proposes the distinction between "in the loop" (a human edits the artifacts) and "on the loop" (a human edits the harness that produces the artifacts), and gives this definition:

"The collection of specifications, quality checks, and workflow guidance that control different levels of loops inside the how loop is the agent's harness."

The value of this formulation is that it makes clear that a Harness is a middle layer that can be continuously improved, not a one-off written prompt.

3. Etymology: from Test Harness to Agent Harness

3.1. The Test Harness in software engineering

In the software-engineering tradition, a Test Harness (test fixture / test rig) refers to an execution environment external to the software under test. Its responsibilities are:

1. Prepare the inputs and dependencies needed for testing (stubs, mocks, fixtures);
2. Drive the component under test to run;
3. Collect the output results;
4. Compare the results against the expected criteria and produce a pass/fail determination.

Its key characteristic is this: it does not execute the business logic of the component under test itself, but it makes that logic verifiable automatically, repeatably, and decisively.

3.2. Homology: why this analogy holds

Mapping the structure of a Test Harness onto the AI scenario yields an almost one-to-one correspondence:

The homology between an Agent Harness and a Test Harness lies in the fact that it exists outside the agent's reasoning logic, does not intervene in the model's internal decision process, yet ensures that the agent's actions are executed, tracked, and constrained within a controlled environment.

One point worth noting: this research was unable to directly capture the original text of the Wikipedia "Test harness" entry; the description above is based on the common software-engineering definition and cross-corroborated through multiple secondary sources, with the specific wording marked [to be verified].

3.3. Horse tack: the original meaning of the word "harness"

The original meaning of the word "harness" is horse tack (from twelfth-century Old French harneis, originally meaning "a full set of battle gear / armor"). This analogy is equally precise:

• The horse decides where to run — corresponding to the model deciding what to output;
• The tack does not take part in the decision — corresponding to a Harness not intervening in the model's internal reasoning;
• The tack governs how force is transmitted, constrains overreach, and prevents derailment — corresponding to a Harness governing context, permissions, and side effects.

The Test Harness emphasizes verifiability, while horse tack emphasizes controllability. Taken together, the two exactly cover the two core values of an AI Harness.

3.4. The point in time when the term was established

As a term in the AI field, "Harness" went through three stages on its way to mainstream acceptance:

Hashimoto's description of the fifth stage (paraphrased) is:

"Every time you find an Agent making a mistake, spend the time to engineer a solution so that it never makes the same mistake again."

This statement reveals the essence of Harness engineering: not making the model stronger, but turning every failure into a mechanism that will not recur.

It must be stated honestly: this research could not confirm the original coiner of the term "Agent Harness" nor the first-hand source of its first appearance. The frequently cited LangChain article The Anatomy of an Agent Harness could not be retrieved for its original text or exact publication date in this research. See Section 8, the information-gap statement, for details.

4. The Six-Layer Capability Model

4.1. Model overview

This project consistently uses the following six-layer capability model throughout. It is the common coordinate system for all documents that follow.

4.2. L1 Context Engineering

The question L1 answers is: before this step of reasoning, which tokens should the model see.

Anthropic's official definition, given in Effective context engineering for AI agents (2025, A level), is: the full set of strategies for curating and maintaining the optimal set of tokens (information) during LLM inference.

The two most critical official concepts in this layer are:

1. context rot: officially characterized as a "performance gradient, not a hard cliff" — as context grows, performance degrades gradually rather than collapsing suddenly at some length. This means context governance must be maintained continuously through engineering, and cannot rely on "setting a single threshold."
2. attention budget: because of the n² coupling property of self-attention, every additional token in the context dilutes the model's attention toward other tokens. Context is therefore a scarce resource, not a free container.

The definition given by Andrej Karpathy on 2025-06-25 is the most widely cited:

"Context engineering is the delicate art and science of filling the context window with just the right information for the next step."

He also pointed out that context engineering is only a small part of "an emerging thick layer of non-trivial software that coordinates individual LLM calls into full LLM apps" — a sentence that is the best annotation for the question "why do we need a six-layer Harness rather than context engineering alone."

4.3. L2 Tooling & Execution

The question L2 answers is: what actions can the model exert on the environment, and where those actions are executed.

This layer contains three categories of components:

One counterintuitive finding in this layer deserves emphasis: the number of tools is not positively correlated with performance. After Vercel cut its tool count from 15 to 2, accuracy rose from 80% to 100%, token consumption fell by 37%, and speed improved 3.5× (C-level source, marked [to be verified]). Anthropic's official documentation likewise recommends that tools be "self-contained, robust to errors, and extremely clear in purpose," avoiding feature overload and ambiguous decision points (A level).

4.4. L3 Orchestration & Control

The question L3 answers is: in what order to do things, who does what, and how to recover after interruption.

The main orchestration forms include:

Anthropic's official guidance on sub-agents is that sub-agents may use tens of thousands of tokens to explore deeply, but only return a condensed summary of 1,000–2,000 tokens (A level). This is a typical architectural trade-off of "trading context for parallelism."

4.5. L4 Memory & State

The question L4 answers is: how state is handed over across sessions and across context windows.

In Effective harnesses for long-running agents (A level), Anthropic frames this difficulty as:

Imagine a software project staffed by engineers on shifts, where each new engineer who comes on duty has no memory of what happened in the previous shift.

Its official solution is the claude-progress.txt progress file plus git history, and it stipulates that a Coding Agent must take three steps at the start of every session:

1. Run pwd to confirm the current working directory;
2. Read the git log and the progress file;
3. Read the feature list and select the highest-priority uncompleted item.

It also stipulates a constraint at the end: the environment must be left in a clean, mergeable state.

This layer also requires distinguishing between two context-handling approaches (A level, from Anthropic's three-agent architecture article):

4.6. L5 Evaluation & Observability

The question L5 answers is: how well it did, how that is judged, and how regression is ensured.

The most important design principle in this layer comes from Anthropic's Harness design for long-running apps (A level): the evaluator and the evaluated must be separated. Anthropic observed that when an agent evaluates its own work it will "confidently praise" it — even when the quality is plainly mediocre in the eyes of a human observer. Their exact words are:

Teaching an independent Evaluator to be skeptical is far easier than getting a Generator to criticize itself.

This design is inspired by generative adversarial networks (GANs): among the three roles of Planner / Generator / Evaluator, the Evaluator sets hard thresholds for each of four dimensions — product depth, functionality, visual design, and code quality — and if any dimension falls below the threshold, that iteration fails and the detailed reasons for the failure are fed back to the Generator.

4.7. L6 Governance & Safety

The question L6 answers is: what cannot be done, how violations are discovered, and what the spend ceiling is.

This layer contains four categories of components: the permission model (RBAC), guardrail models, audit logs, and budget guardrails (Budget Guardrail).

A common misconception that must be cleared up is that governance slows things down. Anthropic's sandbox evidence points to the opposite conclusion — its internal usage found that sandboxing safely reduced permission prompts by 84% (A level). Constraint is not the opposite of autonomy; it is the precondition for autonomy.

4.8. Dependencies and call relationships between the six layers

图 4-1｜六层之间的依赖与调用关系

The six layers are not stacked in parallel; rather, there is a clear direction of dependency:

Reading notes:

• L1 → L2 → L3 is the main execution chain: first decide what to see, then decide what can be done, and finally decide in what order to do things.
• L4 provides cross-session state support for L1/L3, and is a cross-cutting support layer along the time dimension.
• L5 observes the entire chain and produces feedback, which ultimately returns to L1 (changing the composition of the next context).
• L6 is a cross-cutting slice that imposes constraints on all of L1–L5 and records the audit.

5. Boundaries with Adjacent Concepts

5.1. Boundary overview table

5.2. The boundary with Agents

An Agent is the execution unit, and the Harness is the chassis. Their relationship is like "car and road": the same car performs differently on different roads, and the same road can carry different cars.

The criterion: if replacing it does not change "how the model's output distribution is constrained and verified," it belongs to the Agent; if replacing it changes "how the model is orchestrated, observed, constrained, and regression-verified," it belongs to the Harness.

China's official definition of "agent" can serve as a reference point for the Chinese context (from the interpretation of the national standard series Artificial Intelligence — Agent Interconnection by the China Information Economics Network, 2026, A level):

"An agent is a new type of intelligent system with autonomous perception, memory, decision-making, interaction, and execution capabilities, and has become a key carrier of AI products and services."

5.3. The boundary with Agent Frameworks

Agent Frameworks (such as LangChain / LangGraph) primarily cover L2 and L3 — that is, tool invocation and orchestration. They are a subset of the Harness.

The hierarchical relationship among the three is:

The key difference is that a Framework is neutral, while a Harness takes a stand. A Harness will explicitly tell you how context should be organized, how tools should be trimmed, and what counts as "done," whereas a Framework only provides capabilities, not opinions.

5.4. The boundary with Agent Platforms

An Agent Platform is the Harness's productized packaging, layering on top of the Harness:

• A visual orchestration UI;
• Multi-tenant and permission systems;
• Billing and quotas;
• Publishing and operations capabilities for non-technical users.

The criterion: the portion of the runtime that remains after stripping away the UI, tenants, and billing is the Harness.

5.5. The boundary with AI IDEs

An AI IDE is the Harness's vertically integrated form for the coding scenario. On top of a general Harness it layers coding-specific capabilities such as an editor, terminal, diff view, and LSP integration.

Cursor, Claude Code, and Codex CLI can all be classified here, though among them Claude Code and Codex CLI are closer to "terminal-form Harnesses," while Cursor is closer to an "IDE-form Harness."

5.6. The boundary with Models

A Model is the object being carried by the Harness; it does not itself belong to the Harness.

This boundary saw notable blurring in 2025–2026: Anthropic officially noted that products such as Claude Code / Codex have been trained with the model and the Harness together in the post-training stage, producing the side effect of "the model overfitting to a specific Harness structure" — changing the tool logic actually made the model perform worse (A level).

The practical implication of this phenomenon is: the best Harness for your task is not necessarily the one the model was post-trained with.

5.7. The boundary with MLOps / LLMOps

The "shift feedback left," mutation testing, structural testing, and similar terms mentioned in martinfowler.com's two articles represent software-engineering means being reused in the Harness, which is the direct dividing line from MLOps methodology (A level).

5.8. An easy-to-remember analogy: the four-layer computer stack

A set of analogies circulating in the industry (retold from a C-level source; the analogy itself may be cited, but the specific figures and cases within it are marked [to be verified]):

The value of this analogy is: just as an operating system does not produce compute power but determines how that compute power is used, a Harness does not produce intelligence but determines how intelligence is delivered safely and predictably.

6. The Value Proposition of AI Harness

6.1. Turning uncertainty into predictability

A model's output is probabilistic by nature. A Harness does not change this nature, but through context constraints, tool boundaries, evaluation criteria, and regression sets it narrows the probability distribution to an engineering-acceptable range.

Anthropic's sandbox evidence provides a quantitative anchor point: permission prompts were reduced by 84% while safety actually improved (A level). This shows that gains in predictability do not come at the cost of capability.

6.2. Making governance and autonomy a positive-sum game

Conventional wisdom holds that permission controls reduce an agent's autonomy. But three lines of evidence point to the opposite conclusion:

Conclusion: constraint is capability.

6.3. Making capability regression-verifiable

The third value of a Harness is turning "model capability" into an engineering object that can be regression-verified.

SWE-bench (released 2023-10-10; 2,294 task instances across 12 open-source Python repositories; ICLR 2024 Oral; A level) established the principle of programmatic scoring — there is no partial credit for "looks good but fails the tests." Terminal-Bench's principle is likewise "either the automated tests pass or they do not."

Anthropic also offers a concrete recommendation: equip the agent with browser-automation tools (such as Playwright MCP) so it can verify its own artifacts like a human would, which significantly improved performance (A level).

7. Summary

An AI Harness is not a newly invented technical component, but rather a naming and synthesis of existing engineering practice. Its emergence had three necessary conditions, all satisfied simultaneously between late 2025 and early 2026:

1. The model is strong enough: strong enough that failures no longer stem mainly from insufficient model intelligence, but from poorly designed environments;
2. Tools have a standard: MCP turned the tool ecosystem from an N×M integration problem into N+M;
3. Governance has a template: sandbox, permission, and audit practices were proven at scale and quantified by leading vendors.

Its core thesis can be compressed into a single sentence: given model capability, the variance in system performance is determined mainly by the Harness.

The four documents that follow will unfold this in turn: how it evolved step by step from prompt engineering (02–Development History), the layered details of contemporary mainstream implementations (03–Architecture Evolution), the next directions in technology and standardization (04–Future Development), and the judgment that all these facts jointly point to (05–Summary).

8. Information-Gap Statement

This document involves the following information gaps, which are marked at their respective points in the text and summarized here:

1. ISO/IEC-level international standards for agent interconnection: this research could not find the number of an ISO/IEC international standard for agent interconnection that has been published or formally initiated, so no authoritative information is available.
2. Origin of the term "Agent Harness" and the source of its first appearance: no definitive first-hand origin document was found, so no authoritative information is available. What can be confirmed is that Anthropic was already using "harness" to describe the Claude Agent SDK in 2025, Mitchell Hashimoto named "Harness Engineering" on 2026-02-05, and OpenAI brought it into the mainstream on 2026-02-11. Earlier tracing (such as the LangChain blog post The Anatomy of an Agent Harness) could not retrieve the original text or an exact publication date.
3. Original text of the Wikipedia "Test harness" entry: not directly captured, only retold through secondary sources; the specific wording in Section 3.1 is marked [to be verified].
4. Current data on the official Terminal-Bench leaderboard: the official site (tbench.ai) was not directly crawled; all leaderboard figures in Section 1.3 are third-party retellings and are marked [to be verified].
5. All comparison data in Section 1.3: including Factory Droid 77.3%, OpenAI Simple Codex 75.1%, KRAFTON 74.7%, Bigai 71.9%, LangChain 52.8% → 66.5%, and Vercel 80% → 100%, all from B/C-level sources and marked [to be verified].
6. The 1/200 and 99% figures for Codex Auto-review: from B-level sources, marked [to be verified].
7. Original source of the four-layer computer-stack analogy: the analogy itself has been retold by many parties and may be cited, but its original source and the accompanying specific figures and cases are marked [to be verified].

9. References

1. Effective context engineering for AI agents — Anthropic, 2025. https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents
2. Effective harnesses for long-running agents — Anthropic, 2025. https://www.anthropic.com/engineering/effective-harnesses-for-long-running-agents
3. Harness design for long-running application development — Anthropic, 2026. https://www.anthropic.com/engineering/harness-design-long-running-apps
4. Harness engineering: leveraging Codex in an agent-first world — OpenAI, 2026-02-11. https://openai.com/index/harness-engineering/
5. Harness engineering for coding agent users — Birgitta Böckeler, martinfowler.com, 2026. https://martinfowler.com/articles/harness-engineering.html
6. Humans and Agents in Software Engineering Loops — martinfowler.com, 2026. https://martinfowler.com/articles/exploring-gen-ai/humans-and-agents.html
7. My AI Adoption Journey — Mitchell Hashimoto, 2026-02-05. https://mitchellh.com/writing/my-ai-adoption-journey
8. 2025 Stack Overflow Developer Survey — Stack Overflow, 2025-07-29. https://survey.stackoverflow.co/2025/
9. Equipping agents for the real world with Agent Skills — Anthropic, 2025-10-16. https://www.anthropic.com/engineering/equipping-agents-for-the-real-world-with-agent-skills
10. SWE-bench: Can Language Models Resolve Real-World GitHub Issues? — Jimenez, Yang, et al. (Princeton), 2023-10 (ICLR 2024 Oral). https://www.swebench.com/; arXiv:2310.06770
11. Terminal-Bench — Stanford / Laude Institute, 2025. https://www.tbench.ai/
12. Interpretations of the national standard series "Artificial Intelligence — Agent Interconnection" — China Information Economics Network, 2026. https://cinic.org.cn/xw/zcdt/1643418.html
13. Agent Harness: The Core Paradigm of AI Engineering in 2026 — Tencent Cloud Developer Community, 2026. https://developer.cloud.tencent.com/article/2698416
14. Terminal-Bench: The CLI Autonomy Standard — AgentMarketCap, 2026-04-09. https://agentmarketcap.ai/blog/2026/04/09/terminal-bench-cli-autonomy-standard-coding-agents