Definitions

1. The Formal Definition of AI Harness

1.1. Unified Definition

This whitepaper in its entirety (as well as all 176 research documents in this project) uniformly adopts the following definition, without any rewording:

**AI Harness (AI Stewardship Layer / Agent Runtime Framework)**: an engineering carrier layer located between the Foundation Model and real business systems. It is not responsible for improving the model's own intelligence, but rather for converting the model's **uncertainty** into engineering **predictability**.

The word "Harness" is borrowed from software engineering's *Test Harness*: a test harness does not produce functionality, but it makes functionality verifiable automatically, repeatedly, and deterministically. AI Harness is analogous — it makes model capabilities orchestratable, observable, constrainable, and regression-verifiable.

Four key words in the definition deserve emphasis one by one: **position** (between the model and business systems), **responsibility** (carrying rather than enhancing intelligence), **goal** (predictability rather than capability ceiling), and **means** (orchestration, observation, constraint, and regression verification).

1.2. The Four Qualification Conditions of the Definition

To avoid ambiguity, the definition includes four qualification conditions that must all be satisfied simultaneously:

The four qualification conditions together draw a clear boundary line: Harness is an **engineering layer**, not a model layer, and not a business layer. Any problem that tries to be solved with a "stronger model" or "smarter prompts" lies outside Harness's scope; any problem where "the same model performs differently in a different environment" lies inside it.

1.3. Corroboration of the Definition from Multiple Angles

In addition to the unified definition, the following four first-hand statements from vendors and the methodology community can corroborate this definition from different angles:

**Statement one · Anthropic (product definition)**. In *Effective harnesses for long-running agents* (2025, A-level), Anthropic directly defines the Claude Agent SDK as a harness: "The Claude Agent SDK is a powerful, general-purpose agent harness adept at coding, as well as other tasks that require the model to use tools to gather context, plan, and execute." This is direct evidence that the word "Harness" has been written into a product definition by a leading vendor.

**Statement two · OpenAI (three pillars)**. In *Harness engineering* (2026-02-11, A-level), OpenAI presents the core philosophy "Human Steer, Agent Execute" and splits Harness engineering into three pillars:

**Statement three · martinfowler.com (methodological definition)**. Birgitta Böckeler splits Harness into three dimensions: guides (guidance, feedforward control — telling the agent what to do before it acts), sensors (sensing, feedback control — checking after it acts whether the result is satisfactory), and behavioural harness (behavioral constraints — organizing the first two into an executable system of constraints). One sentence in the same article deserves to be quoted again and again: "A good harness should not necessarily aim to fully eliminate human input, but to direct it to where our input is most important."

**Statement four · martinfowler.com (middle-layer positioning)**. *Humans and Agents in Software Engineering Loops* (2026, A-level) draws the distinction between "in the loop" (humans modifying artifacts) and "on the loop" (the harness that modifies the artifacts being produced) and defines: "The collection of specifications, quality checks, and workflow guidance that control different levels of loops inside the how loop is the agent's harness." This statement makes clear that Harness is a **middle layer that can be continuously improved**, not a one-off prompt written once.

2. Etymology and Analogies

2.1. Test Harness in Software Engineering

In the software engineering tradition, a *Test Harness* (test fixture / test scaffolding) refers to **an execution environment external to the software under test**. Its responsibilities are:

1. Prepare the inputs and dependencies required for testing (stubs, mocks, fixtures);
2. Drive the component under test to run;
3. Collect the output results;
4. Compare the results against the expected criteria and produce a pass or fail verdict.

Its key characteristic is: **it does not execute the business logic of the component under test itself, but it makes that logic verifiable automatically, repeatedly, and deterministically**. It should be noted that this project was unable to directly capture the original text of the relevant encyclopedia entries; the above description is based on the prevailing definition in software engineering and has been cross-corroborated through multiple secondary sources, and its specific wording is marked `[to be verified]`.

2.2. Isomorphism: Why This Analogy Holds

Mapping the structure of a Test Harness onto the AI scenario produces an almost one-to-one correspondence:

The isomorphism between Agent Harness and Test Harness lies in this: it **exists outside the agent's reasoning logic** and **does not intervene in the model's internal decision-making process**, yet it ensures that the agent's actions are executed, tracked, and constrained in a controlled environment. A test fixture does not produce functionality, but it makes functionality verifiable; AI Harness does not produce intelligence, but it makes intelligence deliverable.

2.3. Horse Harness: The Original Meaning of the Word

The original meaning of the word "harness" is **horse harness** (derived from the twelfth-century Old French *harneis*, whose original meaning was "full set of battle gear / armor"). This analogy is equally precise:

• The horse decides where to run — corresponding to the model deciding what to output;
• The harness does not participate in decision-making — corresponding to Harness not intervening in the model's internal reasoning;
• The harness governs how force is transmitted, constrains going out of bounds, and prevents derailment — corresponding to Harness governing context, permissions, and side effects.

Test Harness emphasizes **verifiability**, while the horse harness emphasizes **controllability**. Together, the two precisely cover the two core values of AI Harness.

2.4. Timeline of How the Name Was Established

As a term accepted by the mainstream in the AI field, "Harness" went through three stages (all from A-level sources):

Hashimoto's description of the fifth stage (paraphrased) is: "Every time you find the Agent making a mistake, take the time to engineer a solution so that it never makes the same mistake again." This sentence reveals the essence of Harness engineering: **not making the model stronger, but solidifying every failure into a mechanism that will not recur**.

To be honest: this project was unable to confirm the originator of the term "Agent Harness" or the first-hand source where it first appeared. The frequently mentioned LangChain article *The Anatomy of an Agent Harness* could not be located for its original text or exact publication date. This whitepaper makes no claim of attribution for originating the term.

3. Why Harness Needs to Stand as Its Own Layer

A common objection is: context management, tool calling, and evaluation — these things are already scattered across frameworks, platforms, and best practices, so why invent a new concept and a new layer? This chapter answers with three mutually independent arguments.

3.1. Argument One: The Evaluation Variable Argument

**Argument form**: if a factor is proven in controlled evaluation to systematically change results, it is not an implementation detail but an independent variable; an independent variable requires an independent engineering discipline.

**Factual basis**:

1. Terminal-Bench (Stanford + Laude Institute) methodology explicitly states: "Every result is a combination of a model plus an agent framework... leaderboards rank systems, not models" (A-level). The evaluation community has already written Harness into the methodology as a variable.
2. In SWE-bench history, Claude 3.5 Sonnet paired with an agentic harness reached 49.0% in the second half of 2024 — **the first time Harness was written into the scorecard of a public benchmark** (C-level, `[to be verified]`).
3. Multiple controlled cases of the same model with different scores (see 01-Introduction, section 3.3; all numbers `[to be verified]`): merely changing the agent framework can produce a difference of more than 2 percentage points; merely restructuring the Harness can produce a ranking jump of more than 13 percentage points.
4. The post-training coupling of the model and Harness has been officially confirmed: Anthropic points out that leading coding products already train the model together with a specific Harness structure during post-training, and changing the tool logic actually makes the model perform worse (A-level).

**Conclusion (judgment)**: Harness is already an explicit variable in evaluation and a competitive variable in industry. A factor that can systematically determine results must be independently named, independently designed, and independently governed — this is precisely what "standing as its own layer" means.

3.2. Argument Two: The Industry Differentiation Argument

**Argument form**: if eight vastly different industries, when facing the same capability model, exhibit **systematic, explainable** differences in emphasis, then that model is a real, existing common structure rather than a product brochure from some vendor.

**Factual basis**: the research of this project's eight industry groups (see 04-Practice for details) yielded the following distribution of bottleneck layers:

**Conclusion (judgment)**: eight industries share the same six-layer skeleton, but their weight distributions are entirely different — this is not a coincidence but direct evidence that "Harness is an independent layer of engineering capability": precisely because it is an independent layer, it can be re-weighted by different industries according to their own risk structures and criteria shapes. If Harness were merely a function of the model or an appendage of a framework, this cross-industry structural difference would be inexplicable.

3.3. Argument Three: The Failure Mode Argument

**Argument form**: if the responsibilities of Harness are split apart to be borne separately by the model, the framework, and the business code, each piece will fail in a predictable way; these failure modes are precisely the objects that the six-layer model is designed to guard against.

The practice of the second generation (roughly 2023–2025) provides a complete record of counterexamples (see 01-Overview / Chapter 4 of 02-History):

**Conclusion (judgment)**: the six types of failure modes span six different stages, and any single-point solution (longer context, a better framework, a stronger model) can cover only one or two of them. They require a layer of **unified engineering infrastructure with a clear stance** to guard against them as a whole — and this layer of infrastructure is Harness. Unlike a "neutral" framework, Harness will explicitly tell you how context should be organized, how tools should be pruned, and what counts as "done".

3.4. Common Conclusion of the Three Arguments

The three arguments are mutually independent, yet their conclusions agree:

1. The evaluation variable argument shows that Harness **must be measured as a variable**;
2. The industry differentiation argument shows that Harness **can be independently weighted as a layer**;
3. The failure mode argument shows that Harness **must be uniformly built as a layer**.

Taken together, the three support the central thesis of this whitepaper: **given a fixed model capability, the variance in system performance is determined mainly by Harness**.

4. Layer-by-Layer Definition of the Six-Layer Capability Model

图 4-1｜AI Harness 六层能力模型与层间依赖关系

数据来源：基于本文分析绘制的示意图。

4.1. Model Overview

This whitepaper in its entirety uniformly uses the following six-layer capability model; it serves as the common frame of reference for all subsequent chapters:

The following sections expand each layer one by one following a unified structure: responsibility, key components, representative implementations, and typical failure modes.

4.2. L1 Context Engineering Layer

**Responsibility**: before each step of reasoning, decide which tokens the model should see. Anthropic's official definition: a set of strategies for curating and maintaining the optimal set of tokens (information) during LLM inference (A-level).

**Two key official concepts**:

1. **context rot**: as the context grows longer, performance degrades gradually (a performance gradient) rather than collapsing abruptly at some length — therefore context governance must be carried out continuously through engineering means and cannot rely on "pinning a threshold".
2. **attention budget**: due to the relational nature of self-attention, every additional token in the context dilutes the model's attention to other tokens. Context is a **scarce resource**, not a free container.

**Key components and representative implementations**: retrieval (RAG, Contextual Retrieval, GraphRAG), compression (Context Compaction), caching (KV Cache reuse), assembly (Prompt Assembly), on-demand loading (the three-level progressive disclosure of Agent Skills).

**Typical failure modes**: context bloat pushes critical instructions out; retrieval returns deleted old interfaces; stuffing the entire knowledge base into the prompt yields negative returns (multi-document Q&A exhibits a U-shaped positional effect, and performance can even fall below closed-book when the relevant information is buried in the middle).

4.3. L2 Tooling and Execution Layer

**Responsibility**: decide which actions the model can exert on the environment, and where those actions are executed.

**Key components and representative implementations**:

**A counterintuitive piece of evidence**: tool count and performance are not positively correlated. After Vercel cut its tools from 15 to 2, accuracy rose from 80% to 100%, token consumption dropped 37%, and speed increased 3.5× (C-level, `[to be verified]`). Anthropic officially also recommends that tools should be "self-contained, robust to errors, and extremely clear in purpose" (A-level).

**Typical failure modes**: sandbox escape; tools returning enormous output that pollutes the context; concurrent write conflicts; destructive actions without a human gate.

4.4. L3 Orchestration and Control Layer

**Responsibility**: decide in what order to do things, who does them, and how to recover after an interruption.

**Key orchestration forms**:

Anthropic's official sub-agent parameters: a sub-agent can explore deeply using tens of thousands of tokens but only returns a condensed summary of 1,000–2,000 tokens (A-level) — a typical architectural trade-off of "trading context for parallelism".

**Typical failure modes**: loops that fail to converge; sub-agent contexts cross-contaminating each other; state lost after interruption, and side effects re-executed after recovery.

4.5. L4 Memory and State Layer

**Responsibility**: how state is handed over across sessions and across context windows. Anthropic's analogy: like a software project where engineers work in shifts, each new engineer arrives with no memory of what happened in the previous shift (A-level). The official solution is a progress file plus version-control history, with the agent required to do three things at the start of every session: confirm the working directory, read the history and progress, and read the task list to select the highest-priority item; at the end, it must leave the environment in a **clean, mergeable state**.

**The difference between the two context handover mechanisms** (A-level):

**Typical failure modes**: checkpoints do not include tool side effects; replay after recovery causes duplicate writes; erroneous information enters long-term memory and keeps propagating (context poisoning).

4.6. L5 Evaluation and Observability Layer

**Responsibility**: decide how well it was done, how to judge, and how to regress.

**The most important design principle** (Anthropic's *Harness design for long-running apps*, A-level): **the evaluator and the evaluated must be separated**. When an agent is asked to evaluate its own work, it will "confidently praise" it — even when the quality is clearly mediocre. The official conclusion is that tuning an independent Evaluator to be skeptical is far easier than having the Generator critique itself.

**Key components and representative implementations**: trajectory tracing (Trace), evaluation sets (Eval Set), golden datasets (Golden Dataset), regression sets and regression gating, online metrics (Online Metrics), and the principle of programmatic scoring (jointly established by SWE-bench and Terminal-Bench: "if the test passes, it passes", with no partial credit for "looks good but fails the test").

**Typical failure modes**: looking only at the final result without examining the process; regression sets repeatedly "optimized" into overfitting; evaluation-criteria drift (cross-leaderboard comparisons are largely invalid); evaluation cost so high that regression comes to a halt.

4.7. L6 Governance and Safety Layer

**Responsibility**: decide what cannot be done, how violations are discovered, and that spending is capped.

**Key components and representative implementations**: permission model (RBAC), guardrail model (Guardrails), audit logs, budget guardrail (Budget Guardrail), credential protection, and three-level configuration scope (project-level / user-level / organization-level).

**A piece of evidence that must be clarified**: the common misconception is that "governance slows down autonomy". Three groups of evidence point to the opposite conclusion (judgment: constraint is capability):

**Typical failure modes**: permissions granted too broadly at once; audit logs that omit input context and therefore cannot be reviewed; conflating observability (L5, serving optimization) with audit (L6, serving evidence) — the two differ in retention period, tamper-resistance requirements, and access principals.

4.8. Dependency and Call Relationships Among the Six Layers

图 4-2｜六层之间的依赖与调用关系

The six layers are not stacked side by side in parallel; rather, they have clear dependency directions:

        ┌─────────────────────────────────────────┐
        │  L6 治理与安全层（贯穿全层，横向切面）  │
        └─────────────────────────────────────────┘
                          ▲ 约束与审计
   L1 上下文 ──► L2 工具与执行 ──► L3 编排与控制
        ▲              │                  │
        │              ▼                  ▼
        └──────── L4 记忆与状态 ◄──────────┘
                          │
                          ▼
                  L5 评估与观测
                          │
                          └──► 反馈回 L1（改变下次看到什么）

Reading notes:

• **L1 → L2 → L3** is the main execution chain: first decide what to see, then decide what can be done, and finally decide in what order to do things.
• **L4** provides cross-session state support for L1 / L3 and is the horizontal support layer of the time dimension.
• **L5** observes the entire chain and produces feedback, which ultimately returns to L1.
• **L6** is a horizontal cross-cutting concern that imposes constraints on all of L1–L5 and records audit.

5. Boundaries with Adjacent Concepts

5.1. Summary Table of Boundaries

This whitepaper in its entirety uniformly uses the following boundaries:

5.2. Concept-by-Concept Clarification

**Boundary with Agent**. The Agent is the execution unit and Harness is the chassis, with a "car and road" relationship between the two. Criterion: if replacing it does not change "how the model's output distribution is constrained and verified", it belongs to the Agent; if replacing it changes "how the model is orchestrated, observed, constrained, and regression-verified", it belongs to Harness. The official Chinese definition of "agent" can serve as the Chinese-context benchmark (A-level): "An agent is a new type of intelligent system with autonomous perception, memory, decision-making, interaction, and execution capabilities, and has become a key carrier of AI products and services."

**Boundary with Agent Framework**. A Framework (such as LangChain / LangGraph) mainly covers L2 and L3 and is a subset of Harness. The key difference is that a Framework is **neutral** while Harness **takes a stance** — Harness explicitly tells you how context should be organized, how tools should be pruned, and what counts as "done", whereas a Framework only provides building blocks without offering opinions. This is precisely the blind spot of the second-generation "framework-as-solution" mindset: a Framework does not solve production problems.

**Boundary with Agent Platform**. A Platform is the productized packaging of Harness, layering on top of it visual orchestration UI, multi-tenancy and permission systems, billing and quotas, and release/operation capabilities aimed at non-technical users. Criterion: the runtime that remains after stripping away the UI, tenancy, and billing is Harness.

**Boundary with AI IDE**. An AI IDE is Harness's vertical integration form in the coding scenario, layering on coding-specific capabilities such as editors, terminals, diff views, and LSP integration. Among them, terminal-type products (Claude Code, Codex CLI) are closer to "Harness in terminal form", while editor-type products (Cursor) are closer to "Harness in IDE form".

**Boundary with Model**. The Model is the carried object and does not belong to Harness. In 2025–2026 this boundary saw noteworthy blurring: leading vendors have already trained the model together with the Harness during post-training, producing the side effect of "the model overfitting to a specific Harness structure" — changing the tool logic actually makes the model perform worse (A-level). The practical implication is: **the best Harness for your task is not necessarily the one the model was post-trained with**.

**Boundary with MLOps / LLMOps**.

5.3. A Memorable Analogy: The Computer Four-Layer Stack

A set of analogies circulates in the industry (the analogy itself is quotable as relayed by multiple parties; its original source and accompanying cases are marked `[to be verified]`):

Just as an operating system does not produce compute but determines how compute is used, Harness does not produce intelligence but determines how intelligence is delivered safely and predictably. Note: this analogy is used only to aid understanding and not as a basis for argument; "AI operating system" is not used as a formal term (see Chapter 6).

6. Common Misuses and Clarifications

7. Summary

1. **Definition**: AI Harness is an engineering carrier layer located between the foundation model and real business systems, converting the model's uncertainty into engineering predictability; it does not increase intelligence, but it makes intelligence orchestratable, observable, constrainable, and regression-verifiable.
2. **Etymology**: the verifiability of the Test Harness plus the controllability of the horse harness together make up the two faces of this word.
1. **The three-fold argument for standing as its own layer**: the evaluation variable argument (it is an explicit variable in evaluation), the industry differentiation argument (eight industries give systematically different weights to the same six-layer skeleton), and the failure mode argument (six types of cross-stage failures require unified prevention).
3. **The six-layer capability model**: L1 what to see, L2 what can be done, L3 in what order, L4 what to remember, L5 how well it was done, L6 what cannot be done — the six layers have dependency directions, L6 runs across horizontally, and L5 feeds feedback back to L1.
4. **Boundaries**: the Agent is the execution unit, the Framework is a subset, the Platform is a packaging, the AI IDE is a vertical form, the Model is the carried object, and MLOps focuses on a different lifecycle.
5. **Central thesis**: given a fixed model capability, the variance in system performance is determined mainly by Harness.

8. Statement of Information Gaps

1. **All comparative numbers for the same model with different scores** (77.3% vs. 75.1%, 52.8% vs. 66.5%, Vercel 80% vs. 100%, Claude 3.5 Sonnet + agentic harness 49.0%): from B/C-level sources, marked `[to be verified]`.
2. **The original text of the Wikipedia "Test harness" entry**: not directly captured; the wording in section 2.1 is based on the prevailing definition and cross-corroborated through secondary sources, marked `[to be verified]`.
3. **The original source of the term "Agent Harness"**: no first-hand originating literature was found; the original text and publication date of LangChain's *The Anatomy of an Agent Harness* could not be located, **no authoritative information is currently available**.
4. **The original source of the computer four-layer stack analogy**: the analogy is quotable, but its original source and accompanying cases are marked `[to be verified]`.
5. **Codex Auto-review's 1/200 and 99%**: B-level source, marked `[to be verified]`.
1. **The bottleneck layer distribution in the industry differentiation argument**: an analytical conclusion (judgment) from this project's eight industry groups, not a rating under any public standard; this should be noted when cited.

9. References

1. Effective harnesses for long-running agents — Anthropic, 2025. https://www.anthropic.com/engineering/effective-harnesses-for-long-running-agents
2. Harness design for long-running application development — Anthropic, 2026. https://www.anthropic.com/engineering/harness-design-long-running-apps
3. Effective context engineering for AI agents — Anthropic, 2025. https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents
4. Harness engineering: leveraging Codex in an agent-first world — OpenAI, 2026-02-11. https://openai.com/index/harness-engineering/
5. Harness engineering for coding agent users — Birgitta Böckeler, martinfowler.com, 2026. https://martinfowler.com/articles/harness-engineering.html
6. Humans and Agents in Software Engineering Loops — martinfowler.com, 2026. https://martinfowler.com/articles/exploring-gen-ai/humans-and-agents.html
7. My AI Adoption Journey — Mitchell Hashimoto, 2026-02-05. https://mitchellh.com/writing/my-ai-adoption-journey
8. Introducing the Model Context Protocol — Anthropic, 2024-11-25. https://www.anthropic.com/news/model-context-protocol
9. SWE-bench: Can Language Models Resolve Real-World GitHub Issues? — Jimenez, Yang, et al. (Princeton), 2023-10 (ICLR 2024 Oral). https://www.swebench.com/
10. Terminal-Bench official site — Stanford / Laude Institute, 2025. https://www.tbench.ai/
11. Interpretation of the national standard series "Artificial Intelligence Agent Interconnection" — China Industrial Economic Information Network, 2026. https://cinic.org.cn/xw/zcdt/1643418.html
12. Sandboxing: a safer and more autonomous approach — Anthropic, 2025. https://www.anthropic.com/engineering/claude-code-sandboxing