Eval (Agent Evaluation System)

1. Introduction

1.1. Background: The Crisis of Trust in Model Scores and the Failure of Self-Assessment Narratives

Between 2025 and 2026, AI adoption in the software engineering industry has passed the saturation threshold, yet the question "how effective is it?" has become increasingly hard to answer. Three sets of facts together sketch out this predicament:

• Public leaderboard numbers no longer represent real capability. On 2026-02-23, OpenAI released Why we no longer evaluate SWE-bench Verified, formally abandoning this benchmark once regarded as the coding-agent industry standard: re-examination found that at least 59.4% of the frequently-failed tasks contained flawed tests (rejecting functionally correct submissions), and that every frontier model tested could reproduce verbatim fragments of the gold patch from the task ID alone — score gains increasingly reflect training-set exposure (training contamination) rather than real capability.
• Subjective perception and objective measurement systematically diverge. In a randomized controlled trial released by METR on 2025-07-10, experienced open-source developers were measured to be 19% slower with AI tools, while the same developers self-reported being 20% faster. DORA 2025 (2025-11-12) shows over 90% of developers already use AI at work, about 80% perceive productivity gains, and 30% of tasks take over 4 hours per run — perceived gains are widespread, but perception is not measurement. The Stack Overflow 2025 developer survey (2025-07-30) likewise shows 84% of developers are using or planning to use AI tools, while 46% express distrust in the accuracy of AI output.
• Evaluation itself becomes the object of evaluation. When a team tries to answer "is our agent doing well?", it quickly discovers: input–output evaluation cannot see erroneous reasoning in the process, public benchmarks diverge from production distributions, and LLM-as-a-Judge itself hallucinates, has preferences, and contradicts itself across similar cases. The reliability of the evaluation system itself becomes a more fundamental concern than model capability.

These three sets of facts point to a single conclusion: any team that has not established an internal evaluation system cannot answer whether a given change made things better or worse. The reason the Eval direction exists is to turn this question from a subjective narrative into reproducible engineering measurement.

1.2. Definition

Eval direction (Agent Evaluation System): the engineering field that researches and builds measurement infrastructure for the quality of agent systems. It does not make agents stronger; it makes "strong or weak" decidable. Its core working objects fall into four categories:

1. Regression Set (Regression Set / Eval Set): solidify typical tasks and known failures into a dataset with expected outputs, then fully re-run and retain scorecards after every change (prompt, model, tool, or Harness configuration).
2. Trajectory Evaluation: rather than only looking at the final answer, evaluate the "path the agent took" — which tools were invoked, whether parameters were correct, whether the step count was reasonable, whether it fell into loops, and whether required steps were missed.
3. Evaluator Stack: a three-layer collaboration of code evaluators (deterministic, millisecond-scale), LLM-as-a-Judge (scalable semantic judgment, second-scale), and human annotation (Ground Truth and calibration), with continuous calibration of the evaluators themselves.
4. Online Evaluation: continuously apply evaluation to production traffic at a sampling rate, capturing drift, novel inputs, and real-world tool failures.

1.3. Position in the AI Harness Architecture

图 1-1｜Eval 在 Harness 六层能力模型中的定位

数据来源：基于本文分析绘制的示意图。

Per the six-layer capability model in the project parameter card, this direction's primary layer is the L5 Evaluation and Observability layer, with secondary layers being the L1 Context Engineering layer and the L4 Memory and State layer. This direction has a unique structural feature: the evaluation system itself is part of the Harness — the Golden Set is an assembly object of L1, the evaluator prompt is an artifact of L1, evaluation results are gating inputs to L3, and evaluation metadata is deposited as organizational memory in L4.

Linkage with the preceding documents: 01-agents.md is concerned with "who is doing the work," and 02-coding.md is concerned with "what is produced," and both reference evaluation gating through the implementation checklist in section 04.3; this direction provides the internal construction of that gate. Methodological review of evaluation standards themselves is undertaken by 06-benchmark.md — this direction answers "how well are we doing," while the Benchmark direction answers "is the ruler we judge with accurate."

1.4. The Bottleneck Layer

This direction's bottleneck is not in model capability, nor in collection infrastructure, but in the following three links:

1. The reliability of the evaluators themselves. LLM-as-a-Judge is subject to systematic bias: hallucination, style preference, preference for verbose answers, and self-contradiction across similar cases. If the evaluator uses the same model as the system under test, the self-scoring risk is further amplified. The calibration baseline given by the industry is: agreement between the Judge and human annotation should reach about 80% or higher; when disagreement exceeds 20%, first tighten the scoring criteria or lower the Judge temperature before relying on it for gating and monitoring.
2. The Evaluation–Production Gap. High scores on public benchmarks do not equal success under the conventions of a private codebase. Benchmarks serve only as baseline verification; teams must build private evaluation sets that match the production data distribution — and this is precisely the part that public leaderboards cannot help with.
3. Variance control. Agents are non-deterministic systems: repeated runs of the same configuration produce results with significant variance (see the case in 3.3 and 06-benchmark.md). A single-run score may be noise; a conclusion that does not account for variance cannot serve as a basis for decisions.

1.5. Value

• Turn "going by feel" into "going by scorecard". Every change becomes attributable: whether it improved or worsened, on which dimension, and at what cost.
• Turn failure into an asset. Once a production failure trajectory is solidified, it permanently becomes a regression-set item — the same mistake does not get paid for twice.
• Let teams dare to make changes. Only teams with a regression gate dare to keep iterating on prompts, upgrading models, and swapping tools; teams without a gate can only freeze the system.
• Make external narratives defensible. Citing any effect number can always be accompanied by the evaluation date, configuration, and sample size, holding up to the methodological scrutiny described in 06-benchmark.md.

2. Glossary

3. Case Studies

Note: The following cases are compiled from publicly verifiable primary sources. All effect figures in the cases are the figures published by the source; when citing them, the evaluation date and configuration must be noted together with the figure.

3.1. Case 1: An LLM-as-a-Judge Evaluation Loop in a Large-Scale Production Environment

Source: Google SRE, AI in SRE: How Google is Engineering the Future of Reliable Operations, https://sre.google/resources/practices-and-processes/ai-engineering-reliable-operations/

3.1.1. Background

Google's AI Operator, an AI on-call agent, has run across thousands of incidents and needs continuous improvement, but it is impossible to manually score every execution trajectory. At the same time, SRE scenarios place extremely high demands on evaluation: evaluation must not only judge whether things were "done correctly" but also drive improvement; otherwise evaluation is merely a cost.

3.1.2. Approach

Google established an evaluation-driven improvement loop:

• All execution trajectories are retained: every incident-handling trace is stored in Spanner, for rigorous debugging and continuous improvement.
• Golden Data comparison: the evaluation framework analyzes incident metadata and compares the agent's automated actions against human experts' ideal responses ("Golden Data").
• LLM-as-a-Judge evaluation: LLM-as-a-Judge is used to evaluate agent performance, determining success scenarios (correct diagnosis and mitigation) and failure scenarios (misdiagnosed root cause and failed mitigation).
• Evaluation results directly drive improvement: in failure scenarios, the Judge automatically generates a critique of the agent's logic and files a bug ticket containing a concrete implementation plan to improve the AI Operator's future performance.
• Promotion gating: promotion across autonomy levels (L1 assisted diagnosis to L4 full autonomy) must first demonstrate a sustained, statistically significant success rate on human-validated Golden evaluation data.

3.1.3. Results

• Evaluation is no longer an after-the-fact record but becomes an engine of improvement: the Judge's judgment is directly converted into an actionable bug ticket, forming a complete loop of "trace → Golden Data comparison → LLM evaluation → automated ticket → improve and re-evaluate."
• Golden Data becomes the basis for promoting autonomy: only when performance on Golden data is consistently statistically significant can the agent obtain a higher autonomy level for a specific bounded scenario. This design directly ties the evaluation system to security governance (L6).
• This case is currently the only publicly released AI on-call production practice with a large scale (thousands of incidents) and a complete evaluation loop.

3.2. Case 2: Internal Evaluation and Variance Attribution for a Multi-Agent System

Source: Anthropic, How we built our multi-agent research system, 2025-06-13, https://www.anthropic.com/engineering/multi-agent-research-system

3.2.1. Background

When building its multi-agent research system, Anthropic faced two evaluation challenges: first, the effectiveness of a multi-agent system cannot be measured with general-purpose benchmarks, so it had to build its own internal evaluation; second, agent behavior has high variance, so single-run scores are unstable and the dominant drivers of variance must be identified.

3.2.2. Approach

• Build an internal evaluation set: use an internal research eval as the primary metric, supplemented by human evaluation. Anthropic explicitly advises: do not wait until hundreds of test cases are gathered to start; instead, start immediately with small-scale testing on a few cases.
• Use LLM-as-a-Judge and human evaluation together: the LLM Judge handles scoring at scale, while human testers are responsible for discovering systematic blind spots.
• Variance attribution analysis: attribute the performance variance through decomposition to identify the dominant drivers.

3.2.3. Results

• The multi-agent system (Opus 4 orchestrator + Sonnet 4 sub-agents) scored 90.2% higher than a single agent on Anthropic's internal research eval (note: this is an internal evaluation result and cannot be extrapolated as a general industry benchmark).
• Cost profile: a normal agent flow consumes roughly 4× the tokens of a chat interaction, while multi-agent research consumes about 15× (multi-agent : single-agent : chat ≈ 15 : 4 : 1) — evaluation must be paired with cost, otherwise the value of the 90.2% improvement cannot be judged.
• Variance attribution on BrowseComp: token usage alone explains 80% of performance variance; adding tool-call counts and model selection explains 95% combined. This shows that in agent evaluation, the level of resource consumption itself is the strongest predictor, and evaluation design must control or record the token budget.
• Empirical evidence that human evaluation is irreplaceable: human testers found that early agents consistently preferred SEO-optimized content farms over authoritative sources (academic PDFs, personal blogs) — a systematic blind spot that the LLM Judge did not catch, resolved only after adding a source-quality heuristic.
• Production-shape parameters: the orchestrator spins up 3–5 sub-agents in parallel, and each sub-agent returns a condensed summary of 1,000–2,000 tokens, which avoids process context contaminating the main evaluation.

3.3. Case 3: Controlled Comparative Evaluation of Vendor Harnesses

Source: GitHub official blog, Evaluating performance and efficiency of the GitHub Copilot agentic harness across models and tasks, 2026, https://github.blog/ai-and-ml/github-copilot/evaluating-performance-and-efficiency-of-the-github-copilot-agentic-harness-across-models-and-tasks/

3.3.1. Background

The model provides raw intelligence, while the Harness determines how effectively that intelligence is applied — GitHub's own words are "the harness shapes how effectively that intelligence is applied". When a vendor wants to prove that its own Harness is not inferior to, or even better than, the Harness that comes with the model vendor (Claude Code, Codex CLI), a single ordinary benchmark run cannot produce a credible conclusion: the agent's run-to-run variance drowns out real differences.

3.3.2. Approach

GitHub's evaluation methodology (the core value of this case) is a set of controlled-variable evaluation (the φ method):

• Controlled-variable checklist: the same model, the same benchmark tasks, normalized context window, the same reasoning effort, the same tool selection and MCP servers — only the Harness itself is the variable.
• Variance analysis: on Terminal-Bench 2.0, each agent-model combination is run at least 5 times, with dispersion characterized by a ±1σ ellipse.
• Multi-dimensional benchmark matrix: SWE-bench Verified (coding standard), SWE-bench Pro (complex engineering), SkillsBench (skill extensibility), Terminal-Bench (terminal workflows), Win-Hill (cross-OS generalization), avoiding the bias of any single benchmark.

3.3.3. Results

• Under a fixed model and task, the GitHub Copilot agentic harness achieves a task completion rate on par with the harness that ships with the model vendor, while its token consumption is lower in most configurations.
• Key methodological conclusion: the observed differences fall within the run-to-run variance caused by model randomness — "a leaderboard difference of a few percentage points is very likely just run variance." This conclusion directly challenges the validity of single-run leaderboard comparisons without variance analysis, and is one of the empirical pillars of the core argument of 06-benchmark.md (that the harness has become an evaluation variable).
• Implication for enterprise teams: if an internal evaluation runs only once per configuration, the score should be treated as a single sample rather than a capability estimate; when budget allows, run at least 5 repetitions and report the interval.

4. Practice Standards

4.1. AGENTS.md Specification

The following is the complete AGENTS.md text for this direction, placed in a code block so it can be copied directly. When using it, overlay the group-level AGENTS.md common clauses; where they conflict, resolve them by the nearest-first principle.

# AGENTS.md —— 软件工程 · Eval 方向（智能体评估体系）

## 角色与边界

- 本文件约束**建设与运行智能体评估体系**的工作：回归集构建、轨迹评估器开发、LLM-as-a-Judge 设计与校准、在线评估与采样策略、评估结果接入发布门控。
- 允许自行完成：编写评估用例与判定脚本、设计评分准则、实现代码评估器、配置采样率、维护回归集与失败样本库。
- 禁止自行完成：修改或删除已有回归集条目的期望产出、绕过评估门禁合入变更、调整在线评估采样率以隐藏退化、把 Judge 评分直接作为对外宣传数字。
- 边界判定：凡影响"某变更能否通过门禁"的改动（阈值、判定逻辑、样本集），一律视为门禁相关变更，须人工评审并留痕。

## 环境假设

- 提供 Trace 采集与存储，智能体全过程工具调用可回放。
- 提供评估数据集存储与版本管理，回归集条目带唯一 ID 与版本号。
- 提供 LLM Judge 运行环境，Judge 所用模型版本与温度可配置且被记录。
- 提供评估运行器，支持批量执行、超时控制与成本计量。
- 提供评分历史存储，支持跨版本趋势查看。
- 缺失任一项时，在评估报告中显式声明缺失项，不得静默降级。

## 上下文加载顺序（Context Budget）

1. 评估任务契约：被测对象、评估维度、通过阈值、预算上限（固定，不压缩）
2. 失败模式分类学：本系统已知的失败家族与典型样本
3. 相关评分准则（Rubric）与 Judge prompt 当前版本
4. 回归集条目清单（ID、维度、期望产出类型）
5. 评分历史：近版本分数趋势与已知波动区间
6. 采样与门禁配置
7. 被测变更的差异说明（按需加载）

## 工具契约

- 评估运行工具：输入为数据集版本 + 被测配置，输出为评分卡；必须记录每次运行的配置快照。
- Judge 调用工具：必须固定并记录模型版本、温度、prompt 版本；输出必须为结构化 schema（label / score / failure_mode / explanation）。
- Trace 查询工具：支持按任务 ID、时间窗、失败类别检索。
- 所有评估工具输出禁止包含被评估数据之外的用户敏感信息；确需脱敏示例时先脱敏再入库。

## 任务执行流程（SOP）

1. 明确评估问题：这次评估要回答什么（回归检测？版本对比？上限校准？）。
2. 选择评估维度：轨迹 / 工具使用 / 任务完成 / 多轮稳定，明确各维度的指标与判定方式。
3. 构建或扩充数据集：优先从生产失败轨迹转化；新用例须覆盖已知失败类别与边界案例。
4. 为需要语义判断的维度设计 Judge：一条 Judge 只判一个准则，写明证据要求、通过/失败条件、边界案例与结构化输出 schema。
5. 校准：人类标注样本与 Judge 比对，一致性达到约 80% 以上方可用于门禁；不达标先收紧 Rubric 或换 Judge 模型。
6. 运行评估：每配置至少多次运行并报告均值与离散度；记录成本与延迟。
7. 产出评分卡与结论：结论必须注明配置、日期、样本量与方差；不带四要素的分数不得写入报告。
8. 回流：失败样本永久加入回归集；评估器缺陷记入评估器版本变更记录。

## 验证与证据要求

必须满足并留存证据：

| 验证项 | 判定方式 | 证据 |
|---|---|---|
| 判定可复现 | 同一配置重复运行，结果在报告的方差区间内 | 多次运行记录 |
| Judge 与人类一致 | 校准样本上一致性不低于约 80% | 校准报告 + 样本清单 |
| 无自我评分 | Judge 模型与被测智能体所用模型不同版本或经隔离验证 | 配置记录 |
| 回归集防过拟合 | 回归集未参与被测对象的任何优化循环；有held-out子集 | 数据集版本记录 |
| 成本配对 | 成功率与单任务成本、延迟同时报告 | 评分卡 |
| 门禁生效 | 低于阈值的变更被实测拒绝合入 | 门禁触发记录 |
| 评估器版本可追溯 | 每个分数可对应到评估器 prompt 与 Judge 模型版本 | 版本台账 |

禁止以单次运行分数作为结论；禁止以"多数用例通过"替代逐条失败清单。

## 失败与升级策略

| 失败 | 处理 |
|---|---|
| Judge 与人类分歧超过 20% | 停用该 Judge 的门禁资格，收紧 Rubric 或更换 Judge 模型后重新校准 |
| 评估分数与主观体感背离 | 优先怀疑评估集分布偏差；检查评估集是否覆盖真实任务类型 |
| 同配置分数大幅波动 | 扩大重复次数并报告区间；波动持续则排查沙箱环境漂移 |
| 回归集疑似被过拟合 | 引入 held-out 子集与轮换集；冻结部分条目不进入优化回路 |
| 在线指标与离线评分背离 | 检查采样代表性；生产失败样本回流扩充离线集 |
| 评估成本超预算 | 先降采样率与 Judge 覆盖面，禁止牺牲重复次数换取覆盖面 |

升级三要素：现象与数据、已排除的原因、建议选项。缺一无效。

## 安全与合规红线

1. 禁止评估数据集包含未脱敏的用户数据、密钥与生产凭证。
2. 禁止被测智能体访问评估集的期望产出；评测环境与被测环境必须隔离。
3. 禁止修改门禁阈值以放行未达标变更；阈值调整须人工评审并记录。
4. 禁止在报告中混用自评口径与实测口径的数字；引用厂商数字必须标注 self-reported。
5. 禁止删除评分历史；历史分数是趋势分析与质疑复盘的证据。
6. 涉及对外发布的效果声明，须能回溯到评分卡原始记录。

## 禁止事项

1. 禁止用最终答案正确掩盖过程缺陷；轨迹维度必须独立评估。
2. 禁止让一条 Judge 同时评判正确性、语气、安全、效率等多个准则。
3. 禁止只因轨迹与参考路径不同就判失败——只要替代路径安全、正确且不更低效，应判通过。
4. 禁止把公开榜单分数作为内部能力结论；榜单只作上限校准。
5. 禁止引用不带评测日期、配置与样本量的分数。
6. 禁止使用 `[待填写]`、`XX`、`___` 之外的非标准占位符；不确定处统一使用 `[待填写]`。

## 输出格式

- 评分卡：维度分数、配置快照、评测日期、样本量、均值与离散度、单任务成本
- 失败清单：逐条列出失败用例、失败类别、根因假设
- 校准报告：一致性数值、分歧样本分析、评估器版本
- 回归集变更记录：新增/冻结条目及理由
- 评估器变更记录：prompt/Rubric/Judge 模型的版本差异与影响

## 评估与自检

| 自检项 | 判定 |
|---|---|
| 每个维度是否都有独立评估器 | 是 / 否 |
| Judge 是否单准则、结构化输出、与被测模型隔离 | 是 / 否 |
| 是否经过人类校准且一致性达标 | 是 / 否 |
| 分数是否带四要素（模型、日期、配置、样本量） | 是 / 否 |
| 成功率是否与成本延迟配对报告 | 是 / 否 |
| 失败样本是否已回流回归集 | 是 / 否 |
| 门禁是否实测拒绝过未达标变更 | 是 / 否 |
| 评估器版本是否可追溯 | 是 / 否 |

任一"否"须显式列出；自检不通过即任务未完成。

4.2. SKILL.md Specification

---
name: agent-eval-regression-suite
description: 为智能体系统构建并运行一套回归评估体系：从失败模式分类学出发设计评估维度，建立 Golden Set 与回归集，配置三层评估器（代码断言、LLM Judge、人工校准），接入 CI 门禁并产出带方差与成本的评分卡。当需要为智能体建立质量门禁、在模型或提示词变更后做回归验证、或评估分数出现不可解释波动时使用。
version: 1.0
created: 2026-09-12
---

# 智能体回归评估体系构建

## 适用场景

- 智能体系统即将进入持续迭代，需要建立"每次变更都可归因"的度量基线。
- 模型升级、提示词修改、工具替换后，需要判断是变好还是变坏。
- 评估分数出现不可解释的波动，需要做方差分析与归因。
- 生产出现新的失败模式，需要固化为回归条目防止复发。

不适用于：模型预训练评测、公开榜单刷榜、无判定标准的纯主观体验调研。

## 前置条件

| 类别 | 要求 | 缺失时处置 |
|---|---|---|
| Trace | 智能体全过程可回放 | 先补 Trace 采集，否则只能做 IO 评估并声明局限 |
| 任务类型清单 | 主要任务类型与占比已知 | 依据生产日志聚类补齐 |
| 判定标准 | 至少核心维度有可判定或可语义判断的标准 | 先从代码评估器可覆盖的维度起步 |
| Judge 环境 | 独立 LLM 可调用且版本可固定 | 暂缓语义维度，先跑确定性检查 |
| 预算 | 评估运行的 token/调用预算上限 | 设定保守默认上限并记录 |

## 输入

- 被测智能体配置（模型版本、Harness 配置、工具清单）
- 生产失败轨迹样本（如有）
- 已知失败模式清单与边界案例
- 通过阈值与成本预算

## 输出

- 回归集数据集（条目带 ID、维度、期望产出、来源标注）
- 三层评估器配置（代码断言脚本、Judge prompt 与 schema、人工标注规程）
- 评分卡（各维度分数、均值与离散度、单任务成本、配置快照）
- 校准报告（Judge 与人类一致性、分歧样本分析）
- 失败清单与回流记录

## 执行步骤

1. **失败模式分类先行**：不要从通用"helpfulness evaluator"出发，先按失败家族（路由错误、工具误用、参数非法、跳过必需校验、重复动作、检索无关、恢复失败、无依据断言、目标漂移、未经批准的关键操作）归类已知失败。
2. **构建起步集**：从 30–50 个精心标注的用例开始，覆盖主要任务类型加少量对抗性边界；从少量用例立刻起步，不要等凑齐数百条。
3. **维度分解**：按轨迹（步数、循环、必需步骤）、工具使用（选型、参数、错误恢复）、任务完成（有/无 Ground Truth 的达成度）、多轮（上下文保持、目标漂移）四维选择指标。
4. **三层评估器配置**：
   - 每条采样 trace 先跑廉价的代码检查（确定性、毫秒级）；
   - 需要语义判断的子集交给 LLM Judge——一条 Judge 只判一个准则，输出结构化 schema（label: pass/fail/insufficient_evidence，score 0.0–1.0，failure_mode，explanation）；
   - 分歧或低置信度案例路由到人工标注队列。
5. **Judge 校准**：人类标注样本比对，一致性达到约 80% 以上方可用于门禁；分歧记录而非强制统一——分歧本身可能揭示 Rubric 歧义。
6. **方差控制**：每个配置至少多次运行，报告均值与离散度；对照 run-to-run 方差解读差异，单次采样不得作为结论。
7. **接入门禁**：触及提示词/工具/模型配置的变更必须跑场景评估并以 status check 发布分数；任务成功率低于基线或单任务成本超限则构建失败。
8. **回流闭环**：生产失败 trace → 转为数据集条目 → 在评估中复现 → 修复 → 永久保留为回归测试项。
9. **持续校准**：agent 或 Harness 变更、prompt/Rubric 变更、Judge 模型变更、工具或上下文变更、用户分布变更、门限变更——六类触发条件任一发生即重新校准评估器，并递增评估器版本号。

## 质量标准（DoD）

1. 评估维度覆盖轨迹、工具使用、任务完成至少三维，且维度间独立打分。
2. 每条 Judge 单准则、结构化输出、证据要求明确，且与被测模型隔离。
3. Judge 与人类标注一致性不低于约 80%，校准报告可追溯。
4. 回归集含 held-out 子集，未参与被测对象的优化循环。
5. 分数报告包含四要素（模型、日期、配置、样本量）与重复次数。
6. 成功率与单任务成本、延迟配对报告。
7. 失败样本已回流，生产失败 trace 的转化记录完整。
8. 门禁经实测拒绝过未达标变更（有触发记录）。
9. 评估器版本台账完整，"产品质量变化"与"度量方式变化"可区分。
10. 评估数据无未脱敏敏感信息，期望产出对被测智能体不可见。

## 常见失败与处理

| 失败 | 表现 | 处理 |
|---|---|---|
| 评估—生产鸿沟 | 公开基准高分但私有库任务失败 | 构建匹配生产分布的私有评估集，基准只作基线验证 |
| Judge 自我评分 | Judge 与被测同模型，分数系统性偏高 | 更换独立模型作 Judge，或引入隔离验证 |
| 多准则混评 | 一条 Judge 同时打六个维度分，结果不可解释 | 拆分为独立评估器，逐维度出分 |
| 路径惩罚误判 | 替代解法被参考路径比对判失败 | 改为结果导向判定：安全、正确、不更低效即通过 |
| 用重试虚增成功率 | 成功率高但单任务成本飙升 | 成功率与成本强制配对，设成本上限门禁 |
| 回归集过拟合 | 离线分数涨、线上指标跌 | 引入 held-out 与轮换集；检查条目是否泄漏进优化回路 |
| 方差淹没差异 | 版本间差异小于 run-to-run 波动 | 增加重复次数并报告区间；差异在方差区间内即判"无可辨别差异" |

## 示例

**场景**：某团队的代码修复智能体升级了提示词，需要判断是否合入。

- 起步集：42 条用例（30 条典型修复 + 8 条边界 + 4 条对抗输入），全部来自历史任务与生产失败轨迹。
- 维度：任务完成（测试通过率，代码评估器）、轨迹质量（必需工具命中 + 步数上限，代码评估器）、修复说明质量（单准则 Judge，判"是否说明了根因与验证方式"）。
- Judge 校准：人类标注 50 条，与 Judge 一致性 86%，达标，允许用于门禁。
- 运行：新旧配置各跑 5 次。新配置任务成功率 78.4% ± 2.1，旧配置 77.9% ± 2.3；差异落在方差区间内，判"无可辨别差异"；但新配置单任务成本下降 12%，为主要收益依据。
- 门禁：合入；评分卡与失败清单（3 条轨迹类失败已回流回归集）随 PR 归档。

4.3. Implementation Checklist

Evaluation Design Stage

• [ ] The evaluation question is clear: at least one of regression detection / version comparison / upper-bound calibration
• [ ] A failure-mode taxonomy has been established, covering the known failure families of this system
• [ ] Evaluation dimensions cover at least three of trajectory / tool use / task completion / multi-turn, with independent scoring
• [ ] A starter evaluation set of 30–50 items covers the main task types and adversarial boundaries
• [ ] Regression-set items carry a unique ID, version number, and source annotation
• [ ] The regression set includes a held-out subset that does not participate in the optimization loop of the object under test

Evaluators and Calibration

• [ ] Code evaluators cover all checks that can be decided deterministically
• [ ] Each LLM Judge uses a single criterion, structured output (label / score / failure_mode / explanation), and clear evidence requirements
• [ ] The Judge model is isolated from the agent model under test, with its version and temperature recorded
• [ ] Judge–human annotation agreement reaches about 80% or higher, with a traceable calibration report
• [ ] The human annotation protocol covers nine sample types (clearly pass, clearly fail, boundary, each failure category, different users and task segments, different versions, adversarial phrasing, correct answer with bad process, good process with external failure)
• [ ] Annotators label without seeing the automated scores, and disagreements are recorded rather than forcibly unified

Execution and Variance

• [ ] Each configuration is run multiple times, reporting mean and dispersion
• [ ] Single-run scores are not used as a conclusion
• [ ] Success rate is reported together with per-task cost and latency
• [ ] A configuration snapshot (model, Harness, tools, date) is retained for each run
• [ ] The six types of evaluator re-calibration triggers have monitoring and an owner

Gating and Feedback

• [ ] Changes touching prompt/tool/model configuration are forced to run evaluation and publish scores as a status check
• [ ] The build fails when success rate is below baseline or cost exceeds the cap, and this has actually been triggered
• [ ] There is a fixed channel for converting production failure traces into regression items
• [ ] Gate threshold adjustments have a human-review record
• [ ] The evaluator version ledger is complete, and scores can be traced back to the evaluator version

Compliance

• [ ] Evaluation data contains no un-anonymized sensitive information
• [ ] Expected outputs are not visible to the agent under test
• [ ] Self-assessed and measured figures are not mixed; vendor figures are labeled self-reported

5. Summary

The question the Eval direction answers is: when an agent system keeps iterating, who proves that each change makes things better rather than worse?

This direction has four core conclusions:

1. Evaluation is the L5 layer of the Harness, and the evaluation system itself is also part of the Harness. The Golden Set is an artifact of context engineering, evaluation results are gating inputs to the orchestration layer, the failure-sample library is organizational memory, and evaluation metadata is the raw material for governance audits. Treating evaluation as a "testing stage" outside the Harness is the most common positioning mistake of this direction.
2. Trajectory evaluation is the watershed between agent evaluation and traditional software testing. A correct answer can hide faulty reasoning: a hallucinated tool call may happen to get lucky, and a convoluted path may still reach the goal. Looking only at input–output means paying for latency, cost, and security risk without realizing it. But trajectory evaluation must be outcome-oriented — as long as an alternative path is safe, correct, and no less efficient, it should not be judged a failure for deviating from the reference path.
3. LLM-as-a-Judge is a necessity, but also a tool with limits. Its correct usage is: single criterion, structured output, isolation from the model under test, human calibration (about 80% or higher agreement), and routing of disagreements to humans. Anthropic's practice also reminds us: the Judge cannot catch every systematic blind spot (such as content-source quality bias), and human evaluation cannot be fully replaced.
4. Scores without variance and cost have no decision value. GitHub's controlled comparison (at least 5 runs per configuration, dispersion characterized by ±1σ) proves that differences of a few percentage points may just be run variance; Anthropic's attribution analysis proves that the token budget itself can explain most of the performance variance. Evaluation conclusions must be presented in the form of "mean + dispersion + per-task cost."

The recommendation for teams rolling this out is: start with 30–50 real failure samples, first build deterministic checks, then introduce single-criterion Judges, and finally connect gating and online sampling. The evaluation system and agent capability should grow together — an agent system without a regression set is jumping off a cliff in the dark with every prompt change.

Information Gap Statement

1. Public AI SRE benchmarks: no results at all. This round of search found no authoritative public benchmark for evaluating AI fault localization / RCA / on-call capability; the Golden Set and Judge loop described in this direction exist only in internal-evaluation form, so enterprises can only build their own when implementing.
2. Public DevOps / IaC agent benchmarks: a severe gap. Infrastructure/DevOps-type tasks account for only 0.41% (3 items) of the full SWE-Lancer set, and public monetized benchmarks have almost no coverage of the infrastructure direction.
3. Chinese GB/T-type AI coding standards: none found. This document cites no domestic standard numbers, and strictly forbids fabricating numbers later.
4. The specific calibration threshold for LLM-as-a-Judge (about 80% agreement) comes from vendor methodology documents (Arize, assisters.io, etc.) and is an engineering-experience measure rather than a standard provision; when citing it, the source nature must be noted.
5. For the DORA 2025 usage figures, this group's preceding documents uniformly use the over-90% measure (the report's survey window was 2025-06-13 to 2025-07-21); different retold sources have both 90%/95% formulations, and this document uses the measure consistent with the preceding documents.
6. The 90.2% improvement in Anthropic's internal research eval in Case 3.2 is an internal evaluation result and cannot be extrapolated as a general industry benchmark or a basis for cross-vendor comparison.

6. References

1. Why we no longer evaluate SWE-bench Verified — OpenAI (Mia Glaese & Olivia Watkins), 2026-02-23. https://openai.com/index/why-we-no-longer-evaluate-swe-bench-verified/
2. AI in SRE: How Google is Engineering the Future of Reliable Operations — Google SRE. https://sre.google/resources/practices-and-processes/ai-engineering-reliable-operations/
3. How we built our multi-agent research system — Anthropic, 2025-06-13. https://www.anthropic.com/engineering/multi-agent-research-system
4. Evaluating performance and efficiency of the GitHub Copilot agentic harness across models and tasks — GitHub Blog, 2026. https://github.blog/ai-and-ml/github-copilot/evaluating-performance-and-efficiency-of-the-github-copilot-agentic-harness-across-models-and-tasks/
5. AI agent tracing and evaluation: The complete developer guide — Arize AI, 2026. https://arize.com/?p=30432/
6. LLM Evaluation Framework: Trajectories vs. Outputs — LangChain, 2026. https://www.langchain.com/resources/llm-evaluation-framework
7. AI agent evaluation: trajectory, tool calls, and task completion — Langfuse, 2026. https://langfuse.com/resources/engineering/ai-agent-evaluation
8. AI Agent Evaluation Framework: Metrics & Best Practices — annotation.academy, 2026. https://annotation.academy/blog/ai-agent-evaluation-framework
9. How to Test and Evaluate AI Agents in 2026 — assisters.io, 2026. https://www.assisters.io/blogs/how-to-test-and-evaluate-ai-agents
10. SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks — arXiv:2602.12670, 2026-02-13. https://arxiv.org/html/2602.12670v1
11. SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering? — OpenAI, arXiv:2502.12115. https://arxiv.org/html/2502.12115v4
12. Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity (randomized controlled trial) — METR, 2025-07-10. https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/
13. State of AI-assisted Software Development 2025 — DORA / Google Cloud, 2025-11-12. https://dora.dev/research/2025/dora-report/
14. Stack Overflow 2025 Developer Survey — Stack Overflow, 2025-07-30. https://survey.stackoverflow.co/2025/
15. Terminal-Bench official leaderboard — Laude Institute et al. https://www.tbench.ai/