Benchmark (Benchmarks and Evaluation Methodology)

1. Introduction

1.1. Background: The Triple Trust Crisis of Leaderboard Narratives

Since SWE-bench Verified became the de facto industry standard for coding agents in 2024, the "leaderboard score" was once the common currency for the industry's narrative of model capability progress. From 2025 to 2026, this currency went through three crises of trust:

• First: the test itself is flawed. OpenAI deprecated SWE-bench Verified on 2026-02-23; its review found that at least 59.4% of the frequently failed tasks contained flawed tests — either rejecting functionally correct submissions (tests too narrow) or requiring functionality never mentioned in the task (tests too broad). The benchmark no longer measures "what it claims to measure".
• Second: training contamination. OpenAI's audit showed that every frontier model tested (including GPT-5.2, Claude Opus 4.5, and Gemini 3 Flash) could reproduce verbatim fragments of the gold patch given only the task ID; score improvements increasingly reflect training-period exposure rather than genuine capability.
• Third: caliber drift. The score differences for the same model across different scaffolds (evaluation scaffolds / Harness), different reasoning efforts, and different evaluation organizations can exceed the differences between model generations. Horizontal comparison across leaderboards has essentially broken down.

Meanwhile, alternative benchmarks (SWE-bench Pro, the Terminal-Bench family, SWE-bench-Live, SWE-Lancer, SkillsBench) are rebuilding credibility with "longer-horizon tasks + less contamination + dynamic updates + stricter anti-cheating". This rebuilding process itself constitutes the rationale for the Benchmark direction as an independent engineering direction: the methodology of evaluation has become an object that must be specially designed, reviewed, and maintained.

1.2. Core Thesis: Harness Has Become an Evaluation Variable

The core thesis of this document is: discussing model scores apart from the Harness is meaningless. This thesis is supported by three groups of verifiable facts and runs throughout the document:

1. Switching scaffolds for the same model can differ by 11.6 points. On SWE-bench Pro, Claude Opus 4.5 scores 45.9% through the standardized SWE-Agent scaffold, and 51.8% through the Auggie CLI; Opus 4.6 + Morph's WarpGrep v2 search sub-agent reaches 57.5% — part of the difference comes not from a better model but from better orchestration (see Section 3.2).
2. The variance of multiple runs of the same configuration can reach about ±3 points. In its Terminal-Bench 2.0 variance analysis, GitHub officially ran each agent-model combination at least 5 times, used ±1σ to characterize dispersion, and concluded that the observed harness-to-harness differences fall within the run-to-run variance of model randomness (see Section 3.3 of 05-eval.md and Section 3.3 of this document).
3. Scores for the same benchmark drift systematically across evaluation organizations. On Terminal-Bench 2.1, the top score on the official leaderboard (tbench.ai) differs by several points from that of the independent evaluation organization (Artificial Analysis, Terminus 2 + e2b sandbox, pass@1 averaged over 3 repetitions) — the evaluation environment itself is part of the score.

The inference drawn from these three groups of facts is: every score on a public leaderboard is a composite product of "model × Harness × configuration × run variance". Citing a score without noting these four variables is equivalent to citing an unexplainable number. This is also the methodological basis for the challenge "inconsistent evaluation calibers and leaderboard drift" in Section 4.5 of this group's README.md.

1.3. Definitions

Benchmark direction (benchmarks and evaluation methodology): the engineering field that studies and reviews evaluation benchmarks and evaluation protocols themselves. It is not responsible for scoring a given model or product (that is the daily work of the Eval direction), but rather for answering:

• Does a given benchmark actually measure the capability it claims to measure (construct validity)?
• Has the benchmark already been contaminated by training data? How can contamination be detected and resisted?
• How should evaluation protocols (scaffold, reasoning effort, step and cost caps, number of repetitions, judging method) be designed so that scores are comparable and reproducible?
• Between scores from two leaderboards, two dates, and two configurations, which comparisons are legitimate?
• How is a benchmark's lifecycle managed (release, revision, deprecation, succession)?

Division of labor with 05-eval.md: the Eval direction builds its own measurements, while the Benchmark direction reviews the public ruler and provides methodological constraints for internal evaluation.

1.4. Position in the AI Harness System

图 1-1｜Benchmark 方向在六层 Harness 能力模型中的定位

数据来源：基于本文分析绘制的示意图。

According to the six-layer capability model of the project parameter card, the primary layer of this direction is L5 Evaluation and Observation Layer (methodology side), with secondary layers of L2 Tools and Execution Layer (running containerized evaluation harnesses) and L6 Governance and Security Layer (contamination prevention, anti-cheating, submission verification).

1.5. Value

• Make every cited number defensible. Score citations with the four elements (model + version, benchmark, evaluation date, configuration) withstand scrutiny.
• Keep selection decisions from being misled by leaderboards. After identifying caliber drift, run variance, and training contamination, technology selection returns to the internal regression set (05-eval.md) rather than public leaderboards.
• Provide a methodological template for building your own evaluations. Terminal-Bench's four-stage quality control, SWE-bench Pro's contamination-resistant design, and SWE-Lancer's economic valuation are all direct references for designing an enterprise's internal evaluation set.
• Bring the industry narrative back to honesty. Distinguishing vendor self-reports from independent verification, and official leaderboards from third-party scaffold leaderboards, is the basic discipline of technical communication.

2. Glossary

3. Case Studies

Note: The following case studies are compiled from public, verifiable primary sources. The scores in the case studies are as published by their sources; when citing them, the evaluation date and configuration must be stated alongside.

3.1. Case 1: The Official Deprecation of SWE-bench Verified

Source: OpenAI, "Why we no longer evaluate SWE-bench Verified" (Mia Glaese & Olivia Watkins), 2026-02-23, https://openai.com/index/why-we-no-longer-evaluate-swe-bench-verified/

3.1.1. Background

SWE-bench Verified (500 human-verified open-source Python repository bug-fixing tasks, first released 2024-08) was the most widely cited benchmark for coding agents. By early 2026, its score over the most recent 6 months had only risen from 74.9% to 80.9%, while model vendors were still using it as a headline marketing figure. OpenAI judged that the benchmark "no longer reflects a meaningful improvement in the model's real software development ability" and decided to formally deprecate it.

3.1.2. Approach (Audit and Evidence)

OpenAI's deprecation decision was built on two groups of audit evidence:

• Flawed-test audit: reviewing the 27.6% subset of tasks that models frequently failed, it found that at least 59.4% of them contained flawed tests — rejecting functionally correct submissions; in another audit of 138 frequently lost tasks, over 60% were unsolvable per the problem statement, of which 49 tests were too narrow and 26 too broad.
• Training-contamination audit: every frontier model tested by OpenAI could reproduce verbatim fragments of the gold patch given the task ID; the chain-of-thought trace of GPT-5.2 even exposed knowledge of unspecified test requirements, hinting that the test patch appeared in the training data.
• Official recommendation: migrate to SWE-bench Pro.

3.1.3. Impact and Subsequent Landscape

• After the deprecation announcement, Google DeepMind, Anthropic, and Meta continued to report Verified scores but began also giving Pro figures; meanwhile OpenAI itself still used a 88.7% Verified result as its headline number when releasing GPT-5.5 on 2026-04-23 — a dual-track reporting landscape is expected to run through 2026.
• The two calibers of Verified's current top scores must be presented side by side: Claude Opus 4.7's 87.6% (2026-04-16) and GPT-5.5's 88.7% (2026-04-23), both being third-party compilation calibers, marked [To be verified], and when citing them one must also state the fact that the benchmark was deprecated on 2026-02-23, otherwise it constitutes misleading.
• The methodological lesson established by this case: a benchmark is an asset that expires. Any external claim based on a benchmark score should state the benchmark's current status (in use / revised / deprecated).

3.2. Case 2: SWE-bench Pro and the Scaffolding Effect

Source: Scale AI (+ Princeton), SWE-bench Pro official leaderboard, released 2025-09-21, https://scaleapi.github.io/SWE-bench_Pro-os/

3.2.1. Background

SWE-bench Pro (1,865 problems from 41 actively maintained repositories, split into public / held-out / commercial subsets) is a direct response to Verified's flaws: tasks modify at least 10 lines of code, reference solutions span 4.1 files and 107.4 lines on average, and professional engineers need hours to days; by drawing on GPL copyleft repositories and private proprietary codebases to build a dual legal-and-access barrier, no model under test could generate a verbatim gold patch at release — i.e., a natively contamination-resistant design.

3.2.2. Approach (Evaluation Protocol)

• Unified scaffold: the official leaderboard adopts the standardized SWE-Agent scaffold, stripping away harness variables to compare only models — a key move by the benchmark to proactively unify the harness.
• Pass@1 protocol; no cost cap, with a round cap of 250; the 95% confidence interval is computed using binomial proportion statistics (caliber of 730 problems).
• Failure-mode analysis: GPT-5 is used as a judge to classify error modes over the last 20 rounds of unresolved instances, with a human-calibrated agreement rate of 87%.

3.2.3. Impact

• Official leaderboard results (unified SWE-Agent scaffold caliber): Claude Opus 4.5 achieved 55.53% on 2025-11-26, the highest on the official leaderboard; GPT-5.2-Codex scored 51.6%. At release (2025-09 cold start), all models had a Pass@1 below 25%.
• The Scaffolding effect (the most valuable finding of this direction): the same model with different scaffolds can differ by up to 11.6 points — Claude Opus 4.5 scores 45.9% through the standardized SWE-Agent scaffold, and 51.8% through Augment Code's Auggie CLI; Opus 4.6 + Morph's WarpGrep v2 search sub-agent reaches 57.5% (third-party scaffold caliber). Conclusion: part of the difference comes not from a better model but from better orchestration.
• Comparability discipline: the official unified-scaffold leaderboard and results that include third-party scaffolds are two different calibers, and their scores cannot be compared horizontally; the caliber must be stated when citing. That high scores including third-party scaffolds (e.g., 57.5%) exceed the official leaderboard is precisely direct evidence that "the Harness has become an evaluation variable".

3.3. Case 3: Terminal-Bench's Quality Control and Dual-Leaderboard Drift

Source: Terminal-Bench official leaderboard (tbench.ai) and Artificial Analysis's independent evaluation, https://www.tbench.ai/leaderboard/terminal-bench/2.1 and https://artificialanalysis.ai/evaluations/terminalbench-v2-1

3.3.1. Background

The Terminal-Bench family (first release 2025-05-19; 2.0 released 2025-11-07 with 89 tasks + the new evaluation harness Harbor; 2.1 released 2026-05-06, fixing 28 tasks) measures agents' ability on long-horizon terminal tasks. Because terminal tasks are judged fully automatically (looking only at results, not at which commands were used), it becomes a natural test bed for examining "the credibility of the evaluation environment".

3.3.2. Approach (Quality Control and Anti-Cheating)

• Four-stage quality control: ① automated verification (reference solutions must pass, placeholder solutions must fail); ② LLM-assisted review (instruction ambiguity, cheating opportunities, inconsistencies between description and tests); ③ human-expert review of each task; ④ adversarial testing (a dedicated "cheating" agent tries to exploit loopholes).
• Scoring principle: tasks are judged only by results, not by which commands the agent chose.
• Anti-cheating upgrade (from 2026-08): every submission must provide the agent's rollout trajectory, which maintainers verify to ensure that only the problem statement and Docker image were provided to the agent, and that the gold patch and judging results were not leaked; submissions verified by humans are marked with a Verified tag.
• Example of a cheating penalty: on the Terminal-Bench 2.1 leaderboard, one submission of Cursor CLI + Grok 4.5 was penalized -9.0% for PR Hacks.

3.3.3. Impact

• Capability-progress anchor: on Terminal-Bench 2.0, Claude Sonnet 4.5 became the first score to break 50% on this leaderboard at 51.0% on 2025-09-29, and GPT-5.2 reached 54.0% on 2025-12-11; the 2.0 paper states that frontier models and agents scored below 65% (another formulation is that the best combination, GPT-5.2 + Codex CLI, reached only 62.9%; the two numbers come from different secondary sources, marked [To be verified]).
• Dual-leaderboard drift (the core empirical evidence of this case): for the top score on Terminal-Bench 2.1, the official leaderboard gives Claude Code + Fable 5 at 83.8% ± 1.2 (xhigh, 2026-06-07); Artificial Analysis's independent evaluation (Terminus 2 + e2b sandbox, pass@1 averaged over 3 repetitions) gives Claude Fable 5 at 84.6% — the two calibers must be presented side by side, and taking either one alone constitutes a caliber error. The few-point difference comes from differences in the evaluation environment, the number of repetitions, and judging details — the best empirical evidence of "harness / environment differences causing score drift".
• Methodological implication: the consistency between the official and independent leaderboards (difference below the sub-1-point magnitude) shows that the benchmark is broadly robust; the existence of the difference shows that the evaluation environment is a component of the score, and citing a score apart from its environment is meaningless.

4. Practice Standards

4.1. AGENTS.md Specification

The following is the complete original AGENTS.md text of this direction, placed in a code block for direct copying. When used, the common clauses of the group-level AGENTS.md must be layered on top, and conflicts are resolved by the proximity-first principle.

# AGENTS.md —— 软件工程 · Benchmark 方向（基准与评测方法学）

## 角色与边界

- 本文件约束**建设、运行与审查评测基准**的工作：基准任务构建、评测 harness 搭建与维护、评测协议设计、污染检测、榜单数据治理、分数口径审查。
- 允许自行完成：编写基准任务与环境镜像、实现判定器、配置统一 scaffold、执行污染检测、整理榜单口径与引用规范。
- 禁止自行完成：修改已发布基准的任务或判定逻辑（属基准修订，须走版本化流程）、删除或改写历史评测结果、为特定被测对象调整判定参数、把未经提交验证的成绩标注为 Verified。
- 边界判定：凡影响"分数是否可比较"的改动（scaffold、判定器、环境镜像、协议参数），一律视为协议变更，须版本化并公告。

## 环境假设

- 提供容器化评测环境，每任务独立环境，互不泄漏状态。
- 提供统一 scaffold 的可执行实现，其版本被固定并记录。
- 提供判定器自动化验证：参考解必须通过、占位解必须失败。
- 提供轨迹存档：每次评测运行的完整轨迹可提交与回放。
- 提供成本与轮次计量：token、调用次数、时长、轮次上限可配置且被记录。
- 缺失任一项时，在评测报告中声明缺失项与影响，不得静默降级。

## 上下文加载顺序（Context Budget）

1. 评测任务契约：被测对象、协议参数、重复次数、成本上限（固定，不压缩）
2. 基准版本说明：任务数、子集划分、修订记录、当前状态（在用/修订/弃用）
3. 统一 scaffold 与判定器说明
4. 历史分数库：同基准历史成绩与已知方差区间
5. 污染检测规程与历史检测结果
6. 任务环境与镜像清单（按需加载）

## 工具契约

- 环境构建工具：镜像构建必须可复现，digest 被记录；禁止在运行时动态修改镜像。
- 判定器：输入为智能体产出与环境终态，输出为二元判定；判定逻辑版本化。
- 轨迹存档工具：保存完整 rollout 轨迹，支持回放校验。
- 方差分析工具：支持同配置重复运行与区间计算（均值 ± 离散度）。
- 所有工具输出不得包含 gold patch 或判定细节给被测智能体。

## 任务执行流程（SOP）

1. 定义构念：该基准测的是什么能力，边界在哪里，不测什么。
2. 构建任务：每任务配唯一环境、人工编写的参考解、全面的验证测试；问题陈述不得引用题面之外的信息。
3. 质量控制四阶段：自动化验证（参考解成功、占位解失败）→ LLM 辅助审阅（歧义、作弊机会、描述与测试不一致）→ 人类专家逐任务审阅 → 对抗测试（作弊智能体攻击漏洞）。
4. 固定协议：scaffold 版本、推理努力度、步数/成本上限、重复次数、判定方式，全部写入协议文档并版本化。
5. 抗污染设计：采用 held-out 子集、动态更新（如最新 issue 走 test split）、私有仓库屏障；记录任务进入基准的日期以支持污染审计。
6. 运行评测：每配置按协议重复运行，报告均值与区间；记录每次运行的环境 digest 与配置快照。
7. 提交验证：审查 rollout 轨迹，确认仅问题陈述与镜像被提供给智能体；验证通过方标注 Verified。
8. 发布与口径声明：公布分数时同时公布协议版本、重复次数、方差区间与已知的口径限制。

## 验证与证据要求

必须满足并留存证据：

| 验证项 | 判定方式 | 证据 |
|---|---|---|
| 判定器有效 | 参考解通过、占位解失败，逐任务实测 | 验证记录 |
| 环境可复现 | 同镜像同配置重复运行结果在方差区间内 | 重复运行记录 |
| 无泄漏 | 智能体无法从环境中获得 gold patch 或判定细节 | 隔离测试 + 轨迹审查 |
| 无作弊提分 | 对抗测试通过；PR Hacks 类行为有检测与惩罚 | 对抗测试报告 |
| 方差已量化 | 同配置至少多次运行并报告区间 | 方差分析记录 |
| 口径可追溯 | 每个分数对应协议版本、scaffold 版本、日期 | 协议台账 |
| 污染可审计 | 任务入集日期可查，定期做复现检测 | 污染检测记录 |

禁止以单次运行成绩作为榜单数据；禁止引用未经验证的提交进入官方口径。

## 失败与升级策略

| 失败 | 处理 |
|---|---|
| 发现缺陷测试（过窄/过宽） | 冻结该任务，走修订流程；修订记录公告；涉及历史分数的须标注口径变化 |
| 发现训练污染证据 | 检测扩大化；污染任务下架或标注；公布检测结果 |
| 发现判分作弊 | 撤销 Verified 标注；按协议施惩罚分；作弊模式纳入对抗测试用例 |
| 环境漂移（镜像依赖失效） | 重建镜像并重新验证全部任务；历史分数标注环境版本差异 |
| 两评测环境分数系统性不一致 | 双榜并列发布并注明差异来源；禁止单取其一 |

升级三要素：现象与证据、影响面（哪些任务/分数受影响）、建议选项。

## 安全与合规红线

1. gold patch、判定细节与 held-out 子集对被测智能体严格保密。
2. 商业子集仓库的使用遵守其许可与访问协议；GPL copyleft 抽取须留法律审查记录。
3. 榜单数据不得为任何被测方定向修改；口径调整须公告且历史数据保留原值。
4. 厂商自述成绩与经提交验证的成绩必须区分展示；自述数字标注 self-reported。
5. 基准弃用或重大修订时，须同步更新所有下游引用口径。

## 禁止事项

1. 禁止横向比较不同基准、不同 scaffold、不同推理努力度配置下的分数。
2. 禁止引用不带评测日期与配置的分数。
3. 禁止把训练污染未明的基准当作能力结论依据。
4. 禁止把单次运行分数写入榜单。
5. 禁止为榜单好看而调整判定器。
6. 禁止使用 `[待填写]`、`XX`、`___` 之外的非标准占位符；不确定处统一使用 `[待填写]`。

## 输出格式

- 协议文档：构念定义、任务规格、scaffold 与判定器版本、重复次数、成本上限
- 分数报告：均值 ± 区间、配置快照、评测日期、Verified 状态
- 口径说明：官方榜/第三方 scaffold 榜/独立机构榜的区分与不可比性声明
- 污染检测报告：检测方法、范围、结论
- 基准修订记录：版本、变更内容、对历史分数口径的影响

## 评估与自检

| 自检项 | 判定 |
|---|---|
| 构念定义是否明确（测什么、不测什么） | 是 / 否 |
| 判定器是否经参考解/占位解双向验证 | 是 / 否 |
| 环境是否可复现且 digest 被记录 | 是 / 否 |
| gold patch 与判定细节是否对被测方隔离 | 是 / 否 |
| 方差是否量化并随分数发布 | 是 / 否 |
| 口径声明是否完整（协议版本、scaffold、日期） | 是 / 否 |
| 污染检测是否有记录且定期执行 | 是 / 否 |
| 作弊检测与惩罚机制是否实测生效 | 是 / 否 |

任一"否"须显式列出；自检不通过即任务未完成。

4.2. SKILL.md Specification

---
name: benchmark-score-audit
description: 对一份基准分数或榜单结论做方法学审查与合规引用：核查四要素（模型版本、基准、评测日期、配置），判定口径可比性（官方榜/第三方 scaffold 榜/独立机构榜），识别训练污染与判分作弊风险，并输出可辩护的引用文本。当需要在文档、汇报或选型中引用基准分数，或对两份榜单数字的差异做归因时使用。
version: 1.0
created: 2026-09-12
---

# 基准分数审查与合规引用

## 适用场景

- 需要在文档、汇报或选型材料中引用公开基准分数。
- 两份榜单（或同一榜单两个日期/配置）的数字出现差异，需要判断是能力差异还是口径差异。
- 收到厂商的 benchmark 声明，需要判断其可信度与引用条件。
- 需要为团队制定分数引用规范。

不适用于：构建企业内部回归评估集（属 05-eval.md 范畴）、基准任务的日常执行。

## 前置条件

| 类别 | 要求 | 缺失时处置 |
|---|---|---|
| 分数来源 | 可指认的原始出处（官方榜、论文、官方博客） | 停止引用，先溯源 |
| 配置信息 | 推理努力度、scaffold、重复次数至少其一已知 | 标注"配置未明"，禁止用于横向比较 |
| 基准状态 | 该基准当前在用/修订/弃用状态已知 | 查证基准官方站与弃用公告 |
| 比较意图 | 明确比较对象与目的 | 无明确目的时不做跨榜比较 |

## 输入

- 待审查的分数声明（数字、模型、基准、日期、来源）
- 相关基准的官方协议文档与榜单页
- 涉事模型/产品的官方发布材料（用于交叉核对）

## 输出

- 分数审查卡：四要素核查结果、口径判定、可信度等级（官方提交验证 / 官方榜未验证 / 厂商自述 / 第三方汇编）
- 可比性结论：哪些数字可以同表呈现、哪些必须并列注明、哪些禁止比较
- 引用文本：合规的分数引用写法（含日期、配置与状态声明）
- 风险清单：污染、作弊、缺陷测试、口径漂移等已识别风险

## 执行步骤

1. **四要素核查**：模型名 + 版本、基准名 + 版本、评测日期、配置（推理努力度 / scaffold / 重复次数）。任一缺失即标注"不可用于比较"。
2. **来源分级**：官方榜且经提交验证（Verified）> 官方榜未验证 > 厂商自述（self-reported）> 第三方汇编。厂商自述须与独立榜对照，差异显著者标注 [待核实] 并双口径并列。
3. **基准状态核查**：查证基准是否已被官方弃用（如 SWE-bench Verified 于 2026-02-23 被弃用）；已弃用基准的分数引用必须附带弃用事实与理由。
4. **口径可比性判定**：
   - 同基准 + 同 scaffold + 同配置 + 同日期 → 可直接比较；
   - 同基准 + 不同 scaffold / 配置 → 只能并列注明，不可排序；
   - 不同基准 → 禁止比较；跨榜引用只能作为各自独立的证据。
5. **污染风险评估**：该基准是否有官方或第三方的污染审计？分数提升曲线是否与其发布时间线的暴露规律吻合？有已知污染证据（如全前沿模型可复现 gold patch）的分数，引用时必须附带污染声明。
6. **方差判断**：两个分数的差异是否小于已知的 run-to-run 方差（如同配置 ±3 分量级）？在方差区间内的差异应表述为"无可辨别差异"而非优劣结论。
7. **作弊与缺陷检查**：该成绩是否经 Verified 提交验证？是否存在 PR Hacks 惩罚记录？该基准是否已有缺陷测试的官方审计（如 ≥59.4% 缺陷测试比例）？
8. **产出引用文本**：按"模型版本 + 基准 + 分数 + 评测日期 + 配置 + 来源属性"的固定格式输出；结论句注明"基于 [基准状态] 的 [口径] 数据"。

## 质量标准（DoD）

1. 每个被引用的分数四要素齐全，缺项被显式标注。
2. 来源属性（官方验证 / 官方榜 / 厂商自述 / 第三方汇编）被标注。
3. 基准当前状态（在用 / 修订 / 弃用）被核查并写入引用文本。
4. 口径可比性结论明确：可比较 / 并列注明 / 禁止比较三档。
5. 已知污染与缺陷测试的官方审计结论未被遗漏。
6. 方差区间内的差异未被表述为优劣结论。
7. 厂商自述与独立验证的数字未被混用排序。
8. 输出的引用文本可直接复制使用，且无需读者再做口径判断。

## 常见失败与处理

| 失败 | 表现 | 处理 |
|---|---|---|
| 跨榜比较 | "A 模型在 X 榜 80 分高于 B 模型在 Y 榜 78 分" | 判定为口径错误；改为各榜独立陈述 |
| 配置遗漏 | 分数未注明 xhigh 等推理档位 | 补查原始发布；查不到则标注"配置未明"并降级引用 |
| 时效错位 | 用已弃用基准的分数论证当下能力 | 附弃用事实；改用继任基准或最新独立评测 |
| 榜首之争 | 两个口径的"最高分"冲突 | 并列呈现 + 注明口径差异来源 |
| 厂商自述当实测 | 自述分数与官方榜并列排序 | 拆分展示，自述注明 self-reported 与 [待核实] |
| 方差误读 | 差异小于 ±3 分被解读为代际进步 | 改述为"落在运行方差内，无可辨别差异" |
| 污染失察 | 引用已被官方审计出缺陷测试的分数而无声明 | 补充审计结论引用 |

## 示例

**场景**：选型报告需要引用两个编码智能体的最新能力证据。

- 待审数字甲："Claude Opus 4.5，SWE-bench Pro 55.53%，2025-11-26，官方榜（统一 SWE-Agent scaffold）"——四要素齐全，来源为官方榜，基准在用 → 可直接引用。
- 待审数字乙："Claude Opus 4.7，SWE-bench Verified 87.6%，2026-04-16"——基准已于 2026-02-23 被弃用，且为第三方汇编口径 → 引用时必须附弃用声明与 [待核实] 标注，且不得与甲排序比较。
- 待审数字丙："某厂商自述其产品内部评测达成同等水平"——无公开榜单与提交验证 → 仅可作线索引用，标注 self-reported 与 [待核实]，不得进入能力对比表。
- 结论：报告采用甲为主证据，乙作为"历史口径参考"附弃用声明，丙降级为背景叙述。

4.3. Implementation Checklist

Citation and dissemination

• [ ] Every cited score has the four elements: model + version, benchmark, evaluation date, configuration
• [ ] The reasoning-effort tier (medium / high / xhigh / max) is noted alongside the score
• [ ] The benchmark's current status (in use / revised / deprecated) has been checked, and citations of deprecated benchmarks include the deprecation fact
• [ ] The three calibers — official leaderboard, third-party scaffold leaderboard, independent organization leaderboard — are not mixed
• [ ] Vendor self-reported figures are marked self-reported and not mixed with measured figures
• [ ] Differences falling within the run-to-run variance interval are not expressed as superiority conclusions
• [ ] Known contamination-audit findings (e.g., gold patch reproduction) are presented alongside score citations

Evaluation construction

• [ ] The benchmark's construct is clearly defined: what capability is measured and what is not
• [ ] Each task has a unique environment, a human-written reference solution, and comprehensive verification tests
• [ ] The four-stage quality control (automated verification / LLM review / expert review / adversarial testing) has been executed and recorded
• [ ] The unified scaffold version is pinned and written into the protocol document
• [ ] Protocol parameters (number of repetitions, cost and round caps, reasoning effort) are versioned
• [ ] Each configuration is run multiple times, and scores are published as mean ± interval
• [ ] The environment image digest is recorded, and reruns are reproducible

Governance and anti-cheating

• [ ] The gold patch, judging details, and held-out subsets are strictly isolated from the party under test
• [ ] The submission-verification process is in effect: Verified can only be marked after trajectory review passes
• [ ] PR-Hacks-type cheating has detection means and penalty records (e.g., negative-score penalties)
• [ ] Flawed tests have a process channel for discovery, freeze, and revision
• [ ] Contamination detection runs periodically, and task entry dates are ascertainable
• [ ] The use of commercial / copyleft repositories has legal review records

Lifecycle

• [ ] Benchmark revision and deprecation have an announcement mechanism, and downstream citation calibers are updated in sync
• [ ] Historical scores retain their original values, and caliber adjustments are not retroactively rewritten
• [ ] The relationship between internal and public benchmarks is clear: public leaderboards are used for ceiling calibration, internal regression sets for capability conclusions

5. Summary

The question the Benchmark direction answers is: is the ruler we use to judge how strong models are itself accurate?

This direction has four core conclusions:

1. Public leaderboard scores are composite products, not pure capability values. Every score is the product of "model × Harness × configuration × run variance". Switching scaffolds for the same model can differ by 11.6 points (SWE-bench Pro: SWE-Agent 45.9% vs Auggie CLI 51.8% vs WarpGrep v2 57.5%), the variance of multiple runs of the same configuration is at the ±3-point magnitude, and the same benchmark drifts by several points across evaluation organizations (Terminal-Bench 2.1 official 83.8% vs independent evaluation 84.6%) — discussing model scores apart from the Harness and caliber is meaningless.
2. A benchmark is an asset that expires, and the deprecation announcement is part of its lifecycle. The official deprecation of SWE-bench Verified (2026-02-23, ≥59.4% flawed tests + all frontier models reproducing the gold patch) shows that construct validity and training contamination are not theoretical risks but facts that have already occurred. Before citing any benchmark score, first check whether it is still alive.
3. The direction of the new generation of benchmarks is "longer horizon + contamination resistance + unified scaffold + submission verification". SWE-bench Pro's private-repository barrier, SWE-bench-Live's dynamic updates and Windows compatibility findings, Terminal-Bench's four-stage quality control and Verified tag, and SWE-Lancer's economic valuation together outline the evolution path of evaluation protocols.
4. The ultimate basis for capability conclusions must be internal evaluation; public leaderboards serve only as ceiling calibration. This is precisely the solution to the failure path "only doing Benchmark, not Eval" described in this group's README.md: using the private regression set of 05-eval.md to answer "whether we do well", and using this direction's methodological discipline to ensure that "every number we cite is defensible".

The recommendation for implementation teams is: establish a score-citation convention (four elements + source attribute + benchmark status) and enforce it across the team; citations of vendor self-reports must be marked self-reported; cross-caliber numbers are always stated side by side and never ranked. Maintaining skepticism about evaluation methodology is not a denial of industry progress, but a way to keep real progress from being diluted by noise.

Information Gap Statement

1. Public AI SRE benchmarks: completely no results. This round of search found no public AI on-call / fault-location benchmark; Google's Golden Data evaluation system is an internal form.
2. Public DevOps / IaC agent benchmarks: a serious gap. Infrastructure/DevOps-type tasks account for only 0.41% (3 tasks) of the entire SWE-Lancer set; GitHub's Win-Hill is an internal benchmark. Public evaluation in the infrastructure direction has long been absent.
3. China's GB/T-type AI coding standards: not found. This document cites no domestic standard numbers, and any later additions must strictly avoid fabrication.
4. Items whose original text is pending and marked [To be verified] when cited: Microsoft's "Saving SWE-Bench" paper (the claim that "existing benchmarks overestimate agent capability by over 50%" is only relayed secondhand, with no arXiv number obtained); the three original papers FeatureBench / FEA-Bench / FeatBench (the descriptions of naming conflicts and each paper's flaws were not obtained from the originals); SWE-bench Verified's current top scores (Claude Opus 4.7's 87.6% and GPT-5.5's 88.7%, third-party compilation caliber, and the benchmark has been deprecated; this document presents them side by side only as historical-caliber reference).
5. After SWE-bench Verified's deprecation, the fact that Google DeepMind, Anthropic, Meta, and OpenAI each continue to cite the benchmark is partly based on third-party compilations (credibility C); the judgment of a dual-track reporting landscape is this document's analysis.
6. SWE-bench Pro's official unified-scaffold leaderboard and results that include third-party scaffolds are two calibers; this document takes the official leaderboard (Opus 4.5 55.53%, 2025-11-26) as authoritative, and the high third-party scaffold scores (e.g., 57.5%) are cited only as empirical evidence of the Scaffolding effect, not entering official-caliber comparison.
7. Domestic big-tech (Tencent / ByteDance / Alibaba) AI coding productivity figures are all from self-media sources (C level +); this document adopts none of them as hard data.
8. The "vendor self-report vs independent verification" case of OpenAI's self-reported GPT-6 Astra Terminal-Bench 4.0 result (57.7%) being inconsistent with that day's official leaderboard (top score 57.9%) is based on a third-party compilation (credibility C); this document only explains it in the information gaps and does not use it as hard evidence in the body.

6. References

1. Why we no longer evaluate SWE-bench Verified — OpenAI (Mia Glaese & Olivia Watkins), 2026-02-23. https://openai.com/index/why-we-no-longer-evaluate-swe-bench-verified/
2. SWE-bench Official Site and Leaderboards — SWE-bench team, 2023–2026. https://swe-bench.github.io/
3. SWE-bench Pro Official Site and Leaderboard — Scale AI (+ Princeton), 2025-09-21. https://scaleapi.github.io/SWE-bench_Pro-os/
4. Terminal-Bench Official Leaderboard (2.1) — Laude Institute et al. https://www.tbench.ai/leaderboard/terminal-bench/2.1
5. Terminal-Bench v2.1 Independent Evaluation — Artificial Analysis. https://artificialanalysis.ai/evaluations/terminalbench-v2-1
6. Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces — Merrill, M. A., et al., arXiv:2601.11868, 2026-01. https://arxiv.org/html/2601.11868v1
7. SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks — arXiv:2602.12670, 2026-02-13. https://arxiv.org/html/2602.12670v1
8. SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering? — OpenAI, arXiv:2502.12115. https://arxiv.org/html/2502.12115v4
9. SWE-bench-Live Official Leaderboard — Microsoft et al., arXiv:2505.23419. https://swe-bench-live.github.io/
10. Evaluating performance and efficiency of the GitHub Copilot agentic harness across models and tasks — GitHub Blog, 2026. https://github.blog/ai-and-ml/github-copilot/evaluating-performance-and-efficiency-of-the-github-copilot-agentic-harness-across-models-and-tasks/
11. Before You Score the Model, Score the Benchmark — Centre for Software Excellence, 2026. https://centre-for-software-excellence.github.io/?currentRoute=docs/blog/before-you-score-the-model-score-the-benchmark
12. How we built our multi-agent research system — Anthropic, 2025-06-13. https://www.anthropic.com/engineering/multi-agent-research-system
13. AI in SRE: How Google is Engineering the Future of Reliable Operations — Google SRE. https://sre.google/resources/practices-and-processes/ai-engineering-reliable-operations/
14. State of AI-assisted Software Development 2025 — DORA / Google Cloud, 2025-11-12. https://dora.dev/research/2025/dora-report/
15. Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity (randomized controlled trial) — METR, 2025-07-10. https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/
16. Stack Overflow 2025 Developer Survey — Stack Overflow, 2025-07-30. https://survey.stackoverflow.co/2025/