Benchmarks and Evaluation Methodology: When Harness Becomes an Evaluation Variable

1. Introduction: Why Benchmarks Must Be Rebuilt in the Harness Era

1.1. Background: The Triple Trust Crisis of Leaderboard Narratives

Since 2024, when SWE-bench Verified became the de facto industry standard for coding agents, "leaderboard scores" were once the universal currency for the industry to narrate model capability progress. By 2025–2026, this currency experienced a triple trust crisis:

• First: the tests themselves are defective. OpenAI published "Why we no longer evaluate SWE-bench Verified" on 2026-02-23, formally retiring the benchmark. Its review found that at least 59.4% of the problems models commonly fail have defective tests — either rejecting functionally correct submissions (tests too narrow) or requiring features never mentioned in the problem statement (tests too broad). The benchmark no longer tests "what it claims to test" (see section 4.1 for details).
• Second: training contamination. OpenAI's audit showed that all tested frontier models (including GPT-5.2, Claude Opus 4.5, Gemini 3 Flash) could reproduce verbatim fragments of the gold patch from the task ID; GPT-5.2's chain-of-thought trajectories even revealed knowledge of unspecified test requirements, implying the test patches themselves appeared in the training data. Score improvements increasingly reflect training-time exposure, not true capability.
• Third: criterion drift. The same model's score differences across different scaffolds (evaluation scaffolds / Harnesses), different reasoning efforts, and different evaluation organizations can exceed the differences between model generations. Horizontal comparison between leaderboards is largely ineffective (see section 3 for details).

The triple crisis together points to a structural fact: the object of agent evaluation has never been the "model" but the system composed of "model + Harness". When the Harness becomes an inseparable variable in evaluation, the leaderboard system built on the "pure model score" narrative must be rebuilt — this is precisely the reason this chapter stands as an independent chapter of the white paper.

1.2. Core Argument: The Harness Has Become an Evaluation Variable

This chapter's core argument is: talking about model scores without the Harness is meaningless. This argument is supported by three sets of verifiable facts and runs through the whole document:

1. Swapping the scaffold on the same model can differ by 11.6 points. On SWE-bench Pro, Claude Opus 4.5 via the SWE-Agent standardized scaffold is 45.9%, switching to Augment Code's Auggie CLI is 51.8%; Opus 4.6 + Morph's WarpGrep v2 search sub-agent reaches 57.5% (third-party scaffold criterion) — part of the difference does not come from a better model but from better orchestration (see section 3.1 for details).
2. Multiple runs of the same configuration have a variance of about ±3 points. GitHub's variance analysis on Terminal-Bench 2.0 ran each agent-model combination at least 5 times, characterized the dispersion with ±1σ, and concluded: the observed inter-harness differences fall within the run-to-run variance caused by model randomness (see section 3.2 for details).
3. The same benchmark's scores systematically drift across different evaluation organizations. On Terminal-Bench 2.1, the official leaderboard (tbench.ai) top score and the independent evaluation organization Artificial Analysis (Terminus 2 + e2b sandbox, pass@1 averaged over 3 repetitions) score differ by several points — the evaluation environment itself is part of the score (see section 3.3 for details).

The combined inference of the three sets of facts is: every score on a public leaderboard is a composite product of "model × Harness × configuration × run variance". Citing a score without noting these four variables is like citing an unexplainable number.

1.3. This Chapter's Positioning in the White Paper

This chapter answers two questions: "Is the ruler we use to judge model strength itself accurate?" (sections 2–5) and "Since the public ruler cannot be fully trusted, how should enterprises build their own measurement?" (section 6). The division of labor with other chapters of the white paper is:

• The L5 (Evaluation and Observability Layer) in Chapter 3's six-layer capability model defines the Harness's evaluation responsibility; this chapter unfolds from the evaluation methodology side — reviewing public benchmarks, pointing out industry evaluation gaps, and giving methodology recommendations for building your own benchmarks.
• Chapter 4 (Architecture and Engineering Practice) discusses how to build the Harness; this chapter discusses how to measure the construction results.
• The platform comparisons in Chapter 6 (Industry Landscape), whenever they involve scores, all observe the citation discipline established in this chapter: with evaluation date, configuration, and scaffold explanation, otherwise not comparable.

2. Inventory of Mainstream Benchmarks

2.1. Citation Discipline of the Inventory

This chapter's inventory follows two methodology baselines:

1. All scores must carry the evaluation date, configuration, and scaffold explanation, otherwise not comparable. All scores cited in this section note their criterion (official leaderboard / third-party scaffold leaderboard / independent organization leaderboard); those with unclear criteria are flagged as unverified and prohibited from horizontal ranking.
2. The benchmark's status (in use / revised / retired) must be declared together with the score. Scores from retired benchmarks can only serve as historical criterion references and must not be used to argue current capability.

2.2. Mainstream Benchmark Comparison Master Table

2.3. Comparison of Anti-Contamination Designs

The anti-contamination design is the key differentiator among benchmarks (this document's judgment):

2.4. The Cost and Efficiency Dimension

The cost and efficiency dimension is a systematic weakness of mainstream benchmarks (this document's judgment): nearly all use pass rate as the sole primary metric, with cost (token consumption, dollar spend) and latency only displayed as attachments on some leaderboards. On the Terminal-Bench 2.1 official leaderboard, the top score's cost ranges from $134.09 to $2,059.19 — the cost behind the same score can differ by an order of magnitude, and rankings without cost constraints systematically reward "burn money for points" configurations.

2.5. The Status of Each Benchmark

The status of each benchmark (in use / revised / retired) must be declared together with the score (this document's judgment): SWE-bench Verified has been retired (2026-02-23); SWE-bench Pro is the official recommended successor; SWE-bench-Live takes the dynamic-update route; Terminal-Bench 2.1/4.0 are under continuous revision; the others are in use. Any external statement based on benchmark scores should note the benchmark's current status.

3. Core Argument: The Harness Has Become an Evaluation Variable

图 3-1｜评测变量的四维分解：模型 × Harness × 配置 × 运行方差

数据来源：基于本章证据绘制的示意图（信息截止 2026-09-12）。

3.1. The Scaffolding Effect: Swapping the Scaffold on the Same Model Can Differ by 11.6 Points

The most direct evidence that "the Harness has become an evaluation variable" is the scaffolding effect (this document's judgment): the same model's score can differ by up to 11.6 points across different scaffolds. On SWE-bench Pro, Claude Opus 4.5 via the SWE-Agent standardized scaffold is 45.9%, switching to Augment Code's Auggie CLI is 51.8% (a 5.9-point difference); Opus 4.6 + Morph's WarpGrep v2 search sub-agent reaches 57.5% (an 11.6-point difference from the baseline) (third-party scaffold criterion, credibility C). Part of the difference does not come from a better model but from better orchestration.

3.2. Measurement Noise: Multiple Runs of the Same Configuration Have a Variance of About ±3 Points

The second evidence is measurement noise (this document's judgment): multiple runs of the same configuration have a variance of about ±3 points. GitHub's variance analysis on Terminal-Bench 2.0 ran each agent-model combination at least 5 times, characterized the dispersion with ±1σ, and concluded: the observed inter-harness differences fall within the run-to-run variance caused by model randomness. This means that differences smaller than ±3 points cannot be attributed to Harness differences — they are within the noise floor.

3.3. Criterion Drift: The Same Benchmark's Scores Systematically Drift Across Different Evaluation Organizations

The third evidence is criterion drift (this document's judgment): the same benchmark's scores systematically drift across different evaluation organizations. On Terminal-Bench 2.1, the official leaderboard (tbench.ai) top score is 83.8%, and the independent evaluation organization Artificial Analysis (Terminus 2 + e2b sandbox, pass@1 averaged over 3 repetitions) score is 84.6% — a 0.8-point difference. The evaluation environment itself is part of the score.

3.4. Combined Inference: Every Score Is a Composite Product

The combined inference of the three sets of evidence is (this document's judgment): every score on a public leaderboard is a composite product of "model × Harness × configuration × run variance". Citing a score without noting these four variables is like citing an unexplainable number. This is the core argument of this chapter and the methodological foundation for all subsequent discussions.

4. The Crisis of the Benchmarks Themselves

4.1. Defective Tests: At Least 59.4% of Frequently Failed Problems Have Defective Tests

OpenAI's "Why we no longer evaluate SWE-bench Verified" (2026-02-23, Tier A) formally retired the benchmark. Its review found that at least 59.4% of the problems models commonly fail have defective tests — either rejecting functionally correct submissions (tests too narrow) or requiring features never mentioned in the problem statement (tests too broad). The benchmark no longer tests "what it claims to test".

OpenAI continues to report Verified scores after the announcement, but begins to simultaneously give Pro numbers; OpenAI itself, when releasing GPT-5.5 on 2026-04-23, still used the 88.7% Verified score as the headline number (this landscape judgment partially relies on third-party compilations, credibility C). The current top score has two criteria, and this white paper presents them side by side and flags them, without making a choice:

The methodology lesson established by this case: benchmarks are perishable assets. Any external statement based on benchmark scores should note the benchmark's current status.

4.2. Measurement Noise: The Moving Ceiling

A more subtle problem than defective tests is the noise of the evaluation infrastructure itself. The Centre for Software Excellence's "Before You Score the Model, Score the Benchmark" (2026, Tier-C source, transparent methodology) gives a set of key measurements: in an isolated clean environment, 10 independent Oracle (gold patch) runs were performed on SWE-bench Verified — that is, no model solves the problems, the human reference answers are directly applied to run the tests:

The report's conclusion hits the nail on the head: "98.5% sounds high in a vacuum, but a moving ceiling means your evaluation metric is fundamentally noisy." When the standard answers themselves have only about 98.6% reproducibility, a 2-to-3-point difference between tested systems is already close to the measurement floor. This mutually corroborates with section 3.2's run-to-run variance (about ±3 points): the noise floor of agent evaluation is at the single-digit percentage-point level, and any conclusion smaller than that level requires repeated experiments to support it. Multiple public issues in the SWE-bench community (#267, #294, #274, #246, etc.) and the Harbor troubleshooting documentation record the pervasiveness of similar problems.

4.3. Construct Validity Crisis: The Benchmark Does Not Measure Real Work

The third-layer crisis is the most fundamental: even if the tests have no defects, no contamination, and no noise, the benchmark may still measure the wrong thing.

• Task form distortion. Microsoft's "Saving SWE-Bench" paper (only via second-hand paraphrase, arXiv number not obtained) points out: GitHub-issue-derived benchmarks fundamentally misrepresent the way developers actually use coding agents — real interactions mostly happen in informal chats in the IDE, not in the structured prose of archived issues. After the team rewrote SWE-Bench tasks into realistic chat-style queries, they found existing benchmarks overestimate agent capability by more than 50% (ibid.).
• Feature-implementation benchmark naming and quality crisis. FeatureBench / FEA-Bench / FeatBench — three different teams published nearly identically named feature-implementation benchmarks (original papers all not obtained), and each has defects: FeatureBench's problem statements are synthesized by an LLM from the test graph rather than from real PR descriptions, and the tasks feel artificial and extremely difficult — Claude Opus 4.5 is 80.9% on SWE-bench but only 11.0% on FeatureBench, the ceiling is so low that it mainly measures the floor noise; FEA-Bench (Microsoft)'s definition requires introducing new components, but 116 of 1,401 tasks have an empty new_components field. These new benchmarks try to fix SWE-bench's construct problems but expose the immaturity of the benchmark design itself.
• Cost and efficiency absence. Mainstream benchmarks almost all use pass rate as the sole primary metric, with cost (token consumption, dollar spend) and latency only displayed as attachments on some leaderboards. On the Terminal-Bench 2.1 official leaderboard, the top score's cost ranges from $134.09 to $2,059.19 — the cost behind the same score can differ by an order of magnitude, and rankings without cost constraints systematically reward "burn money for points" configurations.

The inference of the construct validity crisis is: between public benchmark scores and enterprise real scenarios lies the "evaluation—production gap". The benchmark only measures a small cross-section of real work, and that cross-section is still being reverse-shaped by training data.

4.4. Divergence Between Vendor Self-Reports and Independent Verification

Model vendors are both the tested and the narrators, and the scores in their release materials must be distinguished from independently verifiable leaderboard submissions. The norms that have formed in the industry include:

• Source tiering: official leaderboard and submission-verified (such as Terminal-Bench's Verified tag from 2026-08, requiring submitters to provide rollout trajectories, with maintainers verifying that only the problem statement and Docker image were provided to the agent, and the gold patch and judgment results were not leaked) > official leaderboard unverified > vendor self-report (self-reported) > third-party compilation.
• Cheating penalties: on the Terminal-Bench 2.1 leaderboard, a Cursor CLI + Grok 4.5 submission was penalized by -9.0% for PR Hacks (exploiting judgment rule loopholes to boost scores) — proving that cheating detection and penalty mechanisms are actually operating in top benchmarks.
• Contrast obligation between self-reports and leaderboards: if a vendor's self-reported score is inconsistent with the day's official leaderboard (such as the GPT-6 Astra case described earlier in section 4.4), the citer has the obligation to point out the difference rather than default to acceptance.

The minimum discipline adopted by this white paper: vendor self-reported numbers must be marked self-reported, and must not be mixed with measured numbers in the same comparison column.

4.5. Benchmarks Are Perishable Assets

Comprehending this chapter's evidence, benchmark management should be treated as an engineering problem isomorphic to software dependency management, with a complete lifecycle:

5. Industry Evaluation Gaps: More Troubling Than Wrong Benchmarks Is No Benchmark

5.1. Gap Map

This chapter systematically inventoried the whole industry's benchmark coverage during writing, with results in the table below. The gaps themselves are an important finding of this white paper: in the Harness six-layer capability model, L5 (Evaluation and Observability) is widely recognized as the core layer, but the benchmarks that serve as its public infrastructure are completely absent in multiple high-value directions.

5.2. AI SRE: Completely No Public Benchmark

The multi-round retrieval (including Chinese and English, three or more rounds of progressive retrieval) on which this white paper relies found no public, authoritative benchmark for evaluating AI fault localization, root cause analysis, or on-call capability. The only public large-scale evaluation practice in this field currently comes from Google SRE: its AI Operator agent runs on thousands of incidents, using internal Golden Data (human expert ideal responses) as the control, with LLM-as-a-Judge judging success and failure scenarios, and failure judgments are automatically converted into bug tickets with implementation plans — but the whole system is internal in form, and the Golden Data is not publicly disclosed.

The consequence of this gap is twofold: on the one hand, enterprises cannot use public benchmarks to compare each vendor's AI SRE capability before procurement, and vendors' MTTR improvement claims (mostly self-reported by preview customers) cannot be cross-verified; on the other hand, it conversely proves that evaluation in this field can only take the privatized route of "enterprise self-built Golden Set + LLM-as-a-Judge closed loop" (see section 6 for details).

5.3. DevOps and IaC: Severe Gap

Public evaluation in the infrastructure direction has long been absent. Quantifiable evidence: of all 1,488 tasks in SWE-Lancer, only 3 are Infrastructure/DevOps class (0.41%), and the Diamond set is 0; GitHub's Win-Hill, used to verify cross-operating-system generalization, is an internal benchmark and not open to the outside. Considering the special risks of IaC scenarios — Terraform state contains secrets and requires precise addressing, and agents running on unfounded knowledge may damage state or destroy stateful resources — this direction is precisely the one that most needs strict evaluation yet has the least public ruler.

5.4. Multi-Agent Collaboration: No Authoritative Public Benchmark

The top empirical evidence for multi-agent systems all comes from vendor internals: Anthropic's multi-agent research system is 90.2% higher than single-agent on its internal research eval (internal evaluation results, cannot be extrapolated as an industry general benchmark), and the accompanying variance attribution analysis (token usage alone explains 80% of the performance variance on BrowseComp) is also internal data. The public side only has limited coverage such as tau-bench, and has not yet formed an industry de facto standard like Terminal-Bench. Multi-agent collaboration is the core scenario of L3 (Orchestration and Control Layer), and its evaluation gap directly constrains the horizontal comparison and selection of orchestration technology.

5.5. The Meaning of the Gaps Themselves

Evaluation gaps are not just "missing convenience"; they have clear structural consequences:

1. Evaluation burden transfers to enterprises. In directions without public benchmarks, every procurement decision must build its own evaluation, otherwise it can only rely on vendor self-reports.
2. Vendor narratives lack checks and balances. Contrasted with section 4.4's source tiering, "completely no public benchmark" means that almost all effectiveness claims in that direction remain at the self-reported tier.
3. The Harness's L5 layer goes from "optional" to "admission requirement". When the public ruler is absent, the enterprise's own evaluation and observation capability (regression sets, trajectory tracking, online metrics) becomes the only credible measurement source — this is the direct motive for this white paper's section 6 methodology recommendations.

6. Methodology Recommendations for Building Your Own Benchmarks

6.1. General Principle: Public Leaderboards Only for Upper-Bound Calibration

The general principle of self-built evaluation is two-stage: use public benchmarks to estimate the model capability ceiling, and use internal regression sets to answer the actual level. The value of public leaderboards is in three aspects: coarse-grained capability tiering (who is in the first tier), methodology templates (the four-stage quality control, controlled experiment design, and anti-contamination design can all be directly borrowed), and hallucination detection (when vendor self-reports and independent leaderboards show systematic divergence, public leaderboards are the developer). But the final basis for capability conclusions must be internal evaluation — the reason is already given in section 4.3: there is a systematic gap between public benchmarks' constructs and enterprise production distributions.

6.2. Golden Set: Turn Ideal Answers into Assets

The Golden Set is a human-verified set of ideal responses, the baseline ground truth of internal evaluation. Methodology points:

6.3. Trajectory Evaluation: Judge Process and Results Separately

Input—output evaluation that only looks at the final answer has a systematic blind spot: correct answers hide wrong reasoning. An agent's hallucinated tool calls may happen to be lucky, retrieving irrelevant documents may not affect the final synthesis, and strategy following in multi-turn dialogue (whether identity was verified, whether interactions were recorded) is completely invisible in the final response. Trajectory evaluation takes the agent's execution path (tool calls, parameters, steps, loops, required steps) as an independent evaluation object, with three points:

1. Four-dimensional decomposition: trajectory (steps, loops, required steps complete), tool use (selection, parameters, error recovery), task completion (goal achieved), multi-turn stability (context retention, goal drift), four dimensions scored independently.
2. Result-oriented judgment: should not be judged as failure because the trajectory deviates from the reference path — an alternative path should be judged as passing when it is safe, correct, and not less efficient. This principle prevents the evaluator from misjudging "style differences" as "capability defects".
3. Paired with cost: task success rate must be reported simultaneously with per-task cost and latency — an agent can inflate success rate through endless retries, and Anthropic's variance attribution shows that the token budget itself can explain most of the performance variance.

6.4. The Boundaries and Calibration of LLM-as-a-Judge

LLM-as-a-Judge is a technology that internal evaluation cannot avoid — many semantic dimensions ("is this diagnosis correct" "is this text on point") cannot be judged by deterministic scripts. But it itself hallucinates, has style preferences, has preferences for verbose answers, and is self-contradictory across similar cases. The boundaries and calibration disciplines that have converged in engineering:

At the same time, the Judge's capability boundaries must be acknowledged: Anthropic's practice provides irreplaceable human evaluation empirical evidence — human testers found that early agents consistently prefer SEO-optimized content farms over authoritative sources (academic PDFs, personal blogs), which is a systematic blind spot that LLM Judges did not capture, and was only resolved after adding source quality heuristics. Judges do large-scale filtering, humans do high-risk adjudication and blind spot exploration, and the two are in a hierarchical relationship rather than a replacement relationship.

6.5. Benchmark Lifecycle Management: Treat Benchmarks as Dependencies

Comprehending this chapter's evidence, benchmark management should be treated as an engineering problem isomorphic to software dependency management, with a complete lifecycle:

The core idea of lifecycle management is: benchmarks are perishable assets, just like software dependencies. They need versioning, revision, contamination detection, retirement, and archiving. Any external statement based on benchmark scores should note the benchmark's current status.

6.6. The Minimum Discipline for Self-Built Evaluation

Based on this chapter's evidence, the minimum discipline for self-built evaluation is summarized as six items:

1. Two-stage principle: use public benchmarks to estimate the capability ceiling, and use internal regression sets to answer the actual level.
2. Golden Set assetization: start from 30 to 50 carefully annotated cases, prioritize production failure trajectories, cover nine types of samples, and ensure isolation requirements.
3. Trajectory evaluation: judge process and results separately, with result-oriented judgment, paired with cost.
4. LLM-as-a-Judge discipline: single-criterion principle, structured output, isolation requirements, human calibration, divergence handling, continuous recalibration.
5. Benchmark lifecycle management: treat benchmarks as dependencies, with versioning, revision, contamination detection, retirement, and archiving.
6. Citation discipline: all scores must carry the evaluation date, configuration, and scaffold explanation, otherwise not comparable; the benchmark's status must be declared together with the score.

6.7. The Division of Labor Between Public and Self-Built Benchmarks

The division of labor between public and self-built benchmarks is clear (this document's judgment): public benchmarks provide coarse-grained capability tiering, methodology templates, and hallucination detection; self-built benchmarks provide fine-grained actual level measurement, production distribution coverage, and continuous regression. The two are complementary, not substitutive. Enterprises should use public benchmarks for upper-bound calibration and self-built benchmarks for actual level measurement.

7. Summary

This chapter has established the core argument that the Harness has become an evaluation variable. Every score on a public leaderboard is a composite product of "model × Harness × configuration × run variance". Citing a score without noting these four variables is like citing an unexplainable number.

The triple trust crisis of leaderboard narratives (defective tests, training contamination, criterion drift) together points to a structural fact: the object of agent evaluation has never been the "model" but the system composed of "model + Harness". When the Harness becomes an inseparable variable in evaluation, the leaderboard system built on the "pure model score" narrative must be rebuilt.

This chapter has also pointed out the industry evaluation gaps (AI SRE, DevOps/IaC, multi-agent collaboration) and given methodology recommendations for building your own benchmarks (two-stage principle, Golden Set assetization, trajectory evaluation, LLM-as-a-Judge discipline, benchmark lifecycle management). The core idea is: benchmarks are perishable assets, just like software dependencies. They need versioning, revision, contamination detection, retirement, and archiving.

For the detailed landing practices of each industry group, see the corresponding chapters of this white paper. For the future evolution of the evaluation methodology, see Chapter 5 (Outlook).

Information Gap Statement

1. Section 2.2: the SWE-bench-Live first release date (2025-05) and publisher (Microsoft + partners) are per third-party compilation (credibility C), not officially confirmed.
2. Section 2.2: the SWE-Lancer first release date (2025-02) and task count (1,488) are per the original paper (Tier A), but the reward function details are not fully disclosed.
3. Section 3.1: the scaffolding effect data (Claude Opus 4.5 via SWE-Agent 45.9%, Auggie CLI 51.8%, WarpGrep v2 57.5%) are per third-party scaffold criterion (credibility C).
4. Section 3.3: the criterion drift data (Terminal-Bench 2.1 official leaderboard 83.8%, Artificial Analysis 84.6%) are per the official leaderboard and independent evaluation organization (Tier A / B).
5. Section 4.1: the SWE-bench Verified defective test rate (59.4%) is per OpenAI's official review (Tier A).
6. Section 4.2: the measurement noise data (10 independent Oracle runs, 98.56% average pass rate) are per the Centre for Software Excellence's report (Tier C, transparent methodology).
7. Section 4.3: Microsoft's "Saving SWE-Bench" paper (only via second-hand paraphrase, arXiv number not obtained) claims existing benchmarks overestimate agent capability by more than 50%.
8. Section 5.2: Google SRE's AI Operator agent (runs on thousands of incidents, internal Golden Data, LLM-as-a-Judge) is per Google SRE's blog (Tier A), but the Golden Data is not publicly disclosed.
9. Section 5.4: Anthropic's multi-agent research system (90.2% higher than single-agent on internal research eval) is per Anthropic's official blog (Tier A), but it is an internal evaluation result and cannot be extrapolated as an industry general benchmark.
10. Section 6.4: the LLM-as-a-Judge human calibration threshold (about 80% consistency) comes from vendor methodology documents (engineering experience criterion rather than standard provisions).

8. References

1. Why we no longer evaluate SWE-bench Verified — OpenAI, 2026-02-23. https://openai.com/index/why-we-no-longer-evaluate-swe-bench-verified/
2. Before You Score the Model, Score the Benchmark — Centre for Software Excellence, 2026. https://centre-software-excelence.com/blog/before-you-score-the-model-score-the-benchmark
3. Saving SWE-Bench — Microsoft, 2025 (only via second-hand paraphrase, arXiv number not obtained). https://arxiv.org/abs/2505.16214
4. SWE-bench Verified — Princeton / community, 2024-08. https://www.swebench.com/
5. SWE-bench Pro — Scale AI (+ Princeton), 2025-09-21. https://www.swebench.com/pro
6. Terminal-Bench — Laude Institute (+ partners), 2025-11. https://www.tbench.ai/
7. OSWorld / OSWorld-Hard — community, 2024 / 2025. https://os-world.github.io/
8. SkillsBench — community, 2026-02-13. https://skillsbench.com/
9. ARC-AGI-2 — FutureSearch, 2025. https://arcprize.org/arc-agi/2
10. GPQA Diamond — community, 2023. https://gpqa.ai/
11. tau-bench — community, 2024. https://tau-bench.com/
12. Artificial Analysis — independent evaluation organization, 2026. https://artificialanalysis.ai/
13. Google SRE: AI Engineering for Reliable Operations — Google SRE, 2025. https://sre.google/resources/practices-and-processes/ai-engineering-reliable-operations/
14. Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity (randomized controlled trial) — METR, 2025-07-10. https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/
15. State of AI-assisted Software Development 2025 — DORA / Google Cloud, 2025-11-12. https://dora.dev/research/2025/dora-report/
16. Stack Overflow 2025 Developer Survey — Stack Overflow, 2025-07-30. https://survey.stackoverflow.co/2025/