Coding (AI Coding)

1. Introduction

1.1. Background: From Inline Completion to Repository-level Understanding

AI coding is the scenario where AI has penetrated software engineering most deeply and where commercialization is most mature. Its evolution can be clearly divided into three stages:

• Stage 1 · Inline Completion (approx. 2021-2023): The model predicts the next block of code based on several lines before and after the cursor. The capability boundary is "the current file + several context lines", the output is "suggested text", and whether to accept it depends entirely on the developer.
• Stage 2 · Conversational Generation (approx. 2023-2024): The model generates functions, classes, scripts, and even entire files in a dialog box. The output grows larger, but lacks a verification loop — whether the generated code compiles or breaks existing behavior is entirely up to a human.
• Stage 3 · Repository-level Agentic Coding (2024-present): The model is given the ability to read the repository, run builds, run tests, and modify multiple files; the output changes from "suggestion" to a "changeset (Patch)". The essential change in this stage is not that models got stronger, but that the Harness took shape: context engineering determines what it sees, tools and sandboxes determine what it can do, and verification gateways determine whether its output can be trusted.

The emergence of Stage 3 turns "AI coding" from an editor feature into an engineering system problem. This is precisely the significance of this direction within the AI Harness system.

1.2. Definition

Coding direction (AI Coding): an engineering field that researches and practices using AI as the primary agent to complete code completion, code generation, refactoring, code review, and repository-level understanding, and to make its output meet engineering acceptance criteria.

This direction covers five types of tasks:

1.3. Positioning in the AI Harness System

图 1-1｜Coding 方向在 Harness 六层能力模型中的定位

数据来源：基于本文分析绘制的示意图。

This direction is the shared main battlefield of layer L1 Context Engineering and L2 Tools and Execution among the six layers.

1.4. The Bottleneck Layer

The bottleneck of this direction lies in the L1 Context Engineering layer, not the commonly assumed "model coding capability". Three pieces of evidence support this judgment:

1. The codebase size far exceeds the context window. Real business repositories often have hundreds of thousands to tens of millions of lines, so "letting the model read the entire repository" is not engineering-feasible. What determines success is the assembly strategy: which 0.1% of the code the model is shown.
2. Wrong context is more dangerous than missing context. Missing context makes the model say "I don't know"; wrong context (e.g., retrieving a deleted legacy interface) makes the model confidently write code that fails to compile or behaves incorrectly.
3. Long tasks suffer significant context decay. DORA 2025 (2025-11-12) shows that 30% of AI tasks take more than 4 hours each, and the quality of context management and compaction for long tasks directly determines the final output.

The second bottleneck lies in the L2 Tools and Execution layer: the model must be able to execute code in order to verify code, and execution inherently carries destructive potential, so the design quality of the sandbox and permission model determines whether this direction can be deployed safely.

1.5. Context Engineering: The First Constraint of This Direction

Represented by Claude Code's instruction-file mechanism (CLAUDE.md), the industry has developed a reusable layered context scheme. The empirical value of this mechanism is that it turns "project conventions" from conversation content into repository assets — conventions are versioned, overridable by proximity, and reviewable by the team.

Table 1-1　The five configuration locations of instruction files (priority from low to high)

Supporting mechanisms:

• Closest Wins: When multiple instruction files are hit at the same time, the one nearest to the file being edited takes priority; on conflicts at the same level, the more specific one prevails.
• @import references: An instruction file can use the @path syntax to reference other files, avoiding copying the same constraint in multiple places. Copying and pasting leads to clause drift — changing one place while forgetting another — which is the most common team-level failure mode.
• No mandatory fields: Instruction files are plain Markdown and do not require specific fields; this lowers the adoption barrier, but it also means the team needs to agree on a skeleton itself.

Note: For the concrete implementation of the above mechanisms, refer to each product's official documentation; AGENTS.md, as the open-source community de-facto standard (hosted by the AAIF under the Linux Foundation since 2025-12-09, and adopted in 60,000+ repositories), uses the same "plain Markdown + closest-wins" philosophy.

1.6. Value

• Turns coding from an individual skill into an organizational capability: Once conventions are written into repository instruction files, newcomers and agents follow the same set of rules.
• Moves the quality gateway forward: Generate-and-verify — defects are intercepted before they reach review.
• Frees review resources for high-value work: Mechanical checks (conventions, bad smells, known vulnerability patterns) are delegated to agents, so human review focuses on design and business correctness.
• Builds predictability: This is the fundamental value of AI Harness — model output is uncertain, but the gateways are certain.

2. Glossary

3. Case Studies

Note: The following case studies are compiled from public, verifiable primary sources. All effectiveness figures state the source party's measurement scope, evaluation date, and configuration; self-reported and measured scopes are labeled separately and must not be mixed.

3.1. Case 1: Measured Effectiveness of an Enterprise AI Coding Assistant

Source: GitHub official blog (enterprise effectiveness research), https://github.blog/

3.1.1. Background

AI coding tools are spreading rapidly among individual developers, but the question facing enterprise decision-makers is more specific: is the investment worthwhile? In which stage is the impact realized? Between 2024 and 2025, discussion of this question long remained at the subjective level of "developers say it is faster", lacking comparable organization-level evidence.

3.1.2. Approach

GitHub conducted effectiveness research around Copilot Enterprise, anchoring measurement on observable development behaviors rather than subjective questionnaires:

• Merge Rate: the proportion of submitted code changes ultimately merged into the mainline — a metric closer to true value than "how much code was generated".
• Review Time: the time from submission to completion of review.
• Task Time: the end-to-end time to complete a task within the IDE.
• It uses a controlled design (usage group vs. baseline group) rather than before/after comparison, to rule out the influence of seasonality and requirement difficulty.

3.1.3. Results

Interpretation and cautions:

1. "15x faster" is a ratio of stage-level time and is very easily misread as overall R&D being 15x faster. A stage-level speedup may be offset by rising costs in downstream stages.
2. The merge-rate improvement is a more convincing metric, but what it improves may be "commit quality" rather than "generation speed".
3. The above are vendor-published figures. When deploying in an organization, you should establish an internal control group (see 05-eval.md) and rely on end-to-end metrics (deployment frequency, change lead time, change failure rate, time to restore service).

3.2. Case 2: AI-ification of the Code Review Stage

Source: Tencent CodeBuddy public practice (2025-08); Alibaba Group public technical practice (2025-09)

3.2.1. Background

Code review is a key bottleneck in the development workflow. It simultaneously carries three kinds of checks: mechanical checks (style, conventions, known vulnerability patterns), correctness checks (whether the logic matches the requirements), and design checks (whether the structure is sound). Human reviewers' time is largely consumed by the first kind, yet the second and third kinds are precisely where human judgment is most needed.

3.2.2. Approach

The review is split into two layers, handled by different mechanisms:

• AI layer (mechanical checks moved forward): Before human review, the agent completes style and convention checks, known defect-pattern scanning, dependency risk alerts, change impact-scope mapping, and test-coverage hints. It produces structured review comments (file, line number, category, suggestion, rationale).
• Human layer (correctness and design): Reviewers only handle the parts AI cannot determine — business-logic correctness, interface-design soundness, and consistency with product intent.
• Key engineering constraint: AI review comments must be traceable to a specific rule or execution result (e.g., lint rule number, static-analysis rule number, test exit code); unsupported subjective evaluation is prohibited.
• Context assembly: The review agent loads the changeset + the nearest-wins instruction files + the symbol index of relevant modules, rather than the entire repository.

3.2.3. Results

Tencent CodeBuddy's public practice (2025-08) reports a Code Review time reduction of over 80%. Improvements of this magnitude come from "mechanical checks being automated and moved forward", not from "review being omitted".

Risk alert: The review speedup carries a hidden cost. When the volume of AI-generated code increases significantly, the total volume of review rises in tandem; if defects introduced by AI-generated code have a higher density than human-written code, "faster review" may correspond to "a higher defect escape rate". Therefore, the correct way to deploy this case study is: as review time drops, the defect escape rate must not rise — a two-metric constraint that must be observed simultaneously.

3.3. Case 3: Defect-Fixing Capability Crossing the 80% Threshold

Source: SWE-bench Verified official leaderboard, https://www.swebench.com/

3.3.1. Background

SWE-bench Verified is the authoritative benchmark for "real open-source repository defect fixing": given a repository and a problem description, the agent must locate and fix the defect so that the specified failing test passes. It is currently the most-cited public number for measuring agentic coding capability.

3.3.2. Approach (evaluation scope)

The benchmark's judgment is executable: success is determined when a test flips from failing to passing. This design avoids subjective scoring, but it also means:

• the score reflects "the fix success rate under a given repository subset, a given problem description, and a given tool configuration";
• there is a distribution gap relative to "whether it works on your company's codebase";
• scores under different inference configurations (e.g., xhigh) are not directly comparable.

3.3.3. Results

Table 3-1　Representative SWE-bench Verified results (all must carry evaluation date and configuration references)

Table 3-2　Comparison of adjacent benchmarks (illustrating cross-leaderboard incomparability)

Interpretation: The same model family can reach 80% on Verified but only 55.53% on Pro. The gap is not "the model got dumber", but a different distribution of task difficulty. This directly shows that public leaderboards can only be used to calibrate the capability ceiling, not to predict internal gains; any practice of writing leaderboard numbers directly into ROI calculations is a measurement-scope error.

4. Practical Standards

4.1. AGENTS.md Specification

The following is the complete original AGENTS.md for this direction, placed in a code block for direct copying. When used, it must be layered on top of the common clauses of the group-level AGENTS.md, with conflicts resolved by the closest-wins principle.

# AGENTS.md —— 软件工程 · Coding 方向（AI 编码）

## 角色与边界

- 本文件约束**在本仓库内做编码工作的 AI 智能体**：补全、生成、重构、代码审查、仓库级理解。
- 允许自行完成：读代码、改工作区文件、在沙箱内跑构建与测试、跑 lint/格式化/静态分析、生成补丁与说明、生成测试用例。
- 禁止自行完成：git push、创建或合并 PR、修改 CI 门禁配置、安装来源不明的依赖、执行生产环境命令、读写密钥。
- 边界判定：涉及**共享状态**或**不可逆**的动作须人工确认；涉及公共接口变更的须人工评审。
- 验收锚点：可编译、可测试、可部署、可回滚。

## 环境假设

- 提供仓库读写（工作区与分支隔离）、构建与测试执行器、lint/格式化/静态分析工具链、包管理。
- 提供容器或微虚机沙箱；**一切构建与测试必须在沙箱内执行**，禁止在宿主直接运行生成代码。
- 提供权威时间源与稳定的 commit SHA 用于变更定位与回滚。
- 提供 Trace 采集（工具调用、文件变更、命令输出摘要）。
- 提供成本计量（Token / 调用 / 时长）与上限设置。
- 缺失任一项时降级执行并显式声明。

## 上下文加载顺序（Context Budget）

1. 任务契约：目标、产出物、验收标准（固定，不压缩）
2. 就近指令文件：当前目录及其祖先链（五个配置位置，就近优先）
3. 仓库地图：目录结构与模块边界（**符号级，非全文**）
4. 相关源码：按"定义—引用"图检索，限影响面内
5. 测试与构建配置：测试入口、构建命令、门禁规则
6. 历史证据：相关近期提交、已知问题（仅摘要）
7. 工具结果：按需追加，超限即裁剪

每一步记入 Token 预算表。超限执行**压缩而非截断**：保留决策依据与约束，丢弃已完成的中间过程；压缩后须明示被移除的类别与被保留的约束清单。

事实来源优先级：可执行的构建/测试结果 > 代码实际行为 > 就近指令文件 > 文档注释 > 模型先验。禁止以模型先验覆盖仓库事实。

## 工具契约

- 副作用分级：R（只读）/ W（工作区写）/ X（沙箱内执行）/ S（共享状态写，需确认）/ D（破坏性，默认拒绝）。
- 输出体积上限：单次工具返回超限时必须截断并给出定位信息（路径 + 行号区间），禁止把完整大输出灌入上下文。
- 非幂等工具重试前必须先确认上次是否已生效。
- 并行子任务必须按文件或模块做**写分区**，禁止并发写同一文件。
- 所有执行类工具必须设置超时。

## 任务执行流程（SOP）

1. 解析契约：明确产出物与验收标准；缺失则停止并澄清。
2. 定位影响面：输出涉及模块、文件、接口、测试的清单。
3. 制定计划：步骤 + 每步验证方式 + 风险点 + 回滚方式（**第一个人工确认点**）。
4. 分步执行：每步完成后立即局部验证；单步失败不继续下一步。
5. 全量验证：见下节。
6. 最小化整理：清理临时文件、调试代码、敏感信息；无关格式化改动独立提交。
7. 交付：五段式变更说明，进入人工评审。
8. 留痕：归档 Trace、验证日志、覆盖率与静态分析报告。
9. 收尾：确认无遗留改动、无悬挂进程、无未清理凭据。

## 验证与证据要求

产物门禁（全部通过）：

| 门禁 | 判定 | 证据 |
|---|---|---|
| 可编译 | 构建退出码 0，无新增 error | 构建日志摘要 + 退出码 |
| 既有测试全绿 | 测试退出码 0，失败数 = 0 | 测试报告（总数/失败数/耗时） |
| 新增测试覆盖 | 新增逻辑有对应测试 | 新增用例清单 + 覆盖率增量 |
| 静态分析 | 无新增 error 级问题 | 分析报告（新增问题数 = 0） |
| 格式化 | 格式化检查通过 | 检查输出 |
| 依赖可追溯 | 锁定文件变更有说明与来源 | 依赖差异表 |

过程门禁（全部通过）：步数未超限 / 无越权尝试 / 无异常重试 / Trace 完整 / 变更集最小化。

**重构任务附加要求**：必须证明行为等价——在重构前后对同一组输入执行同一组测试（或特征测试），输出逐项比对一致；仅"测试通过"不足以证明等价。

**审查任务附加要求**：每条意见必须可溯源到具体规则编号或执行结果，禁止输出无依据的主观评价；须同时给出误报率自评。

禁止的"验证"方式：我认为没问题 / 逻辑上是对的 / 只跑了被修改的那一个测试 / 通过修改断言让测试变绿。

## 失败与升级策略

- 构建或测试失败：最多重试 3 次 → 回滚本地改动 → 上报并附失败证据。
- 工具超时：中断该步，记录状态，改用替代路径或上报。
- 权限被拦截：**不得尝试绕过**，记录并升级。
- 上下文超限：压缩；仍超限则拆分任务重新规划。
- 幻觉引用：以仓库事实为准，重新检索并上报冲突点。
- 需求不清：停止执行并提出澄清问题，禁止自行猜测。

升级三要素：已尝试动作与结果 / 卡点现象与证据 / 建议选项及风险。缺一无效。

## 安全与合规红线

1. 禁止强制推送、删除远程分支、重写已发布历史。
2. 禁止修改或绕过 CI 门禁、测试豁免、评审要求、签名校验。
3. 禁止在代码、配置、提交信息、日志中写入密钥、Token、内部域名、个人信息。
4. 禁止引入来源不明的依赖；新增依赖须说明来源、许可证与维护状态。
5. 禁止安装来源不明的包、执行网络下载脚本、关闭 TLS 校验、禁用安全扫描。
6. 遵循 NIST SP 800-218A（SSDF 1.1，2024-10）的供应链与构建完整性要求。
7. 代码质量度量参考 ISO/IEC 5059:2024；不得以"AI 生成"为由豁免既有质量标准。
8. 遵循 OWASP Top 10 for LLM Applications 2025（2025-11-10）对不安全输出的处理要求。

## 禁止事项

1. 禁止在无证据时声称"已验证""可以合并"。
2. 禁止为通过门禁而修改测试断言、放宽断言、添加豁免注释（除非显式列出并获批）。
3. 禁止跨出影响面清单修改文件（含"顺手"格式化、重命名、删注释）。
4. 禁止使用 `[待填写]`、`XX`、`___` 之外的非标准占位符；不确定处统一使用 `[待填写]`。
5. 禁止以"模型认为"作为技术判断依据。
6. 禁止在未被要求时扩展任务范围。
7. 禁止删除或改写他人的指令文件、门禁配置与安全策略。

## 输出格式

每次交付必须包含五段：

- 变更说明：文件清单 + 每文件一句话说明
- 为什么改：动因与依据（关联任务/缺陷/需求编号）
- 如何验证：执行的命令与结果（退出码与关键数字）、未验证项与原因
- 风险与影响：影响面、风险点（性能/兼容性/数据/安全）
- 回滚方案：具体命令或操作路径、回滚代价与注意事项

变更集最小化；多关注点拆分为多个提交，每提交只做一件事；提交信息说明"为什么"而非仅"做了什么"。

## 评估与自检

| 自检项 | 判定 |
|---|---|
| 影响面清单与实际变更文件一致 | 是 / 否 |
| 全部门禁通过且有证据 | 是 / 否 |
| 存在未验证项 | 无 / 有（列出） |
| 触及权限或安全红线 | 否 / 是（列出） |
| Trace 完整留存 | 是 / 否 |
| 变更集最小化 | 是 / 否 |
| 回滚方案可执行 | 是 / 否 |
| 输出五段齐全 | 是 / 否 |

任一"否"须显式列出；自检不通过即任务未完成。

度量口径：以端到端指标（部署频率、变更前置时间、变更失败率、服务恢复时间）为准；**禁止**以代码行数、补全采纳率或自评分数作为效果结论。

4.2. SKILL.md Specification

---
name: repo-grade-code-change
description: 在真实代码仓库中完成一项有验证门禁的编码变更——定位影响面、装配上下文、制定计划、分步执行、跑通全量门禁、输出五段式变更说明。适用于代码生成、重构、缺陷修复、多文件改动。当需要让 AI 产出能进入主干的变更集时使用。
version: 1.0
created: 2026-09-12
---

# 仓库级编码变更交付

## 适用场景

- 缺陷修复、特性实现、跨模块重构、依赖升级、测试补齐。
- 变更需要进入主干或目标环境，必须过门禁。
- 需要一份可评审、可回滚的变更集与说明。

不适用于：纯咨询性问答（"这段代码什么意思"）、一次性演示、不进入主干的临时实验。

## 前置条件

| 类别 | 要求 | 缺失时处置 |
|---|---|---|
| 任务契约 | 目标、产出物、验收标准 | 停止，先补全 |
| 仓库访问 | 可读 + 分支隔离工作区 | 降级为只读分析 |
| 构建与测试 | 命令可确定且可执行 | **无法进入 DoD，必须声明未验证** |
| 沙箱 | 构建/测试在容器或微虚机内 | 禁止在宿主执行；升级 |
| 权限模型 | 工具副作用分级 + 红线拦截 | 按最低权限执行 |
| Trace | 工具调用与文件变更可记录 | 降级并声明"不可回放" |

"构建与测试"与"沙箱"同时缺失时，本技能不适用。

## 输入

- 任务描述与验收标准
- 仓库名 + 分支 + commit SHA
- 影响面提示（模块/服务清单，可选）
- 相关证据（缺陷编号、日志片段、复现步骤）
- 预算上限（步数 / Token / 时长）

## 输出

- 变更集（补丁）+ 新增测试清单
- 验证证据：构建/测试/静态分析/覆盖率的退出码与关键数字
- 五段式变更说明
- 完整 Trace

## 执行步骤

1. **解析契约**：明确产出物与验收标准；缺失则澄清。
2. **装配上下文**：按固定七步顺序加载；记录 Token 预算表。
3. **定位影响面**：用"定义—引用"图确定涉及模块、文件、接口、测试，输出清单。
4. **制定计划**：步骤、每步验证方式、风险点、回滚方式；按风险等级触发人工确认。
5. **分步执行**：每步完成后立即局部验证；单步失败不继续。
6. **全量验证**：产物门禁六项 + 过程门禁五项，逐项留证据。
7. **最小化整理**：清理临时文件与调试代码；无关格式化独立提交；多关注点拆分提交。
8. **交付留痕**：五段式说明 + Trace + 验证日志归档。
9. **自检收尾**：八项自检；任一"否"显式列出。

## 质量标准（DoD）

1. 可编译：构建退出码 0，无新增 error。
2. 可测试：既有测试全绿；新增逻辑有新增测试；测试不是靠改断言变绿。
3. 可部署：配置与依赖变更已一并处理。
4. 可回滚：方案明确、可执行、不依赖智能体继续参与。
5. 可追溯：变更、验证证据、Trace、确认记录四者互相关联。
6. 可回归：已纳入回归集或有对等手工回归步骤。
7. 合规：未触及权限与安全红线。
8. 最小化：变更文件数与影响面清单一致。
9. 覆盖率不下降；无新增 lint error；公共接口变更有兼容说明。
10. 重构任务额外满足行为等价性证明。

## 常见失败与处理

| 失败 | 表现 | 处理 |
|---|---|---|
| 门禁伪造 | 改断言/加豁免让测试变绿 | 禁止；确需时显式列出并获批 |
| 上下文超限 | 约束被挤出 | 压缩而非截断；仍超限则拆分 |
| 幻觉引用 | 引用了已删除/改名的符号 | 以仓库事实为准，重新检索并上报 |
| 越权尝试 | 试图执行 D 级操作 | 不得绕过；记录并升级 |
| 范围蔓延 | "顺手"改了无关文件 | 回退越界改动，独立提交 |
| 循环不收敛 | 步数超预算无进展 | 强制终止 + 检查点 + 上报 |
| 覆盖率稀释 | 为覆盖率写无断言测试 | 判定为无效覆盖，重做 |
| 回滚失败 | 回滚依赖智能体继续参与 | D4 未通过，必须重做 |

## 示例

**场景**：修复一个跨模块缺陷——订单金额在优惠券叠加时计算错误。

- 契约：修复后 24 例既有测试全绿，新增覆盖 3 种叠加组合的测试；不改变公共方法签名。
- 影响面：`OrderService`、`CouponEngine`、2 处调用方、计价测试文件。
- 计划确认（L2 中风险）：修改计价顺序 + 抽出 `computeStackedDiscount`，保留旧方法为 deprecated 转发。
- 执行：改 2 个文件 → 跑计价测试 → 补齐新测试 → 跑全量。
- 验证：构建退出码 0；既有 24 例全绿；新增 3 例；覆盖率 +0.8%；lint 无新增 error。
- 风险：浮点精度在四舍五入路径上可能有 ±0.01 差异；已用定点小数改写消除。
- 回滚：`git revert <sha>`；旧方法转发保留一个发布周期。
- 自检：八项全"是"，可进入人工评审。

4.3. Deployment Checklist

Context Engineering (L1)

• [ ] Instruction files are placed in layers across the five configuration locations, and the closest-wins rule has been communicated within the team
• [ ] Common constraints use @import references, with no cross-file copy-and-paste (to prevent clause drift)
• [ ] The repository map relies mainly on symbol-level indexing, and full-text search is not used as the primary retrieval surface
• [ ] Context overflow is handled by compaction rather than truncation, preserving all constraints after compaction
• [ ] Each task records a Token budget table, explicitly stating the categories of content excluded
• [ ] "Hallucinated reference" detection is in place: retrieved symbols must be confirmed to exist on the current branch

Tools and Execution (L2)

• [ ] All builds and tests run inside a container or micro-VM sandbox
• [ ] Tool side effects are classified (R/W/X/S/D), and D-level is denied by default
• [ ] Tool output has a size cap; overflowing output is truncated with locating information
• [ ] All execution-type tools have a timeout set
• [ ] Parallel tasks use write partitioning by file/module

Verification Gateways (L5)

• [ ] The six gateways are all in place: build, existing tests, new tests, static analysis, formatting, and dependency traceability
• [ ] Gateways cannot be self-exempted by the agent (generation permission and gateway-modification permission are separated)
• [ ] Coverage is used as a gateway but paired with "effective coverage" spot checks, to prevent coverage for coverage's sake
• [ ] Refactoring tasks have a behavioral-equivalence proof process (before/after comparison, not merely tests passing)
• [ ] A regression suite exists, and the full suite runs on every model/prompt change

Governance and Security (L6)

• [ ] Secret scanning takes effect before commit (including commit messages and logs)
• [ ] New dependencies carry source, license, and maintenance-status explanations
• [ ] Actions such as forced push, branch deletion, and gateway modification are hard-blocked (hooks, not prompts)
• [ ] Production-environment commands are denied by default
• [ ] Audit logs and Trace cannot be altered or deleted by the agent

Measurement and Cognition

• [ ] Effectiveness conclusions are based on end-to-end metrics (the four DORA metrics), not lines of code or acceptance rate
• [ ] Self-reported and measured data are labeled by source attribute separately and not mixed
• [ ] Observation of the "verification time / generation time" ratio is established
• [ ] Citing public leaderboard figures includes all four elements: model + version, leaderboard name, evaluation date, and inference/tool configuration

5. Summary

The Coding direction is the domain where AI has penetrated software engineering most deeply and is also the easiest to misread.

The core conclusion of this direction is: the bottleneck of AI coding lies not in whether the model can write code, but in whether the team can determine that the code it writes is correct. This determination capability is provided by AI Harness — context engineering determines what the model sees, tools and sandboxes determine what it can do, gateways determine whether its output can be trusted, and Trace determines whether it can be reviewed afterwards.

Three judgments worth remembering:

1. L1 context engineering is the main battlefield. Wrong context is more dangerous than missing context; layered instructions + closest-wins + @import is currently the most mature engineering solution.
2. Public leaderboards cannot be used to predict internal gains. The gap between 80.8% on SWE-bench Verified and 55.53% on SWE-bench Pro reflects a difference in task distribution, not fluctuations in model capability.
3. A stage-level speedup is not an overall speedup. GitHub's 15x review speedup and Tencent CodeBuddy's ~80%+ review time reduction only constitute real gains under the premise that "the defect escape rate does not rise".

Finally, be alert to the perception gap: the METR randomized controlled trial (2025-07-10) shows that senior developers using AI were measured 19% slower, while DORA 2025 shows developers self-report productivity +80%. Until an internal objective measurement is established, any conclusion that "AI coding makes us faster" should be treated as an unverified hypothesis, not an established fact.

Information Gap Declaration

1. All SWE-bench Verified 2026 SOTA numbers (87.6% / 88.7% / 93.9%) have not been double-verified and are marked [To be verified]; they are not adopted in this document.
2. The specific evaluation dates of Claude Sonnet 4.5 on SWE-bench Verified (77.2%) and SWE-bench Multimodal (62.6%) are marked [To be filled].
3. The effectiveness figures in Cases 3.1 and 3.2 are all vendor-published measurement scopes (GitHub, Tencent, Alibaba) lacking third-party independent reproduction; they must be labeled when cited.
4. Some vendor internal-practice figures (LangChain 52.8%→66.5%, Vercel 80%→100%, Claude Code 350,000 DAU, Codex Auto-review 1/200 and 99%) lack accessible primary sources and are marked [To be verified]; they are not adopted in this document.
5. Some enterprise case-study details and source URLs in this direction await backfilling after the research report is supplemented.

6. References

1. SWE-bench official leaderboard (including Verified / Multimodal) — Princeton NLP et al. https://www.swebench.com/
2. SWE-bench Pro public leaderboard — Scale AI. https://scale.com/leaderboard/swe_bench_pro_public
3. GitHub official blog (enterprise effectiveness research) — GitHub, Inc., 2025. https://github.blog/
4. AGENTS.md official site — Linux Foundation / Agentic AI Foundation, 2025-12-09. https://agents.md/
5. Claude Code documentation · Memory (CLAUDE.md configuration locations and @import) — Anthropic. https://docs.claude.com/en/docs/claude-code/memory
6. Claude Code documentation · Hooks — Anthropic. https://docs.claude.com/en/docs/claude-code/hooks
7. NIST SP 800-218A SSDF 1.1 — NIST, 2024-10. https://csrc.nist.gov/pubs/sp/800/218/a/final
8. OWASP Top 10 for LLM Applications 2025 — OWASP, 2025-11-10. https://owasp.org/www-project-top-10-for-large-language-model-applications/
9. ISO/IEC 5059:2024 Software measurement — Software product quality measurement — ISO/IEC, 2024.
10. DORA 2025 report — Google Cloud / DORA, 2025-11-12. https://dora.dev/research/2025/dora-report/
11. Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity — METR, 2025-07-10. https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/
12. Stack Overflow 2025 Developer Survey — Stack Overflow, 2025-07-30. https://survey.stackoverflow.co/2025/