Software Engineering Group (Agents / Coding / DevOps / SRE / Eval / Benchmark)

1. Introduction

1.1. Group Positioning and Background

The Software Engineering group is the only concept-native domain in the entire landscape of AI Harness industry empowerment. The term "Harness" was not invented by the AI field, but borrowed directly from software engineering's Test Harness (test fixture): a test fixture itself produces no functionality — it only ensures that the object under test can be verified automatically, repeatedly, and deterministically. AI Harness inherits precisely this semantics — it does not improve the model's own intelligence, but is responsible for converting the model's uncertainty into engineering predictability.

Therefore, the fundamental difference between this group and the other five industry groups is this: the other groups essentially migrate an already-formed Harness methodology into new domains (chips, knowledge, data, creativity, compliance), whereas this group must answer how this methodology itself was created, where its boundaries lie, and where it goes next.

The years 2025 to 2026 are the watershed of this field. Three simultaneous changes pushed "prompt engineering" completely into "runtime engineering":

• The capability side crossed the usability threshold. Claude Opus 4.6 scored 80.8% on SWE-bench Verified on 2026-02-05, becoming the first model on that leaderboard to break 80%; Claude Sonnet 4.5 scored 51.0% on Terminal-Bench 2.0 on 2025-09-29, becoming the first model on that leaderboard to break 50%. The leaderboard is no longer a question of "can it be done", but of "under what constraints it is done and who is accountable if it goes wrong".
• The adoption side crossed the ubiquity threshold. The DORA 2025 report (2025-11-12) shows that 90%+ of developers already use AI at work; Google DORA released AI-assisted software development on 2025-10-08, making AI-assisted development an independent research subject. The Stack Overflow 2025 Developer Survey (2025-07-30) shows 84% of respondents are using or planning to use AI tools.
• A systematic crack appeared on the cognition side. A randomized controlled trial published by METR on 2025-07-10 showed that experienced open-source developers were 19% slower in measured terms after using AI tools, while the same developers self-rated themselves 20% faster. This "self-assessment vs. measured divergence" is the core tension that recurs throughout this group's documents, and it is the first-principles reason AI Harness exists: without the objective measurement and gating that Harness provides, a team's judgment of AI effectiveness will remain stuck in subjective narrative.

This group breaks down AI Harness practice in the software engineering domain into six mutually supporting directions:

Agents (Agent Engineering) → Coding (AI Coding) → DevOps (Delivery & Infrastructure) → SRE (Operations & Reliability) → Eval (Evaluation System) → Benchmark (Benchmarks & Methodology)

This set of six directions is not a parallel feature list but a closed capability loop from execution units to the measurement system: Agents define "who is doing it", Coding defines "what is produced", DevOps defines "how it is delivered", SRE defines "what happens when it breaks", Eval defines "how quality is judged", and Benchmark defines "whether the judgment standard itself is trustworthy".

1.2. Definitions and Boundaries

Following the unified wording of the project parameter card, AI Harness (AI Governance Layer / Agent Runtime Framework) sits between foundation models and real business systems, responsible for converting the model's uncertainty into engineering predictability. Within this group, this conversion manifests concretely as four kinds of predictability:

Boundaries with adjacent concepts (unified throughout the document):

• This group does not include model pre-training, fine-tuning, and inference infrastructure (which belong to 06-Data Science Group and 04-Hardware R&D Group).
• This group does not include document/knowledge-base/ticketing-type collaboration scenarios (which belong to 05-Knowledge Collaboration Group), but shares the AGENTS.md / SKILL.md format and the evaluation methodology.
• Within this group, the change-trail, privilege-separation, and audit-evidence parts of the SRE and DevOps directions share controls with 08-Risk & Compliance Group under ISO/IEC 42001:2023 and NIST SP 800-218A (SSDF 1.1).
• Evaluations of the product forms of AI IDEs and AI Agents platforms in this group are found under 03-Market Research/; this group focuses only on the engineering mechanisms behind them.

1.3. Projection of the Three-Generation Architecture Evolution onto This Group

图 1-1｜软件工程组三代架构演进：从提示词工程到运行时与评估

数据来源：基于本文分析绘制的示意图。

Following the three-generation architecture evolution in the parameter card, the maturity distribution of this group's directions is as follows:

The core change of the third generation is: the agent's output shifts from "a piece of suggested text" to "a compilable, testable, rollback-able, auditable engineering change set". This change directly determines that all documents in this group follow "product contract + verification evidence + governance red lines" as their writing main thread, and also explains why this group's acceptance anchors are uniformly "compilable, testable, deployable, rollback-able".

2. Positioning Comparison of the Six Directions

2.1. Positioning Comparison Table

2.2. Progression and Dependency Relationships Between Directions

This group suggests understanding the relationship among the six directions as a five-level progression of "able to execute → able to deliver → able to fall back → able to measure → able to question the measurement". This progression aligns with the path along which Harness capability moves up from L1/L2 to L3 and then to L5/L6 (the following is this document's analysis):

Key dependency warning: This group's most common failure path in practice is skipping Eval and building Coding directly. Without a regression set and trajectory evaluation, every prompt change to a coding agent is "by feel", and the team will lose control of its quality baseline within weeks. The second most common failure path is only doing Benchmark without Eval: treating public leaderboard scores as proof of one's own capability, while ignoring the distribution gap between the leaderboard and the internal codebase.

3. Six Directions × Harness Six-Layer Capability Mapping

3.1. Mapping Matrix

The "primary/secondary layer" determination in the table below is an analytical conclusion of this document, not obtained from retrieval; please note this when citing.

How to read the matrix:

• L1 Context Engineering carries the highest weight in the Coding direction. The factual sources of coding tasks (codebase, tests, build logs) are huge in volume, and the assembly strategy directly determines success or failure; typical mechanisms are layered instruction files (five configuration locations + the @import mechanism) and Context Compaction.
• L2 Tools & Execution carries equal weight in the Coding and DevOps directions. The former needs sandboxed compilation/test execution, and the latter needs real infrastructure write privileges; therefore the permission-model design priorities of the two differ.
• L3 Orchestration & Control is the shared primary layer of Agents and DevOps. The orchestration object of the former is sub-agents, and that of the latter is pipeline stages, but both depend on an "interruptible, resumable, replayable" runtime.
• L4 Memory & State has generally low weight in this group, which is a notable feature of the Software Engineering group relative to the Knowledge Collaboration group: the codebase itself is the authoritative external memory, and agents neither need nor should internalize code semantics into their own memory over the long term. The SRE direction is an exception — incident memory is a scarce asset.
• L5 Evaluation & Observability carries the highest weight across the SRE, Eval, and Benchmark directions, forming this group's "measurement triangle".
• L6 Governance & Security carries the highest weight in the DevOps direction. Once a pipeline is granted write and release privileges, its security level is equivalent to that of a production change system.

3.2. Common Implementation Forms of Each Layer in This Group

4. Common Challenges and Countermeasures in This Group

4.1. Challenge 1: The Divergence Between Self-Reported Speedup and Measured Slowdown

Phenomenon: The METR randomized controlled trial (2025-07-10) showed that experienced open-source developers were 19% slower in measured terms after using AI tools, while DORA 2025 (2025-11-12) in the same period showed developers self-reporting +80% productivity. The two figures differ by nearly 100 percentage points.

Root-cause analysis: Self-assessment measures the perceived speed of "the writing-code segment", while measurement captures the end-to-end throughput from "receiving the task to merging into trunk". While AI compresses the former, it amplifies several links in the latter — the cost of verifying generated results, the cost of reviewing longer PRs, the cost of context switching, and the cost of handling pseudo-products that "look right but are actually wrong".

Countermeasures:

• Unify measurement to end-to-end. Anchor on the four DORA metrics (deployment frequency, change lead time, change failure rate, service recovery time), and forbid using "lines of code" or "completion adoption rate" as effectiveness conclusions.
• Make verification costs explicit. Establish a "verification time / generation time" ratio metric in the Eval direction; when the ratio remains above 1, the verification stage has become the bottleneck.
• Reach conclusions with A/B instead of before/after comparison. Before/after comparison is polluted by season, task difficulty, and staffing changes; randomized control is the only design that yields causal conclusions.
• Label the source of self-reported data. When citing figures such as +80%, the "self-reported" attribute must be labeled at the same time; it must not be mixed with measured figures.

4.2. Challenge 2: Context Budget and Repository-Level Understanding

Phenomenon: The size of a real codebase far exceeds any context window; "having the model read the entire repository" is not engineering-feasible; yet feeding only the current file leads to erroneous changes across modules.

Root-cause analysis: This is a structural problem at the L1 context-engineering layer, not a model-capability problem. DORA 2025 shows that 30% of tasks take over 4 hours per run, and context degradation is especially severe in long tasks.

Countermeasures:

• Layered instructions + nearest-first. Place instruction files along directory levels, with instructions nearest to the file being edited taking priority; reuse common clauses via the @import mechanism to avoid redundantly consuming budget.
• Prefer symbol-level indexing over full-text retrieval. Use the "definition—reference" graph as the primary retrieval surface rather than keyword full-text matching.
• Compress rather than truncate. Context Compaction should preserve decision rationale and constraints and discard completed intermediate steps; truncation loses constraints and thereby triggers violations.
• Budget visualization. Record a Token budget table for each task, clearly noting which content was excluded and why.

4.3. Challenge 3: Execution Sandbox and Privilege Red Lines

Phenomenon: An agent must be able to execute code in order to verify code, yet execution capability inherently carries destructiveness.

Root-cause analysis: This is an intersecting problem of L2 and L6. Both OWASP Top 10 LLM Applications 2025 (2025-11-10) and the OWASP ASI Top 10 (2026-02-16) list "Excessive Agency" and "privilege abuse" among core risk items; OWASP Top 10 MCP (2025-12-07) further points out the new attack surface introduced by tool servers.

Countermeasures:

• Least privilege by default, temporary elevation on demand. Destructive operations (deleting branches, force-pushing, writing to production databases, terraform apply) are denied by default and require explicit human confirmation.
• Use event hooks for deterministic interception. Relying on the model to "remember not to do it" is unreliable; hard interception must be performed with hooks before and after tool calls; hook exit-code semantics must be agreed within the team and written into AGENTS.md.
• Isolate the sandbox from the host. Builds and tests run inside containers or microVMs; running unreviewed generated code directly on the host environment is forbidden.
• Cost guardrails. Set per-task and per-day Token/call budget caps; when exceeded, interrupt and escalate.

4.4. Challenge 4: Non-Determinism and Regressable Verification

Phenomenon: Repeating the same task produces different results, and traditional unit tests cannot cover the "agent behavior" layer.

Root-cause analysis: The existing software engineering quality system (compilation, unit tests, integration tests, code review) verifies the product, while Harness additionally needs to verify the process that produces the product. This is the layer this group adds relative to traditional software engineering.

Countermeasures:

• Two-layer gating: product gates (compilation + tests + static analysis + coverage) and process gates (trajectory evaluation + step limits + privilege-hit checks) coexist side by side; neither can be omitted.
• Solidify regression sets. Turn typical tasks into regression sets; every prompt/model/tool change must run full regression and retain the scorecard.
• Keep trajectory evidence. The full-process Trace must be replayable; otherwise incidents cannot be reviewed.
• Make the judgment protocol determinate. Use scorers that are decidable or automatically computable wherever possible to reduce reliance on subjective LLM-as-Judge; when using a Judge, fix and record the model version and temperature.

4.5. Challenge 5: Inconsistent Evaluation Protocols and Leaderboard Drift

Phenomenon: The same model shows vastly different figures across different leaderboards, dates, and configurations, making cross-leaderboard comparison basically invalid.

Root-cause analysis: Evaluation configuration (reasoning effort, tool availability, step limits, retry counts, temperature) often affects results more than the differences among the models themselves. For example, GPT-5.2's 52.5% score on ARC-AGI-2 is explicitly labeled as an xhigh-reasoning configuration; directly comparing that figure with other models under default configuration is a protocol error.

Countermeasures:

• Cited figures must carry four elements: model name + version, leaderboard name, evaluation date, and reasoning/tool configuration. All documents in this group strictly comply.
• Distinguish leaderboard comparability levels: comparable within the same leaderboard and configuration > suspicious within the same leaderboard but different configurations > incomparable across leaderboards.
• Public leaderboards as upper-bound reference only. Internal capability is determined by internal regression sets; public leaderboards serve to calibrate the "capability ceiling".
• Run contamination detection regularly. Check whether evaluation tasks appear in the model's training corpus or the internal codebase.

4.6. Challenge 6: Engineering Human-in-the-Loop and Acceptance Anchors

Phenomenon: "Manual review" often degenerates into a formality — a person sees an already-generated large patch, understanding neither its intent nor being able to review it effectively.

Root-cause analysis: Human-in-the-Loop is not a switch, but a set of breakpoint locations that need to be designed. If the breakpoints are placed in the wrong locations, the loop degenerates into a rubber stamp.

Countermeasures:

• Move confirmation points forward to the planning stage. Confirm the approach and impact scope before the agent starts writing code, rather than confirming the result after it is written.
• Set confirmation points by risk tier. Low-risk (formatting, comments, test additions) can auto-merge; medium-risk (logic changes, dependency upgrades) require human review; high-risk (architecture changes, privilege changes, data migration, production releases) require human approval + a rollback plan.
• Unify the acceptance anchors as the four cans: compilable, testable, deployable, rollback-able. If any one is unsatisfied, it must not proceed to the next stage.
• Provide reviewable change descriptions. The agent's output must include: what was changed, why it was changed, how to verify it, possible risks, and how to roll back.

5. Document Navigation

• Just want to solve coding efficiency: read 02-coding.md + the verification and privilege sections of AGENTS.md.
• Want to build an enterprise-grade agent platform: read through in order 1→5, with AGENTS.md as the configuration baseline.
• Need technology selection or external reporting: focus on 06-benchmark.md; first build the defensive mindset that "figures cannot be trusted", then read 05-eval.md to build internal measurement capability.
• Care about production stability: read the release-gate and rollback sections of 04-sre.md and 03-devops.md.

6. Summary

The Software Engineering group is the native domain of the AI Harness concept. Here the Test Harness metaphor is not rhetoric but the methodological source of all this group's engineering practice: it produces no functionality, only making functionality verifiable automatically, repeatedly, and deterministically.

This group's six directions form a closed loop: Agents define the execution structure, Coding produces engineering artifacts, DevOps completes delivery, SRE safeguards operations, Eval measures effect, and Benchmark reviews the measurement itself. If any link in the loop is missing, it manifests as the same symptom — the team cannot answer "did this change get better or worse".

The 2025–2026 data delivers a judgment that must be faced squarely: the capability side has crossed the usability threshold (SWE-bench Verified 80.8%, Terminal-Bench 2.0 51.0%), the adoption side has crossed the ubiquity threshold (90%+ of developers use it), but the measurement side has not yet crossed the credibility threshold (self-reported +80% vs. measured -19%). Almost all of AI Harness's remaining value in the software engineering domain is concentrated in closing this measurement-side gap: making context auditable, execution sandboxable, trajectories replayable, regressions gateable, and figures reproducible.

This also explains the writing main thread of all this group's documents: first define the contract, then define verification, and only then discuss capability.

Information Gap Declaration

During the writing of this document, the following unverified items exist; all have been annotated per the norms, and none have been filled with speculative content:

1. The complete SOTA figures on SWE-bench Verified for 2026 (87.6% / 88.7% / 93.9%) and the full Terminal-Bench 2.0 leaderboard figures have not been double-verified and are marked as [To be verified].
2. Some vendor internal-practice figures (LangChain 52.8%→66.5%, Vercel 80%→100%, Claude Code 350k DAU, Codex Auto-review 1/200 and 99%) lack accessible primary sources and are marked as [To be verified]; this document has not adopted them.
3. Some article-level URLs cited in the direction documents (especially vendor blogs and leaderboard snapshot pages) could not be verified one by one; only accessible official entry points are given in the references.
4. This group's retrieval report (R02) had not been saved to disk at the time of writing; case details and source URLs will be backfilled after they are added.

7. References

1. SWE-bench official leaderboard — Princeton NLP et al. https://www.swebench.com/
2. SWE-bench Pro (public leaderboard) — Scale AI. https://scale.com/leaderboard/swe_bench_pro_public
3. Terminal-Bench official leaderboard — Laude Institute et al. https://www.tbench.ai/
4. SWE-Lancer official leaderboard — Scale AI. https://www.swe-lancer.com/
5. ARC Prize official leaderboard (ARC-AGI-2) — ARC Prize Foundation. https://arcprize.org/leaderboard
6. DORA 2025 State of AI-assisted Software Development — Google Cloud / DORA, 2025-11-12. https://dora.dev/research/2025/dora-report/
7. AI-assisted software development — Google DORA, 2025-10-08. https://dora.dev/
8. Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity (randomized controlled trial) — METR, 2025-07-10. https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/
9. Stack Overflow 2025 Developer Survey — Stack Overflow, 2025-07-30. https://survey.stackoverflow.co/2025/
10. GitHub official blog (Copilot enterprise effectiveness research) — GitHub, Inc., 2025. https://github.blog/
11. ISO/IEC 42001:2023 Artificial Intelligence Management System — ISO/IEC, 2023. https://www.iso.org/standard/42001
12. AI Risk Management Framework (AI RMF 1.0) and GenAI Profile — NIST, 2024-07-26. https://www.nist.gov/itl/ai-risk-management-framework
13. NIST SP 800-218A Secure Software Development Framework (SSDF) 1.1 — NIST, 2024-10. https://csrc.nist.gov/pubs/sp/800/218/a/final
14. OWASP Top 10 for LLM Applications 2025 — OWASP, 2025-11-10. https://owasp.org/www-project-top-10-for-large-language-model-applications/
15. OWASP Agentic AI — Threats and Mitigations (T&C) and Top 10 — OWASP, 2025-12. https://genai.owasp.org/
16. AGENTS.md official site (hosted by Linux Foundation / AAIF) — 2025-12-09. https://agents.md/
17. Claude Code official documentation · Hooks and Memory — Anthropic. https://docs.claude.com/en/docs/claude-code/hooks
18. Model Context Protocol (MCP) official specification — Anthropic et al. https://modelcontextprotocol.io/
19. GitLab release history (18.0 / 19.0 / 20.0) — GitLab Inc. https://about.gitlab.com/releases/
20. ISO/IEC 5059:2024 Software measurement — software product quality measurement — ISO/IEC, 2024.
21. ISO/IEC TR 5469:2024 Artificial intelligence — Functional safety and AI systems — ISO/IEC, 2024.
22. NIST Cybersecurity Framework (CSF) 2.0 — NIST, 2024-02-26. https://www.nist.gov/cyberframework