SRE (AI-Driven Operations and Reliability)

1. Introduction

1.1. Background: The Special Nature of Reliability Work

Site Reliability Engineering (SRE) has a fundamental difference from other branches of software engineering: its object of work is production systems that are actively failing. This introduces three constraints that the other directions do not have:

1. Asymmetric time pressure. Writing code can take all the time needed; during an outage, every minute of mean time to recovery (MTTR) directly corresponds to business loss. The value of an agent is first reflected in "can it produce a usable hypothesis faster," not "can it produce a perfect analysis."
2. Incomplete and contradictory information. At the moment of an outage, metrics, logs, and traces are often partially missing, time-unsynchronized, or even mutually contradictory. This is completely different from the "information is largely trustworthy" premise of other directions.
3. Irreversible actions that may worsen the outage. A coding mistake can be rolled back; executing a wrong mitigation action in production can amplify a localized failure into a global one.

These three points determine that the AI Harness for the SRE direction differs fundamentally from that for the coding direction: the Harness for coding pursues a "generate–verify–retry" loop, while the Harness for SRE must pursue a "hypothesis–verify–contain-loss" loop. The former allows trial and error; the latter must control the blast radius.

1.2. Definition

SRE direction (AI-driven operations and reliability): an engineering field that researches and practices using AI agents to participate in fault localization, root-cause analysis, capacity assessment, chaos engineering, and on-call response, and to shorten recovery time while guaranteeing that the outage is not worsened.

This direction covers five types of tasks:

1.3. Positioning Within the AI Harness System

图 1-1｜SRE 方向在 AI Harness 六层体系中的定位

数据来源：基于本文分析绘制的示意图。

The primary layer of this direction is the L5 Evaluation and Observability layer, with the L4 Memory and State layer and the L6 Governance and Security layer as secondary layers.

1.4. Bottleneck Layer

The bottleneck of this direction lies in the L5 Evaluation and Observability layer, specifically two things:

1. The quality of the observability data itself. AI diagnostic capability is strongly correlated with the input data. If metrics are missing key dimensions, logs are not structured, or trace sampling rates are too low, even the strongest model can only guess. In the industry, in most cases of "AI on-call deployed but ineffective," the root cause lies on the data side, not the model side.
2. The lack of a structured "hypothesis–verify" mechanism. Large models naturally tend to give one plausible-sounding explanation (narrative bias), whereas root-cause analysis precisely needs to "generate multiple candidate hypotheses → falsify each in turn → converge." Without a Harness enforcing this process, the AI can easily anchor onto a wrong hypothesis at the outset and follow it all the way astray.

A secondary bottleneck lies in the L4 Memory and State layer: incident memory is the scarcest asset in SRE, but it usually exists as unstructured postmortem documents that are hard to retrieve and reuse. Structuring historical failures into a four-tuple of "symptom → candidate cause → verification method → effective mitigation" is the precondition for AI to genuinely deliver value.

1.5. Value

• Shorten MTTR: compress the time from "human sees the alert to localization" from tens of minutes down to minutes, especially in scenarios with long cross-system call chains.
• Reduce on-call burden: night alerts are handled by the AI in the first round of triage, filtering noise, aggregating related alerts, and giving initial hypotheses with supporting evidence.
• Make chaos engineering routine: the AI can automatically generate experiment hypotheses, orchestrate injection steps, and determine whether the steady state has been broken, thereby turning "annual drills" into "weekly verification."
• Accumulate incident knowledge: every hypothesis–verify chain produced by each diagnosis becomes retrieval material for the next failure.

2. Glossary

3. Case Studies

Note: The following cases are compiled from verifiable public primary sources. Every effectiveness figure is annotated with the source's methodology, evaluation date, and configuration; self-reported and externally measured figures are labeled separately.

3.1. Case One: The Boundary of Fault-Diagnosis Capability in Terminal Environments

Source: Terminal-Bench 2.0 official leaderboard, https://www.tbench.ai/

3.1.1. Background

SRE daily work is highly dependent on the terminal: inspecting processes, checking configurations, analyzing logs, verifying network connectivity, and modifying system parameters. Unlike programming tasks, every step in such tasks may alter system state, and most steps are irreversible. Therefore, measuring an agent's performance in terminal environments has direct reference value for evaluating the feasibility of the SRE direction.

3.1.2. Approach (from the Evaluation-Design Perspective)

Terminal-Bench 2.0 tasks require the agent to complete dozens to hundreds of operations in a real terminal environment, with success ultimately determined by a set of assertion scripts. This design corresponds to real SRE work in three ways:

• State dependency: the result of each step depends on the system state left by the previous step, and cannot be replaced by pure text-based reasoning.
• Irreversible operations: an erroneous deletion or overwrite cannot be undone by "regenerating," so it must be avoided through planning.
• Determinism: success or failure is determined by assertions, avoiding subjective scoring — this matches the SRE judgment of "whether the service has recovered."

3.1.3. Results

Interpretation (direct implications for SRE adoption):

1. Roughly half of long-horizon terminal tasks still fail. This means that in SRE scenarios, the AI should be positioned as diagnostic assistance rather than autonomous response: it can narrow candidate causes from 20 to 3, but the final action should still be decided by a human.
2. Failure modes matter more than success rate. In reliability scenarios, AI failure is often not "it didn't find a cause" but "it found a wrong but plausible-sounding cause" — the latter is more dangerous, because it causes responders to waste precious recovery time in the wrong direction.
3. The full Terminal-Bench 2.0 leaderboard and per-category scores are labeled [To be verified].

3.2. Case Two: Operational Troubleshooting in GUI Environments

Source: OSWorld-Hard leaderboard (OSWorld project)

3.2.1. Background

A great deal of operations work does not happen on the command line but in GUI environments: cloud provider consoles, monitoring dashboards, ticketing systems, and internal management platforms. These systems often lack a complete API, so operations can only be performed by interacting with the graphical interface. Therefore, the agent's operational ability in GUI environments directly determines what proportion of SRE scenarios it can cover.

3.2.2. Approach (from the Evaluation-Design Perspective)

OSWorld-Hard requires the agent to complete tasks in a real operating system environment, through screen observation and mouse-and-keyboard operations. Compared with terminal tasks, it adds two layers of difficulty:

• Perception difficulty: state can only be inferred from screen pixels, with no structured output.
• Operational precision: a wrong click position can lead to irreversible consequences (for example, clicking "Delete" instead of "Restart").

3.2.3. Results

Interpretation:

1. Scores in GUI environments are also around 60%, in the same order of magnitude as terminal tasks (51.0%–54.0%). This suggests that for tasks requiring continuous operation in real environments, the current ceiling of AI capability is roughly 60%.
2. For SRE, this order of magnitude is sufficient to support "triage and hypothesis generation," but not sufficient to support "unattended automatic response."
3. GUI operations are more irreversible, so the adoption principle for this direction is: read-only analysis can be automated, but write operations must be authorized by tier.

3.3. Case Three: The Reliability Industry Baseline in the AI Era

Source: DORA 2025 State of AI-assisted Software Development, 2025-11-12; Google DORA AI-assisted software development, 2025-10-08; Stack Overflow 2025 Developer Survey, 2025-07-30

3.3.1. Background

AI coding tools have increased the speed at which changes are produced. But reliability engineering has a simple rule: changes are the primary source of failures. If the change speed rises while change quality and verification capability do not improve in step, the change failure rate is bound to rise. Therefore, the SRE direction must answer one question: in the AI era, has reliability gotten better or worse?

3.3.2. Approach

The DORA four-metric framework is adopted as the unified measurement standard — deployment frequency, change lead time, change failure rate, and time to restore service. The value of this framework is that: it measures speed and stability simultaneously, making the trade-off between "faster" and "more stable" visible, avoiding measuring only speed while ignoring stability.

3.3.3. Results

Table 3-1　Key findings on AI adoption status and reliability

Interpretation (the combination this direction must be most wary of):

1. Review time -90% + 46% don't trust accuracy is a contradictory combination: on one hand review investment has dropped sharply, on the other hand nearly half of practitioners do not trust the output. This suggests reviews may have become shallower, and shallower reviews are exactly the entry point for defect escape.
2. 30% of tasks exceed 4 hours each: this shows many tasks have not become simpler because of AI. Long tasks are especially dangerous in outage scenarios — recovery time is stretched out.
3. Self-reported +80% versus measured -19%: this is the most important cognitive warning for this direction. In the reliability domain, erroneous subjective judgment directly leads to wrong capacity decisions and wrong automation authorization.
4. Conclusion: the first task of SRE in the AI era is not "using AI to speed things up," but establishing objective measurement — first be able to measure accurately, then talk about improvement.

4. Practice Standards

4.1. AGENTS.md Specification

The following is the complete AGENTS.md text for this direction, placed in a code block for direct copy. When used, it must be superimposed on the common clauses of the group-level AGENTS.md; conflicts are resolved by the nearest-first principle.

# AGENTS.md —— 软件工程 · SRE 方向（运行与可靠性）

## 角色与边界

- 本文件约束**参与生产系统可靠性工作的 AI 智能体**：故障定位、根因分析、容量评估、混沌工程、值守响应。
- 允许自行完成：只读查询（指标、日志、链路、配置、拓扑）、生成候选假设与验证方法、生成故障时间线、生成复盘草稿、生成容量建议、生成混沌实验方案、按白名单执行**已授权的低风险只读诊断命令**。
- 禁止自行完成：执行任何改变生产状态的动作（重启、扩缩容、切流、限流、回滚、数据修复）、修改告警规则或阈值、删除或归档日志与监控数据、在故障期间执行实验性命令。
- 边界判定：**只读与写入的分界线就是本方向的权限红线**。缓解动作（Mitigation）必须由人执行，除非该动作已被显式列入白名单并获得事前授权。
- 核心原则：**事中优先缓解（Mitigation），而非根因修复（Remediation）**。AI 的目标是缩短恢复时间，不是查明真相。

## 环境假设

- 提供可观测性数据访问：指标（Metrics）、日志（Logs）、链路追踪（Traces），且时间同步、有统一的服务标识。
- 提供拓扑与依赖图；提供近期变更清单（部署、配置、基础设施变更）。
- 提供事故库（Incident DB）的只读访问，含历史故障的结构化记录。
- 提供只读的云/集群 API 访问；写操作默认关闭。
- 提供权威时间源；**所有时间线必须统一时区与时钟源**。
- 提供操作审计日志，且智能体不可删改。
- 缺失任一项时降级执行并显式声明；**时间不同步时必须声明"时间线不可信"**。

## 上下文加载顺序（Context Budget）

1. 任务契约：目标系统、影响面、时间窗、验收标准（固定，不压缩）
2. 就近指令文件：当前服务/目录的运维约定
3. 告警与症状：告警列表、用户报告、SLO 状态
4. 时间窗内的遥测：指标异常段、相关日志摘要、关键链路样本（**裁剪后的**）
5. 拓扑与依赖：受影响服务及其上下游
6. 近期变更：时间窗内的部署/配置/基础设施变更
7. 事故记忆：相似历史故障的四元组（症状/原因/验证/缓解）
8. 工具结果：按需追加

记录 Token 预算表。遥测数据必须裁剪（聚合、采样、异常段提取），**禁止把原始日志全量灌入上下文**。

## 工具契约

- 副作用分级：R（只读查询）/ W（工作区写，如生成报告）/ X（沙箱内执行）/ S（改变生产状态，需确认）/ D（不可逆，默认拒绝）。
- **诊断命令白名单**：只读诊断命令（如查看状态、读取配置、查询指标）可列入白名单自动执行；其余一律需确认。
- **缓解动作白名单**：仅事前经审批的动作（如特定服务的扩容、特定开关的切换）可列入白名单；每次执行必须留痕。
- 一切实验性、探索性命令禁止在生产环境执行。
- 所有执行类工具必须设置超时；超时即失败。
- 输出超限时必须聚合或采样，禁止全量灌入。

## 任务执行流程（SOP）

1. 确认契约：目标系统、影响面、时间窗、是否处于故障中。
2. 采集症状：告警、SLO 状态、用户报告；建立**统一时基**。
3. 建立时间线：按时间排序 症状 / 变更 / 告警 / 缓解动作。
4. 缩小范围：用拓扑与依赖图把范围从"整个系统"缩小到"具体组件"。
5. **生成候选假设（≥3 个）**：每个假设必须写明"若成立，应观察到什么"。
6. **逐一验证**：对每个假设给出可执行的验证方法与预期结果；**验证必须落在数据上，不能落在推理上**。
7. 收敛：保留未被证伪的假设，给出置信度与依据；**禁止只给一个假设**。
8. 给出建议：区分 缓解建议（立即）与 修复建议（事后）。
9. 留痕：归档时间线、假设—验证链、结论与置信度。

## 验证与证据要求

| 验证项 | 判定 | 证据 |
|---|---|---|
| 时间线时基统一 | 全部事件同一时钟源 | 时间线（含时区标注） |
| 假设数量 ≥ 3 | 非单一叙事 | 假设清单 |
| 每个假设有可验证预期 | "若成立应观察到 X" | 假设表 |
| 验证落在数据上 | 有具体的查询/命令与结果 | 验证记录（命令 + 输出摘要） |
| 区分相关与因果 | 已显式说明为何不是仅相关 | 因果论证段落 |
| 区分症状与原因 | 已标注哪些是症状 | 标注 |
| 缓解建议可执行 | 有具体动作与影响面 | 缓解方案 |
| 爆炸半径已评估 | 每个建议动作的影响范围已知 | 影响面评估 |

**禁止**：
- 以"最可能的原因是……"作为唯一结论而不给候选集。
- 以时间上的先后共现作为因果依据（"变更后故障，所以变更导致故障"必须进一步验证）。
- 以无数据的推理替代查询。

**必须**：无法确定根因时，明确输出"根因未确定"并列出已排除的假设——这是有效结论，不得为了给出答案而猜测。

## 失败与升级策略

- 数据缺失（日志缺失、采样不足）：声明"数据不足以判定"，升级人工，禁止猜测。
- 假设全部被证伪：回到采集阶段，扩大时间窗或范围；禁止降低标准强行收敛。
- 缓解动作未授权：不得执行；输出建议与依据，等待授权。
- 操作可能加重故障：立即停止，升级。
- 时间不同步：声明时间线不可信，先对齐时钟。

升级三要素：已尝试动作与结果 / 卡点现象与证据 / 建议选项及风险。缺一无效。

## 安全与合规红线

1. 禁止执行任何未列入白名单的生产写操作。
2. 禁止在故障期间执行实验性或探索性命令。
3. 禁止修改告警规则、阈值、静默策略以"降低噪声"。
4. 禁止删除、归档、覆盖日志、指标、链路与审计数据。
5. 禁止访问与当前故障无关的生产数据（尤其含个人信息的数据）。
6. 缓解动作必须先评估爆炸半径；影响面超出授权范围即升级。
7. AI 系统的功能安全论证参考 ISO/IEC TR 5469:2024；可信赖性论证参考 ISO/IEC TR 24028:2020。
8. 风险管理对齐 NIST CSF 2.0（2024-02-26）的检测（DE）与响应（RS）功能。
9. 故障时间线与处置记录属于成文信息，按 ISO/IEC 42001:2023 的要求留存。

## 禁止事项

1. 禁止给出单一假设而不给候选集。
2. 禁止把相关性表述为因果性。
3. 禁止在无数据时给出结论。
4. 禁止在故障期间擅自执行缓解动作（未列入白名单者）。
5. 禁止使用 `[待填写]`、`XX`、`___` 之外的非标准占位符；不确定处统一使用 `[待填写]`。
6. 禁止以"模型认为"作为技术判断依据。
7. 禁止删除或改写他人的运维手册、告警配置与审计日志。

## 输出格式

故障场景输出必须包含：

- **状态摘要**：当前影响面、SLO 状态、是否已缓解
- **时间线**：时间 / 事件 / 来源 / 置信度
- **候选假设表**：假设 / 若成立应观察到 / 验证方法 / 验证结果 / 结论
- **结论**：未被证伪的假设（含置信度）；或"根因未确定"+ 已排除清单
- **建议**：缓解建议（立即）/ 修复建议（事后）/ 预防建议
- **证据附录**：执行的查询与命令、输出摘要

## 评估与自检

| 自检项 | 判定 |
|---|---|
| 时间线时基统一 | 是 / 否 |
| 候选假设 ≥ 3 个 | 是 / 否 |
| 每个假设有可验证预期 | 是 / 否 |
| 验证落在数据上（有查询与结果） | 是 / 否 |
| 已区分症状与原因 | 是 / 否 |
| 已区分相关与因果 | 是 / 否 |
| 缓解建议含爆炸半径评估 | 是 / 否 |
| 未确定时已明确声明（未强行给答案） | 是 / 否 |

任一"否"须显式列出；自检不通过即任务未完成。

度量口径：MTTD、MTTR、诊断准确率（事后复盘验证）、告警噪声比、缓解动作成功率。禁止以"响应感觉更快了"作为结论。

4.2. SKILL.md Specification

---
name: incident-diagnosis-with-hypothesis-gate
description: 对一次生产故障做结构化诊断——统一时基建立时间线、用拓扑缩小范围、生成不少于三个候选假设并逐一用数据验证、区分症状与原因、区分相关与因果、输出带置信度的结论与分级建议。适用于故障定位、根因分析、值守分诊。当需要 AI 参与故障诊断但必须避免"单一叙事"误判时使用。
version: 1.0
created: 2026-09-12
---

# 带假设门禁的故障诊断

## 适用场景

- 生产告警触发，需要快速定位影响范围与候选原因。
- 故障已恢复，需要做根因分析与复盘。
- 值守分诊：判断告警是否值得唤醒人工、属于哪个服务、可能的原因类别。
- 容量异常（如水位突增）需要归因。

不适用于：无遥测数据的场景（数据不足时本技能应直接声明不可用）、计划内的变更验证、安全事件调查（走安全流程）。

## 前置条件

| 类别 | 要求 | 缺失时处置 |
|---|---|---|
| 时基 | 所有数据源时钟同步 | 声明"时间线不可信"，先对齐 |
| 遥测 | 指标、日志、链路至少其一可用 | 数据不足即声明不可用 |
| 拓扑 | 服务依赖关系可见 | 降级为逐服务排查 |
| 变更记录 | 时间窗内的变更可查 | 声明"变更关联无法验证" |
| 权限 | 只读访问；写操作需白名单 | 按最低权限执行 |

## 输入

- 告警内容或故障描述
- 目标服务与影响面
- 时间窗（起止时间，含时区）
- 可用的遥测数据源清单

## 输出

- 状态摘要（影响面、SLO 状态、是否已缓解）
- 时间线（时间 / 事件 / 来源 / 置信度）
- 候选假设表（假设 / 预期观察 / 验证方法 / 结果 / 结论）
- 结论（含置信度）或"根因未确定 + 已排除清单"
- 三级建议：缓解 / 修复 / 预防

## 执行步骤

1. **确认契约**：目标服务、影响面、时间窗、是否故障中。
2. **统一时基**：确认所有数据源的时钟源与时区；不一致则先标注。
3. **采集症状**：告警、SLO/错误预算状态、用户报告；明确"什么是异常"。
4. **建立时间线**：按序记录 症状 / 变更 / 告警 / 缓解动作，每条标注来源与置信度。
5. **缩小范围**：用拓扑与依赖图从"系统"缩小到"组件"；优先排除法（哪些部分确定正常）。
6. **生成候选假设（≥3）**：每条写明"若该假设成立，应观察到 X"。**禁止只给一个假设**。
7. **逐一验证**：对每条假设执行具体查询或命令，记录输出摘要；验证必须落在数据上。
8. **收敛与置信度**：保留未被证伪的假设，给出置信度与依据；全部证伪则回到第 4 步扩大范围。
9. **区分因果**：对"时间上共现"的证据显式论证为何是因果而非相关（如：有剂量—反应关系、有机制解释、可复现）。
10. **输出建议**：缓解（立即、含爆炸半径）/ 修复（事后）/ 预防（长期）。
11. **留痕**：归档时间线、假设—验证链、结论、证据。

## 质量标准（DoD）

1. 时间线时基统一，含时区标注。
2. 候选假设不少于 3 个，且彼此互斥或可区分。
3. 每条假设都有"若成立应观察到"的可验证预期。
4. 验证落在数据上：有具体查询/命令与输出摘要。
5. 明确区分症状与原因。
6. 明确区分相关与因果，并对因果给出机制解释或复现证据。
7. 结论带置信度；无法确定时明确输出"根因未确定"+ 已排除清单。
8. 缓解建议含爆炸半径评估与回滚方式。
9. 未执行任何未授权的写操作。
10. 全过程留痕，可事后复盘。

## 常见失败与处理

| 失败 | 表现 | 处理 |
|---|---|---|
| 单一叙事 | 只给一个"最可能原因" | 强制候选集 ≥ 3；否则判定不合格 |
| 相关当因果 | "变更后故障，所以是变更导致" | 要求机制解释或复现证据；否则降级为"待验证" |
| 数据不足 | 日志缺失、采样过低 | 声明"数据不足以判定"，不得猜测 |
| 时间不同步 | 各源时间对不上 | 标注"时间线不可信"，先对齐 |
| 假设全证伪 | 无假设成立 | 扩大时间窗或范围，禁止强行收敛 |
| 擅自处置 | 执行了未授权缓解 | 判定为红线违反，回滚并复盘 |
| 噪声淹没 | 告警过多无法定位 | 先做分诊聚合，再定位 |

## 示例

**场景**：支付服务 P99 延迟从 200ms 升至 3s，错误率从 0.1% 升至 4%。

- 时基：全部数据源统一为 UTC+8，已校验 NTP 同步。
- 时间线：14:02 部署 v2.3.1；14:05 延迟开始上升；14:07 错误率上升；14:12 告警触发。
- 范围：支付服务 → 依赖的账务服务（正常）→ 数据库连接池（活跃连接数打满）。
- 候选假设：
  1. 新版本引入慢查询，连接占用时间变长 → 预期：慢查询日志中新 SQL 出现（**验证：存在，成立**）
  2. 流量突增导致连接池不足 → 预期：QPS 显著上升（**验证：QPS 平稳，证伪**）
  3. 数据库侧故障 → 预期：DB 指标异常或其他服务也受影响（**验证：DB 正常且其他服务无异常，证伪**）
- 因果论证：新 SQL 的耗时与连接池占用时长呈剂量—反应关系，且回滚后指标恢复 → 因果成立，非仅相关。
- 建议：缓解（回滚 v2.3.1，爆炸半径=单服务）/ 修复（优化该 SQL 并加索引）/ 预防（上线前慢查询门禁 + 连接池水位告警）。
- 置信度：高（已被回滚验证）。

4.3. Implementation Checklist

Observability foundation (L5, the prerequisite of prerequisites)

• [ ] Metrics, logs, and traces data are all in place, with unified service identifiers
• [ ] All data sources have synchronized clocks; the timeline is trustworthy
• [ ] The trace sampling strategy is defined, and sampling rates on critical paths are sufficient
• [ ] Logs are structured (not plain text) and searchable by field
• [ ] Topology and dependency graphs are maintained and consistent with reality
• [ ] Change records within the time window are available (deployments, configurations, infrastructure)

Diagnostic process (L3 + L5)

• [ ] The diagnostic SOP enforces "candidate hypotheses ≥ 3" and forbids single narratives
• [ ] Every hypothesis must have a verifiable expectation of "if true, one should observe..."
• [ ] Verification results must land on data (with queries and output), not rely on reasoning alone
• [ ] Symptoms are clearly distinguished from causes
• [ ] Correlation is clearly distinguished from causation; causation needs a mechanism explanation or reproducible evidence
• [ ] "Root cause undetermined" is an allowed output, and must be accompanied by an excluded-list

Incident memory (L4)

• [ ] Historical failures are structured as four-tuples: symptom / cause / verification method / effective mitigation
• [ ] The incident database can be retrieved by diagnostic agents
• [ ] Each diagnosis's hypothesis–verify chain is archived as material for the next time
• [ ] Postmortems use a blameless approach, focusing on systems and processes

Action control (L6)

• [ ] Read-only diagnostic commands have a whitelist and can be executed automatically
• [ ] Mitigation actions have a whitelist, and every execution is logged
• [ ] Every mitigation action has a blast-radius assessment
• [ ] Experimental commands are forbidden during an outage (hard rule)
• [ ] The AI must not modify alert rules, thresholds, or silencing policies
• [ ] Logs, metrics, traces, and audit data cannot be deleted by the AI

Chaos engineering

• [ ] Experiments have a clear steady-state hypothesis (measurable normal behavior)
• [ ] Experiments have a blast-radius declaration and an abort switch
• [ ] Experiments are run in a controlled environment or within a controlled time window
• [ ] Experiment results are written back as resilience improvement items

Measurement

• [ ] MTTD and MTTR are being measured and are attributable
• [ ] Diagnostic accuracy is verified through after-the-fact postmortem review (not self-reported)
• [ ] The alert noise ratio is being measured, with a noise-reduction mechanism
• [ ] Mitigation action success rate is being measured
• [ ] Self-reported and measured data are labeled separately and never mixed

5. Summary

The SRE direction is the one in the AI Harness that can least tolerate the AI being confidently wrong. In a coding scenario, a wrong AI output produces a patch that a human can decline to adopt; in an outage scenario, a wrong AI judgment makes responders consume their most precious recovery time in the wrong direction.

The core conclusions of this direction:

1. The bottleneck is in the L5 Evaluation and Observability layer, and first of all in the quality of the observability data itself. Without structured, time-base-unified, and sufficiently covered telemetry data, even the strongest model can only guess. In the industry, the majority of "AI on-call ineffective" cases have their root cause on the data side.
2. The Harness must be used to enforce a "hypothesis–verify" structure to counter the model's narrative bias. Large models tend to give one plausible-sounding explanation, whereas root-cause analysis needs "multiple candidates → falsify each in turn → converge." Candidate hypotheses ≥ 3, each with a verifiable expectation, and verification landing on data — these three are the minimum requirements.
3. Capability ceiling determines positioning. On Terminal-Bench 2.0, 51.0%–54.0%; on OSWorld-Hard, 58.0%–61.4% — this shows that current AI succeeds at about 60% on tasks requiring continuous operation in real environments. This is enough to support "triage and hypothesis generation," but not enough to support "unattended automatic response". Therefore the adoption principle of this direction is: read-only analysis can be automated, but write operations must be authorized by tier.

Finally, this direction must bear in mind the divergent numbers: DORA 2025 shows self-reported productivity +80% and review time -90%, while the METR randomized controlled trial (2025-07-10) shows experienced developers were 19% slower as measured, and Stack Overflow 2025 shows 46% of practitioners do not trust AI output accuracy. In the reliability domain, the bias of subjective judgment translates directly into capacity misjudgment and over-authorization. First be able to measure accurately, then talk about improvement — this is the first principle of SRE in the AI era.

Information Gap Statement

1. The specific evaluation dates for GPT-5.2-Codex (61.4%) and Claude Opus 4.5 (58.0%) on OSWorld-Hard are labeled [To be filled].
2. The full Terminal-Bench 2.0 leaderboard and per-category scores are labeled [To be verified].
3. Each organization's own MTTD / MTTR / diagnostic accuracy baselines are labeled [To be filled] and need to be filled with internal measurements.
4. Concrete practice data for chaos engineering with AI participation lacks verifiable public sources; this document only gives methodological requirements and does not cite quantified effects.
5. Some enterprise case details and source URLs for this direction await backfilling after supplementing the research report.

6. References

1. Terminal-Bench official leaderboard — Laude Institute et al. https://www.tbench.ai/
2. OSWorld official site (OSWorld-Hard) — OSWorld project. https://os-world.github.io/
3. DORA 2025 State of AI-assisted Software Development — Google Cloud / DORA, 2025-11-12. https://dora.dev/research/2025/dora-report/
4. AI-assisted software development — Google DORA, 2025-10-08. https://dora.dev/
5. Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity — METR, 2025-07-10. https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/
6. Stack Overflow 2025 Developer Survey — Stack Overflow, 2025-07-30. https://survey.stackoverflow.co/2025/
7. Site Reliability Engineering (the original SRE book) — Google. https://sre.google/books/
8. OpenTelemetry semantic conventions — OpenTelemetry. https://opentelemetry.io/docs/concepts/semantic-conventions/
9. ISO/IEC TR 5469:2024 Artificial intelligence — Functional safety and AI systems — ISO/IEC, 2024.
10. ISO/IEC TR 24028:2020 Artificial intelligence — Overview of trustworthiness — ISO/IEC, 2020.
11. NIST Cybersecurity Framework (CSF) 2.0 — NIST, 2024-02-26. https://www.nist.gov/cyberframework
12. NIST AI Risk Management Framework (AI RMF 1.0) and GenAI Profile — NIST, 2024-07-26. https://www.nist.gov/itl/ai-risk-management-framework