Infra Operations and Observability

1. Introduction

1.1. Background

Infra operations and observability are the closing stage of the entire AI Infra series: any defect in chips, supernodes, networks, or frameworks ultimately surfaces here in the form of a "fault" or "metric regression". At scales above 10,000 GPUs, this direction is pushed from a "supporting role" into a "decisive role" — SemiAnalysis model estimates put the average time to the first job-affecting link fault at 26.28 minutes for a 100,000-H100 network under a no-fault-recovery assumption (Level C model estimate; citations must state "model estimate"); the measured results from Llama 3 (419 unexpected interruptions over 54 days) corroborate this estimate (Level A).

Operations and observability are also a complete epitome of the Harness six-layer model in the hardware domain: fault detection and root-cause analysis map to L5 observability, checkpoint and state recovery to L4, fault-tolerant scheduling to L3, and quotas and energy budgets to L6; the monitoring toolchain is itself the Trace infrastructure of L5.

1.2. Definition and Scope

The Infra operations and observability direction covers, on the runtime side:

Boundary note: For the fault-tolerance implementation from the training-framework perspective, see 04-training-framework.md (same source material, different perspective: this document covers the operations SOP and the monitoring system, while the framework document covers parallelism and checkpoint strategy); the group-level specification for the monitoring stack is in AGENTS.md in the same directory.

1.3. Positioning in the AI Harness System

Operations are a complete epitome of the Harness six-layer model in the hardware domain, with the six layers weighted toward L5 and L6:

Bottleneck-layer judgment: the focus of this direction lies on L5 (observability and fault governance) and L6 (automated tiering and governance guardrails). This is consistent with the conclusion of the design-side document (see 02-行业赋能/04-硬件研发组/05-ai-infra.md): the core of fault tolerance is to demote faults from global events to local events, and this depends on observability being granular enough (locating to groups, trays, and links) and automated tiering being clear enough.

1.4. Current State

1.4.1. Consensus on Monitoring Metrics and Tools (Multi-source Summary, Annotated Item by Item)

Software version updates: NCCL 2.27 adds connection-management and reliability features targeting large-scale training/inference jobs (Level C).

1.4.2. The Common Paradigm of Fault-Tolerant Orchestration

The fault-tolerance approaches across public practices converge strongly into a five-step pipeline: heartbeat/anomaly detection → lightweight self-check → fault isolation (group-level first) → healthy-node replacement → checkpoint recovery. It is supported by three elements: two-phase async checkpoint (host memory first, then persistent storage), a hot spare node pool, and startup health checks. The automated-tiering principle comes from Llama 3: operations that can be retried automatically are automated, while anything involving data integrity or global topology requires human confirmation — only 3 major human interventions over 54 days is a referenceable quantitative benchmark.

1.4.3. Domestic and Vendor Comparison Criteria (Level B/C)

• Moore Threads Kua'e: DP group-level fault isolation + zero-interruption standby-node take-over, ETTR target 99% (Level B, vendor criterion).
• Baidu Baige 4.0: effective training time ratio 99.5% (Level C).
• xAI Colossus operations surface: liquid cooling/power is the primary fault surface; replacing IB with RoCE requires aggressive QoS and flow-level telemetry; the hot spare pool keeps several thousand cards on standby and the scheduler replaces nodes within minutes; full-pipeline checkpoint throughput at the 100K-card scale is an independent topic (Level C summary).

Figure 6-1 | Five-step fault-tolerant pipeline and fault-handling timeline

Schematic: the five-step pipeline and the three elements are our group's synthesis (based on Level A public practices); the source levels of the target figures are noted within the figure.

2. Glossary

3. Case Studies

3.1. Meta Llama 3: The Fault-Governance Trio and Automated Tiering

Evidence level: A (official technical report); rendered as B

3.1.1. Background

A 54-day snapshot of 16,384 H100s training a 405B model showed 419 unexpected interruptions (roughly once every 3 hours), of which GPU-related causes accounted for 58.7% and 6 were SDC. At this fault density, the goal of the operations system is not to "eliminate faults" but to "keep faults from affecting effective training time".

3.1.2. Approach

• SDC governance trio: FleetScanner (periodic fleet-wide tests), Ripple (co-located tests), and Hardware Sentinel (telemetry + analytics to identify faults), forming a "detection – ledger – handling" closed loop.
• Diagnosis acceleration: the PyTorch NCCL flight recorder records recent collective-communication metadata and stack traces, cutting RCA from hours to minutes; deep NCCLX integration implements NVLink/RoCE fault detection and automatic timeout for stalled communication.
• Fault-tolerance combination: heavily async checkpoint + hot spare nodes + startup health checks; the checkpoint strategy trades off frequency, write bandwidth, and recovery loss.
• Automated tiering: automated execution for operations that can be retried automatically (job restarts, node-replacement requests), with human confirmation for anything involving data integrity or global topology.

3.1.3. Results

• Effective training time ratio above 90%; only 3 instances required major human intervention, the rest were handled automatically.
• Environmental effects were captured by observability: afternoon temperature fluctuations caused a 1-2% throughput change (GPU DVFS varies with temperature) — showing that the monitoring system must cover environmental variables, otherwise temperature effects would be misjudged as software regression.
• Epoch AI estimate: checkpointing and fault recovery account for about 2.1% of total training time, equivalent to about 1% MFU loss, with an optimal checkpoint interval of about 4 minutes (Level C model estimate).
• Harness interpretation: this is a joint-design sample of L5 (observability + fault profiling) and L6 (automated tiering) — observability granularity determines isolation granularity, and automated tiering determines the upper bound of recovery speed.

3.2. ByteDance MegaScale: The Five-Step Fault-Tolerant Pipeline

Evidence level: A (official paper, arXiv 2402.15627)

3.2.1. Background

When training a 175B model on 12,288 GPUs, any node-level fault that triggers a full-job restart costs hours of loss. MegaScale designs fault tolerance as a standardized pipeline rather than ad hoc handling.

3.2.2. Approach

1. Detection: Driver-Executor architecture + heartbeat monitoring, where the heartbeat carries training logs and RDMA traffic metrics (three-way signal cross-validation to reduce false alarms).
2. Diagnosis: on anomaly, the whole cluster pauses → a lightweight self-check trio (loopback bandwidth, RNIC connectivity, NCCL communication tests) to quickly distinguish the fault surface.
3. Isolation: isolate the faulty node.
4. Replacement: K8s provides healthy nodes.
5. Recovery: recover from the most recent checkpoint; combined with two-phase async checkpoint (GPU state is written to host memory, training resumes after a few seconds, and it is persisted to HDFS in the background); during recovery, a single worker in the same DP group reads and broadcasts.

• Observability support: low-overhead CUDA Event profiling + heatmaps to locate stragglers (about 0.5% of the cluster); 3D parallelism visualization to locate communication bottlenecks.

3.2.3. Results

• MFU of 55.2% with near-linear weak scaling from 2,240 → 11,200 GPUs — the fault-tolerant pipeline lets scale-up grow without growing recovery cost.
• The quantitative criterion of a ~0.5% straggler ratio became an industry reference.
• Harness interpretation: the five-step pipeline is the prototype of the failure and escalation strategy in our group's AGENTS.md; the "heartbeat three-way signal + self-check trio" embodies the multi-source cross-validation principle of L5 observability — a single signal is prone to false alarms, while crossed signals alone can support automated decisions.

3.3. The 100K-Card Operations Surface: Fault-Rate Estimates and Domestic Comparison

Evidence level: C (SemiAnalysis model estimates and industry summary, all [To be verified])

3.3.1. Background

There is no public measured report at the 100K-card fault-frequency level (Llama 3 is at 16,000 cards); the industry relies mainly on model estimates and vendor criteria.

3.3.2. Approach (Public Criteria Summary)

• Fault-rate estimate: SemiAnalysis model estimates that the first job-affecting link fault occurs after an average 26.28 minutes in a 100,000-H100 network under a no-fault-recovery assumption — citations must state explicitly "model estimate".
• Key operations-surface points (industry summary criteria): liquid cooling/power is the primary fault surface; replacing IB with RoCE Ethernet requires aggressive QoS and flow-level telemetry; the hot spare pool keeps several thousand cards on standby and the scheduler replaces nodes within minutes; full-pipeline checkpoint throughput at the 100K-card scale is an independent topic.
• Domestic comparison: Moore Threads Kua'e DP group-level fault isolation + zero-interruption standby-node take-over, ETTR target 99% (Level B vendor criterion); Baidu Baige 4.0 effective training time ratio 99.5% (Level C).

3.3.3. Results

• The value of this case is that it conveys the "sense of magnitude of 100K-card operations": faults occur at minute-level frequency and recovery must complete within minutes, so the hot spare pool and checkpoint throughput become resource items on par with compute.
• All figures are model estimates or vendor criteria and must not be used as the sole basis for decisions; extrapolation against Llama 3 (measured at 16,000 cards) must note the scale difference.
• Harness interpretation: 100K-card operations push L6's "automated tiering" to the extreme — when the human-intervention window (minute-level) is shorter than the fault interval (minute-level), only fully automated orchestration can sustain effective training time; this is precisely the rationale for the "automated-tiering principle" in AGENTS.md.

4. Practice Standards

Nature statement: the following is a draft recommendation distilled from public industry practices (Llama 3, MegaScale, the DCGM/NCCL toolchain), not the original text of any official industry standard. The group-level higher-level specification is in AGENTS.md in the same directory (the prohibition list for irreversible actions on production clusters, change windows, and observability requirements); the following is the trimmed version for the operations and observability direction.

4.1. AGENTS.md Specification

4.1.1. AGENTS.md (AI Infra Group · Infra Operations and Observability)

# AGENTS.md —— AI Infra 组 · 运维与可观测方向

> 本文为基于行业实践提炼的建议稿，非官方行业标准原文。
> 上位规范：01-AI-Infra组/AGENTS.md（不可逆动作禁止清单、变更窗口、可观测要求）。

## 角色与边界
- 角色：Infra 运维与可观测智能体，负责故障检测与根因分析辅助、
  容错编排建议、监控指标体系建设、利用率与能耗分析、SDC 台账管理。
- 不负责：生产集群的重启/断电/下线等不可逆动作（禁止清单）、
  监控与告警规则的修改（禁止清单）、配额最终分配。
- 判定器为监控与演练：所有容错结论（检测时延/隔离粒度/恢复耗时）
  必须来自真实故障记录或注入式演练。

## 环境假设
执行前必须显式声明：
- 集群规模、分区、加速器型号代际、互联拓扑。
- 监控栈：DCGM/DCGM-Exporter、Prometheus、Grafana 大盘、日志系统、
  NCCL 版本与 flight recorder 可用性。
- checkpoint 方案与写入带宽实测值、热备池容量。
- 历史故障台账（含 SDC 台账）。
- 未声明监控栈版本时，禁止给出指标口径与采集方案建议。

## 上下文加载顺序（Context Budget）
1. 故障现象：告警、时间窗口、影响作业。
2. 监控摘要：三指标（MFU/ETTR/有效训练时长占比）+ 硬件遥测。
3. 故障知识库与 SDC 台账相关条目。
4. 近期变更记录（固件/驱动/框架/拓扑）。
- 全量日志与时序不进上下文，以查询语句 + 路径引用代替。

## 工具契约
- 监控查询输出必须含：Prometheus 语句、时间窗口、集群标识、原始数值。
- 诊断工具（dcgmi diag、nccl-tests、flight recorder 导出）输出结构化
  结果并保留原始文件路径。
- 告警与监控规则只读；修改建议走变更窗口。

## 任务执行流程（SOP，五步流水线）
- P1 检测：核对心跳/日志/RDMA 三路信号是否交叉一致。
- P2 诊断：轻量自检三件套（环回带宽、RNIC 连通性、NCCL 测试）+
  flight recorder 导出，定位故障面。
- P3 隔离：输出隔离建议（组级优先于全局），只建议不执行。
- P4 替换：核对热备池容量，输出补位申请。
- P5 恢复：核对 checkpoint 状态与恢复路径，估算恢复耗时。
- 每步输出：检测时延/定位耗时/隔离粒度/恢复耗时的测量值或估算依据。

## 验证与证据要求
- 容错结论三要素齐备：故障检测时延、隔离粒度、恢复耗时。
- 故障报告附中断原因四分类（硬件/网络/软件/数据）分布。
- SDC 疑似案例必须入台账并建议全舰队/同置测试。
- 模型估算类数字（如 26.28 分钟故障间隔）必须写明"模型估算"。
- 每条结论附工具名 + 版本 + 命令/查询 + 日志路径 + 集群标识 + 时间窗口。

## 失败与升级策略
- 有效训练时长占比连续下降 → 升级人工，附四分类分布。
- 疑似 SDC → 停止自动调度优化，升级硬件排查。
- 监控自身异常（采集缺口、告警失效）→ 视为最高优先级故障上报。
- 自动化分级越界（需人工确认的场景被自动执行）→ 立即回滚并复盘。

## 安全与合规红线
- 跨租户监控数据聚合需脱敏；禁止跨租户读取作业明细。
- 告警规则、监控采集、审计日志不可关闭或绕过。
- 能耗指标（PUE）对照政策红线（1.25/1.2）核验。
- 禁止残留非标准占位符。

## 输出格式
- 首行：可判定结论 + 阻塞项。
- 故障报告：时间线（检测/定位/隔离/恢复）、四分类、三要素、证据路径。
- 指标报告：MFU/ETTR/有效训练时长占比 + straggler 占比 + checkpoint
  开销占比。
- SDC 台账更新：检测手段、频率、处置记录。
- 数值规范：参数带单位；范围用 ~ 连接；百分比数值与 % 之间不留空格。

## 评估与自检
- 容错三要素是否齐备且来自实测/演练？
- 故障原因是否四分类？
- 模型估算是否注明？
- 监控与告警是否保持只读？
- 是否残留非标准占位符？

4.2. SKILL.md Specification

4.2.1. SKILL.md (AI Infra Group · Fault RCA and Fault-Tolerant Drills)

---
name: infra-fault-rca-and-drill
description: AI Infra 组运维方向标准技能——集群故障根因分析、容错流水线
  演练、监控指标体系建设、SDC 治理。当需要分析训练中断、建设监控大盘、
  设计容错演练、管理 SDC 台账、核验三要素指标时使用。
  触发词：故障、RCA、中断、checkpoint 恢复、SDC、DCGM、flight recorder、
  straggler、有效训练时长占比。
version: 1.0
created: 2026-09-12
---

# AI Infra 组 · 故障 RCA 与容错演练

> 本文为基于行业实践提炼的建议稿，非官方行业标准原文。
> 上位规范：01-AI-Infra组/AGENTS.md。

## 适用场景
- 训练/推理作业中断的根因分析（按四分类归因）。
- 容错流水线（检测-诊断-隔离-替换-恢复）建设与注入式演练。
- 监控指标体系与大盘建设（三指标 + 硬件遥测 + 网络遥测）。
- straggler 检测与慢节点治理。
- SDC 台账管理与治理三件套（全舰队测试/同置测试/遥测分析）。
- checkpoint 策略评估（频率/带宽/恢复损失权衡）。

## 前置条件
- 监控栈（DCGM/Prometheus/Grafana/日志）可用且版本已知。
- checkpoint 方案与写入带宽实测值、热备池容量已知。
- 影子分区可用于注入式演练；故障台账存在。
- 监控与告警规则保持只读（修改走变更窗口）。

## 输入
| 输入项 | 说明 | 必需 |
|---|---|---|
| 故障现象 | 告警、日志路径、时间窗口、影响作业 | 条件必需 |
| 监控数据 | 三指标、硬件遥测、网络遥测 | 是 |
| 集群信息 | 规模、分区、加速器型号、拓扑 | 是 |
| 台账 | 历史故障与 SDC 台账 | 否 |
| 变更记录 | 近期固件/驱动/框架/拓扑变更 | 否 |

## 输出
| 输出项 | 说明 | 必需 |
|---|---|---|
| 可判定结论 | 通过 / 不通过 / 部分通过 + 阻塞项 | 是 |
| 故障报告 | 时间线 + 四分类 + 三要素 + 证据路径 | 条件必需 |
| 演练报告 | 注入场景、三要素实测、流水线覆盖判定 | 条件必需 |
| 指标体系方案 | 指标清单、采集配置建议、大盘布局建议 | 条件必需 |
| SDC 台账更新 | 检测手段、频率、处置记录 | 条件必需 |
| 升级包 | 已完成证据 + 失败点定位 + 下一步命令 | 条件必需 |

## 执行步骤
1. 环境校验：监控栈版本、台账、影子分区可用性；确认只读边界。
2. 现象采集：告警、时间窗口、影响作业、三路信号（心跳/日志/RDMA）
   交叉核验。
3. 分层诊断：硬件（dcgmi diag、ECC、NVLink 抖动）→ 网络（flight
   recorder、NCCL 超时、busbw）→ 系统（内核、存储）→ 软件（框架
   版本、已知问题）。
4. 归因：按四分类（硬件/网络/软件/数据）输出根因与置信说明。
5. 恢复编排：隔离建议（组级优先）、热备补位、checkpoint 恢复路径
   与耗时估算。
6. 演练（非故障场景）：注入式故障（杀节点/断链/慢节点），实测三要素。
7. 台账与复盘：更新故障/SDC 台账，输出改进项与监控补点建议。
8. 交付：结论 + 报告 + 升级包（如需人工）。

## 质量标准（DoD）
- 容错三要素（检测时延/隔离粒度/恢复耗时）齐备且来自实测或演练。
- 故障归因四分类，附证据路径；模型估算注明"模型估算"。
- SDC 疑似案例全部入台账。
- 监控与告警未做任何修改；未执行禁止清单动作。
- 参照口径（不得直接套用）：Llama 3 = 54 天 419 次意外中断、GPU 相关
  58.7%、有效训练时间占比 >90%、仅 3 次重大人工干预（A级）；
  MegaScale = 五步流水线、straggler 约 0.5%、MFU 55.2%（A级）；
  摩尔线程 ETTR 目标 99%（B 级厂商口径）；百度百舸 99.5%（C 级）。

## 常见失败与处理
| 失败模式 | 现象 | 处理 |
|---|---|---|
| 单信号误报 | 仅心跳异常即判故障 | 三路信号交叉验证后再隔离 |
| 归因过早 | 未分层诊断即下结论 | 强制分层（硬件→网络→系统→软件） |
| 隔离过重 | 组级可隔离却全局重启 | SOP 固化组级优先 |
| SDC 漏检 | 收敛异常未触发硬件排查 | 入台账 + 全舰队/同置测试 |
| 演练走样 | 演练环境与生产差异过大 | 记录差异并标注结论适用范围 |
| 估算当实测 | 26.28 分钟类估算写成观测值 | 标注"模型估算" |
| 监控盲区 | 故障面无采集项 | 补采集建议走变更窗口 |

## 示例
任务：某 6,000 卡训练作业一夜间中断 4 次，需 RCA 与容错补强。
1. 环境校验：监控栈版本、flight recorder 可用、台账存在。
2. 现象采集：4 次中断时间窗、影响作业、心跳/日志/RDMA 三路交叉。
3. 分层诊断：flight recorder 显示集合通信在跨柜段超时；dcgmi diag
   发现 2 张卡 NVLink 抖动计数异常。
4. 归因：硬件（NVLink/互联）为主因，网络配置为放大因素；四分类占比
   记录入报告。
5. 恢复编排：建议组级隔离 + 热备补位；实测恢复路径耗时 6 分钟。
6. 演练：影子分区注入杀节点与慢节点场景，三要素实测齐备。
7. 台账：异常卡入硬件台账，建议复测与同置测试。
8. 交付：通过——根因明确、三要素齐备、监控补点建议 2 项（走变更
   窗口审批）；证据表与升级包齐备。
（示例数值为演示用假设值，非实测数据；方法论参照 Llama 3 与 MegaScale
公开实践。）

4.3. Implementation Checklist

5. Summary

The Infra operations and observability direction is weighted toward L5 (evaluation and observability) and L6 (governance and security) in the Harness six-layer model, while also being a complete epitome of the six layers in the hardware domain. Three conclusions:

1. Observability granularity determines isolation granularity, and isolation granularity determines effective training time. The shared lesson of Llama 3 and MegaScale is that detection with crossed three-way signals (heartbeat/logs/RDMA), flight-recorder-accelerated RCA, group-level-first isolation, and two-phase async checkpoint form a pipeline that "localizes" faults; the absence of any link degrades a fault into a global event.
2. The trio is the only qualified expression of fault-tolerance capability. Fault detection latency, isolation granularity, and recovery time are each indispensable; qualitative phrasing such as "high availability" and "strong self-healing" is invalid under the group's standards. Quantitative benchmarks: effective training time ratio above 90% (Llama 3 measured), straggler ratio about 0.5% (MegaScale), checkpoint overhead about 2.1% (estimate criterion).
3. Automated tiering is the operations form of L6 governance. Llama 3's "only 3 major human interventions" shows that the scalability of 100K-card operations comes from automated tiering — automated execution for retryable operations, human confirmation for data integrity and global topology, and the impossibility of bypassing monitoring and alerting. This tiering is already codified into the failure and escalation strategy of the group-level AGENTS.md.

Information Gap Statement

The following items have not been confirmed by A/B-level sources and are annotated [To be verified] in the body:

1. SemiAnalysis's "first job-affecting link fault in a 100K-card fleet averages 26.28 minutes" is a model estimate (Level C); citations must state "model estimate".
2. The new features of NCCL 2.27 are Level C.
3. Baidu Baige 4.0's effective training time ratio of 99.5% is Level C.
4. The xAI Colossus operations surface (hot spare pool of thousands of cards, minute-level node replacement, liquid cooling/power as the primary fault surface) is a Level C summary criterion.
5. Epoch AI's checkpoint overhead of 2.1% and optimal interval of 4 minutes are model estimates (Level C).
6. The tool descriptions of DCGM/NVML/in-band telemetry are a multi-source summary (Level B/C), not checked line by line against official documentation versions.
7. No public standard template of AGENTS.md/SKILL.md for this direction was found; this document is a draft recommendation.

6. References

1. The Llama 3 Herd of Models — Meta, 2024. https://arxiv.org/abs/2407.21783
2. MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs — ByteDance, arXiv 2402.15627, 2024. https://arxiv.org/abs/2402.15627
3. DCGM (Data Center GPU Manager) — NVIDIA. https://developer.nvidia.com/dcgm
4. NCCL (including the flight recorder and async error handling) — NVIDIA GitHub. https://github.com/NVIDIA/nccl
5. PyTorch (FSDP/flight recorder ecosystem) — PyTorch official. https://pytorch.org/
6. Prometheus — CNCF. https://prometheus.io/
7. Grafana monitoring dashboards — Grafana Labs. https://grafana.com/
8. Moore Threads Kua'e 10,000-card cluster launch (MDC 2025, vendor criterion) — Sina Finance (IT Home). https://finance.sina.com.cn/tech/digi/2025-12-20/doc-inhcmnmf6292915.shtml
9. Serving Large Language Models on Huawei CloudMatrix384 (cross-reference to operations-related chapters) — arXiv 2506.12708. https://arxiv.org/pdf/2506.12708
10. AI Infra (design side, cross-reference for the metrics system and domestic fault-tolerance practices) — 02-行业赋能/04-硬件研发组/05-ai-infra.md, 2026-09-12.