AI Infra

1. Introduction

1.1. Background

AI Infra (AI infrastructure) is the physical and software foundation that hosts large-model training and inference. Its special nature: it is the most "self-referential" scenario of AI Harness in the hardware domain — the AI carried by Harness is itself taking part in designing the hardware that carries AI.

This flywheel has been confirmed by multiple public cases: Google uses AlphaChip to design TPUs, the TPUs train better models, and the better models in turn design better chips; NVIDIA uses PrefixRL to design circuits in the Hopper GPU; OpenAI uses an internally extended Codex to contribute to the design of its in-house inference chip, Jalapeño.

From an engineering standpoint, the pressure on AI Infra comes from three directions:

1. Scale pressure. Frontier models (especially MoE architectures) need to spread experts across far more accelerators than before, making per-card HBM capacity and bandwidth a bottleneck.
2. Reliability pressure. At the 10,000-card scale, failures become the norm rather than the exception, and the ability to recover from failures directly determines effective training time.
3. Energy and cost pressure. Data centers are constrained by power and cost, making liquid cooling, PUE, and per-token cost the core competitive metrics.

1.2. Definition and Scope

The AI Infra direction covers the full stack of a compute cluster, from hardware selection to job scheduling; in the AI Harness context it specifically refers to what is participated in or led by agents:

Boundary notes: This direction is tightly coupled with Packaging — HBM and CoWoS are the physical bottlenecks of AI accelerators, and the 13.4 TB of HBM3E in GB200 NVL72 directly determines the model scale that can be hosted; in turn, the demands of AI Infra drive the expansion of CoWoS capacity. The coupling with Test is reflected in Advantest ACS RTDI bringing NVIDIA's AI inference engine into semiconductor production lines, an example of AI Infra technology feeding back into chip testing.

1.3. Position within the AI Harness System

图 1-1｜AI Infra 在 AI Harness 六层模型中的定位与瓶颈

数据来源：基于本文分析绘制的示意图。

Core assessment: the focus of the AI Infra direction lies in L3 orchestration (training fault tolerance and elastic scheduling) and L5 evaluation (MFU / ETTR / PUE / effective training time ratio).

Moore Threads' "DP group-level fault isolation + zero-interruption standby node integration" is a textbook implementation of L3+L4: only the data-parallel group containing the affected nodes is isolated, the other groups continue training, and after a standby node is integrated only the corresponding links need to be rebuilt — with no overall training interruption at any point. Baidu Baige 4.0's claimed "effective training time ratio of 99.5%" is a key L5 metric (this figure is level C).

Needs special attention: MFU, ETTR, and effective training time ratio are the three core quantifiable metrics of the AI Infra direction. Any statement about cluster efficiency should be grounded in these three metrics, avoiding undecidable vague phrasing such as "performance improved" or "efficiency increased".

1.4. Current Status

1.4.1. Standards Status: De Facto Standards as the Norm

No IEEE / ISO / JEDEC standard dedicated to AI compute clusters was found for this direction (excluding component-level standards such as HBM and UCIe). This is a notable difference between this direction and the other four; when writing and implementing, do not fabricate standard numbers.

The current normative form of this direction is de facto standards and vendor specifications:

In addition, there is China's "Training Cluster Comprehensive Performance Evaluation Standard" draft for comments (end of 2025), which incorporates training stability, communication latency jitter, and power deviation rate into the entry metrics; as well as mandatory regulations on cluster energy efficiency ratio, carbon footprint, and data security across 13 countries and regions (2025) — both are level C.

1.4.2. Key Quantitative Parameters

1.4.3. Domestic Cluster Scale-Up Status (Multiple Sources, All [To be verified])

The following entries all come from secondary reprints or vendor marketing, and are not confirmed by primary official sources:

• Huawei Ascend: In 2026-07, the first "national chip training national model" Ascend 10,000-card cluster in the Guangdong-Hong Kong-Macao Greater Bay Area (Shaoguan, Guangdong) came online, with 30 supernodes, 11,520 cards, and total compute of 9,000P; more than 750 sets of Ascend 384 supernodes commercially deployed; Atlas 950 SuperCluster scalable to 500,000 cards; Ascend 950 supernodes forming a 256 TB shared high-speed memory pool (WAIC 2026).
• Shenzhen: China's first 10,000-card Huawei Ascend 910C cluster, with 11,000 petaflops of compute.
• Alibaba: T-Head Zhenwu chip 10,000-card cluster.
• Cambricon: Siyuan 690 in mass production (dual-die packaging, FP16 >700 TFLOPS, 196 GB HBM3).
• Sugon: Sugon 8000, China's first fully domestic 100,000-card supercluster.
• Shipments (IDC): total AI accelerator card shipments in the China market in 2025 were about 4 million; domestic vendors about 1.65 million, with the domestic share breaking 40% for the first time (~41%); NVIDIA about 2.2 million (~55%); Huawei Ascend about 812,000 (20.3%, No. 1 among domestic vendors).

2. Glossary

3. Case Studies

3.1. NVIDIA GB200 NVL72: The Official Benchmark for Rack-Scale Systems

Evidence level: A (NVIDIA official website and official developer blog)

3.1.1. Background

Frontier models such as MoE need to spread experts across far more accelerators than before, making per-card HBM capacity and bandwidth a bottleneck; at the same time, data centers are constrained by power and cost.

3.1.2. Approach

A rack-scale system, where 36 Grace CPUs + 72 Blackwell GPUs are woven through NVLink Switch into a single large-scale NVLink interconnect fabric. Architecture highlights:

• Superchip interconnect: each GB200 superchip uses NVLink-C2C at 900 GB/s bidirectional bandwidth to connect 2 Blackwell GPUs with 1 Grace CPU, providing unified memory space.
• Compute tray (MGX design): contains 2 Grace + 4 Blackwell, with cold plates and liquid-cooling interfaces, PCIe Gen 6, and NVLink cable boxes.
• Switch fabric: 9 NVLink switch trays (144 100 GB ports per tray) fully connecting the 18 NVLink ports of the 72 GPUs.
• Cooling and packaging: dense all-copper cable-box packaging + liquid-cooling system design.
• Software stack: NVIDIA Dynamo disaggregated serving orchestration + NVFP4 + TensorRT-LLM / SGLang / vLLM.

3.1.3. Results

NVIDIA official data (all level A):

• NVFP4 Tensor Core: 1,440 / 720 PFLOPS (sparse/dense).
• FP8/FP6: 720 PFLOPS; INT8: 720 POPS; FP16/BF16: 360 PFLOPS; TF32: 180 PFLOPS; FP32: 5,760 TFLOPS; FP64: 2,880 TFLOPS.
• GPU memory 13.4 TB HBM3E, bandwidth 576 TB/s; NVLink bandwidth 130 TB/s.
• CPU: 2,592 Arm Neoverse V2 cores; 17 TB LPDDR5X / 14 TB/s.
• A single NVLink domain supports up to 576 GPUs, total bandwidth exceeding 1 PB/s, fast memory exceeding 240 TB.
• Compute tray: 80 PFLOPS AI performance + 1.7 TB fast memory.
• Compared with H100: LLM training 4x faster, inference 30x faster; liquid cooling reduces cost and energy by 25x.
• MoE scenario: 72 GPUs work together like one, 1.4 exaflops AI performance + 30 TB fast shared memory; a 10x leap in per-watt performance, and per-token cost reduced by more than 10x; SemiAnalysis InferenceMax (DeepSeek-R1) shows a cost reduction of more than 10x per million tokens compared with H200.

Deployments: AWS, Core42, CoreWeave, Crusoe, Google Cloud, Lambda, Microsoft Azure, Nebius, Nscale, Oracle Cloud Infrastructure, Together AI, and others; customer cases include DeepL (training MoE models on GB200), CoreWeave, and Together AI.

3.2. Moore Threads Kua'e 10,000-Card Cluster: Domestic Full-Stack and Training Fault Tolerance

Evidence level: B (vendor launch data; must be noted as vendor figures)

3.2.1. Background

Domestic accelerator cards must deliver efficient and stable AI training and inference at the 10,000-card scale, and prove compatibility with the mainstream international ecosystem. The core engineering challenge at the 10,000-card scale is not peak compute, but effective training time under normalized failure conditions.

3.2.2. Approach

The MUSA (Meta-computing Unified System Architecture) full stack is self-developed in-house:

• Bottom layer: full-featured GPUs (5th-generation "Huagang" architecture, compute density +50%, a 10x leap in compute energy efficiency, supporting full precision from FP4 to FP64, adding MTFP6/MTFP4 and mixed low-precision acceleration, with TCE-PAIR technology letting two tensor compute engines share data).
• Middle layer: the Kua'e intelligent computing cluster, with more than 1,000 compute nodes per cluster, each node with 8 self-developed OAM modular GPUs, and a 3D fully-interconnected topology achieving sub-microsecond communication latency; the new-generation Scale-up system supports direct connection of 1,024 GPUs per supernode.
• Top layer: the MUSA 5.0 software stack, deeply adapted to PyTorch / PaddlePaddle, with Jax / TensorFlow / TileLang newly added; the MUSIFY automated porting tool converts CUDA-ecosystem C++ source into MUSA-architecture source.
• Training fault-tolerance system (the most valuable part of this case study for Harness): aims to raise ETTR to 99%; supports online diagnosis of training anomalies (catching unresponsive nodes and slow nodes in real time); DP (data parallelism) group-level fault isolation — only the group containing the affected nodes is isolated, the other groups continue training, and after a standby node is integrated only the corresponding links need to be rebuilt, with no overall training interruption at any point.

3.2.3. Results

Vendor launch figures (MDC 2025, 2025-12-20, Beijing):

• Floating-point compute 10 Exa-Flops.
• Dense large-model MFU 60%; MoE large-model MFU 40%.
• Effective training time ratio >90%; training linear scaling efficiency 95%.
• Flash Attention compute utilization >95%; HGEMM operator efficiency 98%; communication efficiency 97%.
• Inference side (with SiliconFlow, DeepSeek R1 671B full model): MTT S5000 single card Prefill >4000 tokens/s, Decode >1000 tokens/s.

Harness interpretation: DP group-level fault isolation is a typical implementation of L3 orchestration working in concert with L4 memory and state — it downgrades a "failure" from a global event to a local event, thereby decoupling fault-recovery cost from cluster scale. This is the key to moving a 10,000-card cluster from "can run" to "can keep running".

3.3. Engineering Challenges of 100,000-Card Clusters

Evidence level: C, all [To be verified] (a multi-source summary of Chinese technical long-form articles; it is recommended to verify the original official sources)

3.3.1. Background

As cluster scale is pushed from a thousand cards to a hundred thousand cards, failures change from "abnormal" to "the norm". Training interruptions and rollbacks become the primary factor affecting effective training time.

3.3.2. Approach

The engineering measures adopted by vendors are highly consistent: faster fault detection, finer-grained fault isolation, faster checkpointing and recovery, and more efficient networking and cooling.

3.3.3. Results

The following figures all come from secondary reprints and have not been confirmed by primary official sources; the nature of the source must be noted when citing:

• xAI Colossus: 100,000 H100s (another figure claims 200,000; the two figures conflict and are presented in parallel here), built in 122 days, liquid-cooled, with a custom RDMA network; during debugging it encountered BIOS mismatches, network card faults, and other issues; PUE dropped below 1.05 (another figure of 1.1); it leased a quarter of the United States' mobile cooling capacity, converted an abandoned factory, and introduced Tesla Megapack energy storage to cope with power fluctuations.
• Tencent HCC: supports 100,000 GPUs based on the Xingmai network, cold-plate liquid-cooled PUE <1.1, MFU 65%; Xingmai Network 2.0 uses the self-developed TiTa protocol, supporting 3.2 TB/s bandwidth and 100,000-card networking.
• Alibaba Cloud Lingjun: three-tier Clos topology, network latency reduced to 2 microseconds, a single-network cluster scaled to 100,000 cards, model compute utilization +20%.
• Baidu Baige 4.0: heterogeneous chip scheduling + automatic elastic training and distributed fault tolerance, effective training time ratio 99.5%; memory optimization techniques raise training efficiency by 30%.
• Meta Llama 3: a 16,000-card cluster experiences a failure every 3 hours, requiring rollback to checkpoints.
• OpenAI GPT-4.5: a 100,000-card cluster exposed hidden infrastructure faults; the systems team fixed them while continuing to train, ultimately raising compute utilization to 60%.
• Energy: H100 single card 700W; a 100,000-card cluster has total power consumption exceeding 70 MW; per-rack power density jumps from 15 kW to 100 kW.

Value and risks of this case study: these figures portray the real engineering difficulty of 100,000-card clusters (especially the data point of Meta experiencing a failure every 3 hours), and are directly relevant to designing Harness's L3 fault-tolerance orchestration. However, because their sources are secondary reprints, they must not be used as the sole basis for decisions; it is recommended to verify each vendor's original official sources before citing.

4. Practice Standards

Nature Statement: The following AGENTS.md and SKILL.md are proposed drafts distilled from public industry practices, not the original text of official industry standards. As of the time this document was written, no public reference template of AGENTS.md or SKILL.md targeting the AI Infra domain was found, nor any dedicated international or national standard for this domain.

4.1. AGENTS.md Specification

4.1.1. AGENTS.md (AI Infra Domain)

# AGENTS.md —— AI Infra

> 本文为基于行业实践提炼的建议稿，非官方行业标准原文。
> 本方向未检索到 IEEE / ISO / JEDEC 专用标准，规范形态以事实标准与厂商规范为主
> （MGX、NVLink / NVLink-C2C、Quantum-X800、Spectrum-X800、BlueField-3、Dynamo、OAM）。
> 禁止虚构标准编号。

## 角色与边界
- 角色：AI Infra 智能体，负责集群作业调度辅助、训练容错编排、性能与能效调优、
  推理服务扩缩容与异常根因定位。
- 判定器为监控系统与基准测试：所有 MFU、ETTR、PUE、有效训练时长占比结论必须由
  真实监控数据或基准测试结果给出，模型不得估算。
- 不负责：集群扩容的商务决策、配额的最终分配、生产环境配置变更的最终放行。
- 生产环境配置变更默认不授予直接执行权限，只产出待审批方案。

## 环境假设
执行前必须显式声明：
- 加速器型号与数量、单卡显存与带宽、单卡功耗。
- 互联拓扑与代际：NVLink / NVLink-C2C 代际、InfiniBand 或 RoCE、交换层级与带宽。
- 服务器形态：MGX 参考设计或等效规范；是否采用 OAM 模组。
- 调度器与版本；监控与日志系统；检查点存储与带宽。
- 散热方式：风冷 / 冷板式液冷 / 浸没式液冷；设计 PUE 与目标 PUE。
- 软件栈：训练框架与版本、推理框架（如 TensorRT-LLM / SGLang / vLLM）、
  编排框架（如 NVIDIA Dynamo）。
- 多租户场景须声明租户隔离策略与配额规则。
- 未声明加速器型号与拓扑时，禁止产出与 MFU、扩展效率、时延相关的任何结论。

## 上下文加载顺序（Context Budget）
1. 集群拓扑与资源现状（节点、GPU、网络、存储）。
2. 作业描述：模型结构、并行策略、批量大小、检查点频率。
3. 监控指标摘要：MFU、吞吐、通信效率、功耗、温度。
4. 故障知识库条目与历史作业画像。
5. 近期变更记录（固件、驱动、框架、拓扑变更）。
- 全量监控时序与完整日志不进上下文，以路径引用 + 查询接口代替。

## 工具契约
- 优先为调度器、监控系统、固件工具封装 MCP 服务器，通过标准化接口调用。
- 工具返回值必须包含：退出码、日志路径、关键指标数值、目标集群标识、工具版本。
- 配置变更必须可回滚，且保留上一版本与差异说明。
- 影响面变更（拓扑、固件、调度策略）必须先在小范围验证再推广。
- 不可逆工具（生产配置下发、固件升级、集群重置）默认不授予直接调用权限。

## 任务执行流程（SOP）
- S1 需求解析：把目标分解为可判定指标（MFU、ETTR、PUE、有效训练时长占比）。
- S2 基线建立：记录基线 MFU、吞吐、通信效率、功耗、故障率。
- S3 候选生成：产出候选调度策略、并行策略、扩缩容方案，每个附理由与预期影响。
- S4 受控验证：在小范围或非生产环境执行，得到客观数值。
- S5 差异分析：与基线对比，改善项与恶化项同时报告。
- S6 风险评估：评估对稳定性、配额、能耗预算的影响。
- S7 证据打包：命令、脚本、日志路径、集群标识、数值、工具版本齐备。
- S8 人工确认：生产配置变更进入人工审批队列。

## 验证与证据要求
- 任何效率结论必须落在三个核心指标上：MFU、ETTR、有效训练时长占比；
  禁止使用"性能提升""效率提高"等不可判定的模糊表述。
- 能耗结论须给出 PUE 与实测功耗，禁止只报相对改善百分比。
- 扩展效率须说明基线规模与对比规模，禁止跨规模直接比较绝对值。
- 容错能力须给出故障检测时延、隔离粒度、恢复耗时三项，缺一不可。
- 模型类建议须附置信说明与样本量（覆盖多少作业、多少节点时）。
- 每条结论须附：工具名 + 版本 + 命令或脚本 + 日志路径 + 关键数值 + 集群标识。

## 失败与升级策略
- 有效训练时长占比连续下降 → 升级人工，输出中断原因分布（硬件 / 网络 / 软件 / 数据）。
- 出现无法归因的慢节点或静默数据损坏 → 停止自动调度优化，升级人工排查。
- 能耗接近或超出预算上限 → 触发功率封顶（power capping），并上报。
- 配置变更在小范围验证未达预期 → 回滚，不得推广。
- 升级时须交付：已完成证据包 + 失败点定位 + 下一步可执行的具体命令。

## 安全与合规红线
- 多租户场景必须严格隔离：不得跨租户读取作业数据、模型权重或日志。
- 自主智能体必须在隔离运行时内执行。
- 生产环境配置变更必须有人工确认与可回滚方案，禁止无人值守下发。
- 配额与预算须设置护栏（Budget Guardrail），超限即熔断。
- 固件与驱动升级须遵循厂商兼容矩阵，禁止跨版本强升。
- 集群能效与数据安全的属地法规须遵守（相关强制法规条目当前级别 C，[待核实]）。
- 所有工具调用、配置变更、人工确认写入不可篡改审计日志。

## 禁止事项
- 禁止编造或臆测 MFU、ETTR、PUE、故障率、吞吐数值。
- 禁止输出"应该没问题""大概率稳定"等不可判定表述。
- 禁止在未声明加速器型号与拓扑的前提下给出效率结论。
- 禁止跨规模直接比较绝对吞吐。
- 禁止跨租户读取数据。
- 禁止无人确认下发生产环境配置变更。
- 禁止虚构标准编号；本方向无专用国际或国家标准，须如实说明以事实标准为主。
- 禁止引用未获确认的第三方集群数据作为基准。

## 输出格式
- 首行给出可判定结论（通过 / 不通过 / 部分通过 + 阻塞项）。
- 证据表：工具、版本、命令或脚本、日志路径、集群标识、关键数值、与基线差值。
- 三指标报告：MFU、ETTR、有效训练时长占比同时列出。
- 能耗报告：PUE、实测功耗、与预算的差值。
- 风险与假设：列出结论依赖的作业类型、规模、软件版本。
- 待人工确认项：生产配置变更、配额调整、扩容建议。
- 数值规范：参数带单位；范围用 ~ 连接；百分比数值与 % 之间不留空格。

## 评估与自检
- 本轮所有数值是否来自真实监控或基准测试输出？
- 是否给出了 MFU、ETTR、有效训练时长占比三项核心指标？
- 是否只报了相对改善百分比而未报绝对值与基线？
- 扩展效率是否说明了基线规模与对比规模？
- 容错建议是否包含检测时延、隔离粒度、恢复耗时三项？
- 是否存在跨租户数据访问？
- 生产配置变更是否有人工确认与回滚方案？
- 输出中是否残留 XX、___ 等非标准占位符？
- Harness 或提示词变更后，是否在固定基准作业集上回归并报告与上一版本的差值？

4.2. SKILL.md Specification

4.2.1. SKILL.md (AI Infra · Training Fault-Tolerance Orchestration and Cluster Efficiency Tuning)

---
name: ai-infra-fault-tolerance-and-efficiency
description: AI Infra 方向的训练容错编排与集群效率调优技能。当需要提升 MFU、
  缩短故障恢复时间、提高有效训练时长占比、优化推理服务编排或能耗，
  并要求结论落在可量化指标上且配置变更可回滚时使用。
version: 1.0
created: 2026-09-12
---

# AI Infra · 训练容错编排与集群效率调优

> 本文为基于行业实践提炼的建议稿，非官方行业标准原文。
> 本方向以事实标准与厂商规范为主，禁止虚构标准编号。

## 适用场景
- 大规模训练作业频繁中断，需要提升有效训练时长占比与 ETTR。
- MFU 偏低，需要定位瓶颈（计算 / 通信 / 数据加载 / 调度碎片）。
- 并行策略（DP / TP / PP）组合需要搜索与比选。
- 慢节点与静默故障检测。
- 推理服务的 prefill / decode 分离编排与自动扩缩容。
- 能耗与 PUE 优化、功率封顶策略。

## 前置条件
- 已声明加速器型号与数量、互联拓扑代际、服务器形态、调度器与监控版本。
- 存在基线数据：MFU、吞吐、通信效率、故障率、有效训练时长占比、PUE。
- 检查点机制可用，且检查点写入带宽已测量。
- 配置变更具备回滚方案与回滚演练记录。
- 智能体运行于隔离运行时；多租户隔离策略已配置。
- 具备判定器：监控指标采集、基准测试、日志分析。

## 输入
| 输入项 | 说明 | 必需 |
|---|---|---|
| 集群拓扑与资源 | 节点数、加速器数、网络层级与带宽、存储带宽 | 是 |
| 作业描述 | 模型结构、并行策略、批量大小、检查点频率 | 是 |
| 基线指标 | MFU、吞吐、通信效率、故障率、有效训练时长占比、PUE | 是 |
| 监控数据 | 时序指标、日志路径、告警记录 | 是 |
| 故障知识库 | 历史故障模式与处置记录 | 否 |
| 约束条件 | 配额上限、能耗预算、SLO | 是 |
| 变更记录 | 近期固件、驱动、框架、拓扑变更 | 否 |

## 输出
| 输出项 | 说明 | 必需 |
|---|---|---|
| 可判定结论 | 通过 / 不通过 / 部分通过 + 阻塞项 | 是 |
| 三指标报告 | MFU、ETTR、有效训练时长占比，含与基线差值 | 是 |
| 容错能力报告 | 故障检测时延、隔离粒度、恢复耗时 | 是 |
| 能耗报告 | PUE、实测功耗、与预算差值 | 是 |
| 变更清单 | 调度策略、并行策略、扩缩容方案，附理由与回滚方案 | 是 |
| 证据表 | 工具、版本、命令或脚本、日志路径、集群标识、数值 | 是 |
| 待人工确认项 | 生产配置变更、配额调整、扩容建议 | 是 |

## 执行步骤
1. 环境校验：核对加速器型号、拓扑代际、调度器与监控版本、多租户隔离；缺失即停止。
2. 基线确认：确认 MFU、吞吐、通信效率、故障率、有效训练时长占比、PUE 与测量窗口。
3. 瓶颈定位：分解耗时为计算、通信、数据加载、调度碎片四类。
4. 候选生成：产出候选调度策略、并行策略、检查点频率、扩缩容方案，每个附理由。
5. 受控验证：在小范围或非生产环境执行，记录命令、退出码、日志路径、集群标识。
6. 差异分析：与基线对比，改善项与恶化项同时报告。
7. 容错评估：评估故障检测时延、隔离粒度、恢复耗时三项。
8. 能耗评估：评估 PUE 与实测功耗，核对预算。
9. 证据打包与交付：给出回滚方案，列出待人工确认项，写入审计日志。

## 质量标准（DoD）
- 结论落在 MFU、ETTR、有效训练时长占比三项核心指标上，无模糊表述。
- 能耗给出 PUE 与实测功耗绝对值，非仅相对改善百分比。
- 扩展效率说明基线规模与对比规模。
- 容错建议包含故障检测时延、隔离粒度、恢复耗时三项。
- 每条结论附工具名 + 版本 + 命令或脚本 + 日志路径 + 关键数值 + 集群标识。
- 配置变更具备可回滚方案与回滚演练记录。
- 生产环境变更均有人工确认记录。
- 无跨租户数据访问。
- 输出中无 XX、___ 等非标准占位符；未确定项统一标注 [待填写] 或 [待核实]。
- 参考基准：NVIDIA GB200 NVL72 官方数据为 72 Blackwell GPU + 36 Grace CPU、
  13.4 TB HBM3E / 576 TB/s、NVLink 130 TB/s、NVFP4 1,440/720 PFLOPS、
  液冷使成本与能耗降低 25 倍、对比 H100 训练 4 倍与推理 30 倍（级别 A）；
  摩尔线程夸娥万卡集群厂商口径为 Dense MFU 60%、MoE MFU 40%、
  线性扩展效率 95%、ETTR 目标 99%（级别 B，厂商口径）。
  项目内应建立自己的基线与目标，不得直接套用。

## 常见失败与处理
| 失败模式 | 现象 | 处理 |
|---|---|---|
| 指标虚高 | 只报相对改善百分比，不报绝对值与基线 | 强制三指标 + 绝对值 + 基线同时报告 |
| 跨规模误比 | 不同规模集群的吞吐直接比较 | 改用扩展效率与单位成本口径 |
| 静默故障 | 训练收敛异常但无告警 | 引入校验和与周期性自检，升级人工 |
| 慢节点 | 个别节点拖慢整体 | 在线检测并隔离，优先组级隔离而非全局重启 |
| 检查点瓶颈 | 检查点写入耗时超过故障平均间隔 | 调整检查点频率或改用异步检查点 |
| 变更不可回滚 | 配置下发后无法恢复 | 禁止下发，补齐回滚方案与演练记录 |
| 能耗越限 | 实测功耗超出预算 | 触发功率封顶并上报 |
| 跨租户越界 | 读取其他租户作业数据 | 立即中断并上报 |

## 示例
任务：某万卡训练作业有效训练时长占比仅 78%，需提升至 90% 以上。
1. 环境校验：确认加速器型号、互联代际、调度器版本、检查点带宽、多租户隔离。
2. 基线确认：记录基线 MFU 52%、有效训练时长占比 78%、平均故障间隔、检查点耗时。
3. 瓶颈定位：分解为计算、通信、数据加载、调度碎片；发现故障恢复占比最大。
4. 候选生成：三类方案 —— 缩短检查点间隔、引入组级故障隔离、提升慢节点检测频率。
5. 受控验证：在非生产环境分别验证，记录命令、日志路径、集群标识。
6. 差异分析：组级隔离使单次恢复耗时从全局重启改为组内恢复，恢复耗时显著下降。
7. 容错评估：故障检测时延、隔离粒度（数据并行组级）、恢复耗时三项均记录。
8. 能耗评估：PUE 与实测功耗无显著变化。
9. 交付：结论"部分通过，有效训练时长占比提升至 91%，遗留 1 个阻塞项
   （检查点写入带宽需扩容）"；证据表含集群标识、命令、日志路径、数值、与基线差值；
   回滚方案为恢复原检查点频率与全局重启策略。
（本示例的结构参考已公开的万卡集群容错实践，相关厂商数字级别 C，[待核实]。）

## 关联
- 与设计方向的自指飞轮：AlphaChip 设计 TPU、PrefixRL 设计 Hopper、
  OpenAI 用内部 Codex 参与 Jalapeño 设计，构成"AI 设计 AI 芯片"的飞轮。
- 与封装方向的耦合：HBM 与 CoWoS 是 AI 加速器的物理瓶颈，
  GB200 NVL72 的 13.4 TB HBM3E 直接决定可承载的模型规模。
- 与测试方向的耦合：Advantest ACS RTDI 把 NVIDIA AI 推理引擎引入产线，
  是 AI Infra 技术反哺半导体测试的实例。

4.3. Implementation Checklist

5. Summary

The focus of the AI Infra direction within the six-layer AI Harness model is L3 orchestration (training fault tolerance and elastic scheduling) and L5 evaluation (MFU / ETTR / PUE / effective training time ratio).

Three engineering conclusions:

1. Evaluation must land on the three core metrics. MFU, ETTR, and effective training time ratio are the only quantifiable metrics in this direction worth targeting. Any statement such as "performance improved" or "efficiency increased" should be required to be converted to these three metrics, while also giving absolute values and baselines.
2. The core of fault tolerance is to downgrade a failure from a global event to a local event. Moore Threads' "DP group-level fault isolation + zero-interruption standby node integration" demonstrates this idea: only the affected data-parallel group is isolated, while the other groups continue training. This decouples fault-recovery cost from cluster scale and is the key to moving a 10,000-card cluster from "can run" to "can keep running".
3. Energy has become a hard constraint. In NVIDIA's official figures, liquid cooling reduces cost and energy by 25x, showing that the choice of cooling path has become as important as the computing path. The scale of more than 70 MW of total power consumption for a 100,000-card cluster () means energy budgets must be jointly optimized with job scheduling.

The thing this direction most needs to be candid about: no IEEE / ISO / JEDEC standard dedicated to AI compute clusters was found (excluding component-level standards such as HBM and UCIe). The current normative form is de facto standards and vendor specifications — MGX, NVLink / NVLink-C2C, Quantum-X800, Spectrum-X800, BlueField-3, Dynamo, OAM. This means Harness for this direction lacks a standardized data-exchange format and evaluation benchmark, so cross-vendor horizontal comparison is currently not feasible. When writing and implementing, do not fabricate standard numbers.

In addition, much of the widely circulated cluster data in this direction (xAI Colossus scale, Tencent HCC MFU 65%, Baidu Baige 99.5%, Meta failing every 3 hours, etc.) comes from secondary reprints; among them, xAI Colossus has two conflicting figures of 100,000 and 200,000 cards, which are presented in parallel in Section 3.3.3. These data are valuable for understanding engineering difficulty, but must not be used as a basis for decisions; it is recommended to verify each vendor's original official sources.

Information Gap Statement

The following entries have not been confirmed by level-A or level-B sources and are marked [To be verified] in the main text:

1. No IEEE / ISO / JEDEC standard dedicated to this direction was found, with de facto standards and vendor specifications being the norm. This is a genuine information gap and must not be filled with fabricated numbers.
2. xAI Colossus: two conflicting figures of 100,000 and 200,000 H100s; built in 122 days; two conflicting figures of PUE below 1.05 and 1.1 (level C). Presented in parallel in Section 3.3.3.
3. Tencent HCC: MFU 65%, Xingmai Network 2.0 self-developed TiTa protocol at 3.2 TB/s, 100,000-card networking (level C).
4. Alibaba Cloud Lingjun: three-tier Clos topology, network latency reduced to 2 microseconds, compute utilization +20% (level C).
5. Baidu Baige 4.0: effective training time ratio 99.5%, memory optimization raising training efficiency by 30% (level C).
6. Meta Llama 3: a 16,000-card cluster failing every 3 hours (level C; it is recommended to verify the original source in Meta's official blog).
7. OpenAI GPT-4.5: a 100,000-card cluster, compute utilization raised to 60% (level C); GPT-4 trained on 25,000 A100s for about 100 days (level C).
8. Energy: H100 single card 700W, 100,000-card cluster total power consumption exceeding 70 MW, per-rack power density rising from 15 kW to 100 kW (level C).
9. Market and scale: 2025 global training cluster market about $28.7B (+62.3%), China $8.3B, liquid cooling penetration 41%, single-cluster power ceiling 35 MW, single-cluster compute density 22.4 EFLOPS (level C, paid-report summary).
10. Domestic clusters: Huawei Ascend 11,520 cards / 9,000P, Atlas 950 SuperCluster 500,000 cards, Ascend 950 supernode 256 TB memory pool, Shenzhen 10,000-card Ascend 910C, Alibaba T-Head 10,000 cards, Cambricon Siyuan 690 specifications, Sugon 100,000-card supercluster (level C, all [To be verified]).
11. Shipment data (IDC): 2025 China AI accelerator card total shipments about 4 million, domestic vendors 1.65 million, Huawei Ascend 812,000 (level C reprint; it is recommended to check the original IDC report).
12. China's "Training Cluster Comprehensive Performance Evaluation Standard" draft for comments (end of 2025) and mandatory regulations on cluster energy efficiency ratio / carbon footprint / data security across 13 countries and regions (level C).
13. No public reference template of AGENTS.md or SKILL.md targeting the AI Infra direction was found; this document is a proposed draft.

6. References

1. NVIDIA GB200 NVL72 — NVIDIA official website. https://www.nvidia.com/en-us/data-center/gb200-nvl72/
2. NVIDIA GB200 NVL72 Delivers Trillion-Parameter LLM Training and Real-Time Inference — NVIDIA Developer Blog. https://developer.nvidia.com/blog/nvidia-gb200-nvl72-delivers-trillion-parameter-llm-training-and-real-time-inference/
3. Mixture-of-Experts Frontier Models (MoE performance and Dynamo orchestration) — NVIDIA official blog. https://blogs.nvidia.com/blog/mixture-of-experts-frontier-models/
4. Moore Threads Kua'e 10,000-card cluster launch (MDC 2025, 2025-12-20) — Sina Finance (IT Home). https://finance.sina.com.cn/tech/digi/2025-12-20/doc-inhcmnmf6292915.shtml
5. Moore Threads MDC 2025 content summary — Zhiyuan Community. https://hub.baai.ac.cn/view/51375
6. Report on the Ascend 10,000-card cluster in the Guangdong-Hong Kong-Macao Greater Bay Area () — China Economic News Network. https://www.cet.com.cn/itpd/itxw/10454411.shtml
7. Report on Ascend supernodes () — eNet. http://www.enet.com.cn/article/2026/0716/A202607161276307.html
8. 2025 China AI accelerator card shipments (citing IDC) — Sina Finance. https://finance.sina.com.cn/jjxw/2026-09-02/doc-iniqmmyw1388522.shtml
9. Large-model training cluster market and energy efficiency data (paid-report summary) — IIM Info. https://m.iim.net.cn/75/view-234533-1.html
10. Report on OpenAI's in-house chip — TrendForce. https://www.trendforce.com/news/2026/09/09/news-openai-says-it-is-working-with-samsung-on-next-gen-chips-ties-could-expand-beyond-memory/
11. Advantest Pioneers a New Era of AI (AI Infra technology feeding back into chip testing) — Advantest. https://www.advantest.com/en/news/2025/2025100602.html
12. How AlphaChip Transformed Computer Chip Design (the self-referential flywheel) — Google DeepMind. https://deepmind.google/blog/how-alphachip-transformed-computer-chip-design/
13. UCIe and HBM related component-level standards — UCIe Consortium. https://www.uciexpress.org/specifications
14. OCP Accelerator Module (OAM) specification — Open Compute Project. https://www.opencompute.org/