Training Frameworks and Scheduling

1. Introduction

1.1. Background

Training frameworks and scheduling form the software core of AI Infra: hardware sets the ceiling, while frameworks determine how close you can get to it. Two engineering mainlines define this direction:

First, parallel efficiency. The three major schools — Megatron-LM, DeepSpeed, and FSDP — follow the paths of "hard-coded optimization," "low-intrusion Offload," and "PyTorch-native composition" respectively, while ten-thousand-GPU systems (e.g., MegaScale's 12,288 GPUs) prove that framework-level customization can push MFU to 55.2% (Level A, arXiv 2402.15627).

Second, sustained operation. At scales above ten thousand GPUs, failures become the norm: the 54-day snapshot of Llama 3 training saw 419 unexpected interruptions, about one every 3 hours (Level A, official technical report). At this point the framework's core value is no longer peak MFU but effective training time — the ability to orchestrate checkpoints, fault detection, isolation, and recovery.

1.2. Definition and Scope

The training frameworks and scheduling direction covers, on the runtime side:

Boundary notes: for inference-side serving frameworks, see 05-inference-framework.md; for the complete SOP and monitoring system of fault-tolerant operations, see 06-ops.md (this document is written from the training perspective, the ops document from the operations perspective — the material overlaps but the perspectives differ).

1.3. Position in the AI Harness System

Training frameworks are the scheduling hub of the L3 orchestration-and-control layer in the Harness six-layer model, while also deeply covering L4 and L5:

Bottleneck-layer judgment: this direction's focus is on L3 orchestration (parallel strategies and fault-tolerant recovery) and L5 evaluation (effective training time). This is consistent with and continues the conclusions of the design-side documents (see 02-行业赋能/04-硬件研发组/05-ai-infra.md): MFU, ETTR, and effective training time ratio are the only quantifiable metrics in this direction worth targeting.

1.4. Current State of Development

1.4.1. The Three Major Schools and Selection Experience (Empirical Range, Not an Official Benchmark)

The table above is the empirical range compiled by the technical community (Level C, not an official benchmark; cite it when referencing). For MoE models, each school drops roughly another 10 percentage points. Selection rule of thumb (same-source Level C): TP must not span NVLink domains (when crossing machines over IB, TP all-reduce slows down 20-40%); PP is only worthwhile at 128+ GPUs; below 1B use DDP/FSDP2.

Trends: after Megatron-Core built in a distributed optimizer, the "Megatron-DeepSpeed combination" was replaced; FSDP2/PyTorch TP/PP converge uniformly based on DTensor; torchtitan became the official reference implementation. Domestic in-house stacks: DeepSeek HAI-LLM (DualPipe open-sourced), Shanghai AI Lab InternEvo, Baidu PaddleNLP/KunlunMirror (Level C compiled).

1.4.2. Ablation and Optimization Methods (MegaScale Breakdown, Level A)

MegaScale's 55.2% vs. Megatron-LM's 41.2% (1.34x) comes from a reusable engineering breakdown: parallel Transformer blocks + sliding-window attention +5.6%, 3D parallel communication overlap +6.2%, efficient operators +1.7%, data pipeline +1.1%, LAMB large batch +3.0%. Among these, communication overlap (Megatron's --tp-comm-overlap splits All-Reduce into Reduce-Scatter + All-Gather and overlaps them with GEMM) is the largest single item common to all schools.

Figure 4-1｜Llama 3's 54-Day Training Failure Profile and Effective Training Time

Data source: Meta's "The Llama 3 Herd of Models" official technical report (Level A); Epoch AI's checkpoint overhead estimate is a Level C model estimate. Bar widths are drawn proportionally for illustration (the second bar uses the full 466 interruptions as its base; bars 3 and 4 are sub-shares relative to the unexpected-interruption bar).

2. Glossary

3. Case Studies

3.1. ByteDance MegaScale: Efficiency and Fault Tolerance in Ten-Thousand-GPU Training

Evidence level: A (official paper, arXiv 2402.15627)

3.1.1. Background

When training a 175B model at 12,288-GPU scale, the default configuration of open-source frameworks cannot simultaneously meet MFU and stability: communication overhead, data-pipeline bottlenecks, and failure recovery all become major losses once scale is amplified.

3.1.2. Approach

• Efficiency: parallel Transformer blocks + sliding-window attention (+5.6% MFU), 3D parallel communication overlap (+6.2%), efficient operators (+1.7%), data-pipeline optimization (+1.1%), LAMB large batch (+3.0%), together raising MFU from Megatron-LM's baseline 41.2% to 55.2%.
• Stability engineering: Driver-Executor architecture + heartbeat monitoring (including training logs, RDMA traffic metrics); on anomalies, pause the whole cluster → lightweight self-check (loopback bandwidth, RNIC connectivity, NCCL communication tests) → isolate the faulty node → K8s provisions a healthy node → recover from the most recent checkpoint.
• Two-phase asynchronous checkpoint: Phase one writes GPU state to host memory (PCIe, a few seconds) and resumes training; phase two writes to HDFS asynchronously in the background; on recovery, a single worker in the same DP group reads and broadcasts, linearly reducing storage-bandwidth pressure.
• Observability: CUDA Event low-overhead profiling tools, with heatmaps locating compute stragglers (roughly 0.5% of the cluster); 3D parallel visualization for locating communication bottlenecks.

3.1.3. Results

• MFU 55.2% (12,288 GPUs, 175B), vs. Megatron-LM 41.2%, a 1.34x improvement.
• Weak scaling: 2,240 → 11,200 GPUs stays near-linear, MFU stable at about 54%.
• Harness reading: MegaScale's "detect-diagnose-isolate-replace-recover" pipeline is a textbook implementation of L3 fault-tolerant orchestration; its two-phase asynchronous checkpoint is an exemplary L4 "artifact-and-checkpoint" — it turns checkpoints from "the cost of pausing training" into "a background task running in parallel with training," directly lifting effective training time.

3.2. Meta Llama 3 405B: The 54-Day Failure Profile and Effective Training Time

Evidence level: A (official technical report "The Llama 3 Herd of Models"); some restatements are Level B

3.2.1. Background

Training the 405B model on 16,384 H100s (TP=8, PP=16, DP=8~128) used about 39.3M H100 GPU-hours in total. At this scale, the failure data was made public to an unprecedented degree, making it the de facto reference for the industry's failure profile.

3.2.2. Approach

• Fault-tolerance combination: heavily asynchronous checkpoints + hot-standby nodes + startup health checks.
• Diagnostic toolchain: PyTorch NCCL flight recorder (records the last N collective-communication metadata entries and stack backtraces) to speed up root-cause analysis; deep NCCLX integration for NVLink/RoCE fault detection and automatic timeout of stalled communication; straggler detection tools that prioritize troubleshooting problem traffic.
• SDC governance triad: FleetScanner (periodic fleet-wide testing), Ripple (co-located testing), Hardware Sentinel (telemetry + analysis to identify faults).
• Checkpoint trade-off: a three-way trade-off between frequency vs. write bandwidth vs. recovery loss (Epoch AI estimates the optimal interval at about 4 minutes; a Level C model estimate).

3.2.3. Results

• Failure profile (54-day snapshot): 466 interruptions = 47 planned + 419 unexpected (about one every 3 hours); among unexpected interruptions, GPU-related 58.7% (GPU faults 148 cases/30.1%, including NVLink; HBM3 memory 72 cases/17.2%); network switch/cable 35 cases (8.4%); only 2 CPU faults in 54 days; 6 cases attributed to GPU SDC.
• Effectiveness: effective training time ratio above 90%; only 3 cases needed major manual intervention, the rest were handled automatically.
• Harness reading: this profile ranks L5 observation priorities toward the hardware domain — GPU-related faults account for nearly 60%, meaning L5 hardware telemetry (ECC, NVLink, HBM) is more worth investing in than application-layer logs; "only 3 manual interventions" is the quantified benchmark for "automated escalation" at the L6 governance layer.

3.3. MLPerf Training v5.0: The Industry-Unified Benchmark Metric

Evidence level: A (MLCommons figures; media restatements are Level B)

3.3.1. Background

Cross-system comparison of training frameworks needs a unified benchmark. MLPerf Training v5.0 (released 2025-06-04) is the latest confirmed training-side figure as of the information cutoff date.

3.3.2. Approach

• 20 submitting organizations, 201 results — an all-time record.
• First-time introduction of a Llama 3.1 405B pretraining benchmark (replacing the GPT-3 benchmark), with submission volume exceeding that of every prior GPT-3 round.
• Lenovo submitted the first power benchmark set, bringing energy consumption into the performance comparison dimension.
• Flagship submission: CoreWeave + NVIDIA + IBM used 2,496 GB200 superchips (the largest GB200 NVL72 cluster in MLPerf history, 34x larger than previous cloud-vendor submissions), completing the 405B training task in 27.3 minutes — over 2x faster than comparable-scale clusters.

3.3.3. Results

• GB200 per-GPU training throughput reaches up to 2.6x equivalent Hopper (cross-cited via O'Reilly books, Level B/C).
• MLPerf Inference v5.0 per-GPU inference throughput is about 3.4x (same-source restatement).
• Harness reading: MLPerf is the industry-standard L5 regression set — when this group's AGENTS.md requires performance figures to state their source, "MLPerf" is a first-class source alongside "official releases/third-party measurements/vendor claims"; when selecting frameworks, MLPerf results serve as the upper-bound reference for same-generation hardware, but real projects must still build their own regression job sets.
• Metric boundary: as of 2026-09-12, no confirmation of new training-side results after MLPerf v5.1 was found; this is flagged as a gap.

4. Practical Standards

Nature statement: the following is a proposed draft distilled from public industry practices (MegaScale, Llama 3, MLPerf, etc.), not the text of an official industry standard. The group-level higher-level norms are in AGENTS.md in the same directory; below is a trimmed version for the training frameworks and scheduling direction.

4.1. AGENTS.md Specification

4.1.1. AGENTS.md (AI Infra Group · Training Frameworks and Scheduling Direction)

# AGENTS.md —— AI Infra 组 · 训练框架与调度方向

> 本文为基于行业实践提炼的建议稿，非官方行业标准原文。
> 上位规范：01-AI-Infra组/AGENTS.md（不可逆动作禁止清单、变更窗口、可观测要求）。

## 角色与边界
- 角色：训练框架与调度智能体，负责并行策略设计、框架选型与版本管理、
  checkpoint 策略、故障检测与恢复编排、MFU/ETTR 诊断。
- 不负责：调度器全局策略修改（属禁止清单）、配额最终分配、
  硬件采购与机房改造。
- 判定器为监控与基准：所有 MFU/ETTR/有效训练时长占比结论必须来自
  真实监控数据或固定基准作业集回归。

## 环境假设
执行前必须显式声明：
- 加速器型号/数量/代际、Scale-up 域边界、Scale-out 网络与跨机带宽。
- 框架与版本（Megatron-LM / DeepSpeed / FSDP2 / 自研栈）、
  调度器（Slurm/K8s）版本与分区拓扑。
- 模型结构（稠密/MoE、专家数）、目标并行策略、批量与序列长度。
- checkpoint 存储与写入带宽实测值、热备池容量。
- 未声明框架版本与并行策略时，禁止产出 MFU/扩展性结论。

## 上下文加载顺序（Context Budget）
1. 作业配置：并行策略、批量、checkpoint 频率、恢复策略。
2. 集群与拓扑：域边界、跨机带宽、分区。
3. 监控摘要：MFU、吞吐、通信占比、straggler、中断记录。
4. 框架变更记录与已知问题清单。

## 工具契约
- 框架与调度器通过封装接口访问；返回值含退出码、日志路径、关键数值、
  作业标识、版本。
- 消融实验必须单变量：每次只改一个并行/优化开关，记录前后 MFU。
- 调度策略变更必须可回滚并先在影子队列验证。

## 任务执行流程（SOP）
- S1 需求解析：目标模型 → 并行约束（TP 不跨域、EP 带宽需求）。
- S2 基线建立：默认配置 MFU、中断频率、恢复耗时。
- S3 消融设计：参照 MegaScale 拆解（通信重叠、算子、数据管线、批量），
  按预期收益排序。
- S4 受控验证：小规模复现消融，再灰度到目标规模。
- S5 容错验证：注入式故障演练（杀节点、拔线模拟），测检测时延/
  隔离粒度/恢复耗时。
- S6 证据打包：与基线差值 + 中断原因分布 + 待人工确认项。

## 验证与证据要求
- MFU 结论注明模型、硬件代际、并行配置、精度格式；经验区间须标
  "经验区间，非官方基准"。
- 容错结论三项齐备：故障检测时延、隔离粒度、恢复耗时。
- 消融各项收益与 MegaScale 类公开拆解对比时注明来源与规模差异。
- 跨规模比较使用弱扩展效率而非绝对吞吐。

## 失败与升级策略
- 有效训练时长占比连续下降 → 升级人工，附中断原因四分类
  （硬件/网络/软件/数据）分布。
- 消融在小规模有效但规模放大后失效 → 停止灰度，回滚并记录。
- 疑似 SDC → 触发周期性全舰队测试与同置测试，升级硬件排查。
- checkpoint 写入耗时逼近故障平均间隔 → 调整为两阶段异步方案。

## 安全与合规红线
- 多租户配额与优先级变更走人工审批。
- 框架版本升级遵循兼容矩阵，禁止跨版本强升（属禁止清单）。
- 训练数据与 checkpoint 访问按租户隔离。
- 禁止残留非标准占位符。

## 输出格式
- 首行：可判定结论 + 阻塞项。
- 消融表：每项开关、MFU 变化、规模条件、来源。
- 容错报告：检测时延、隔离粒度、恢复耗时 + 中断原因分布。
- 调度变更：影响面、影子队列验证结果、回滚命令。
- 数值规范：参数带单位；范围用 ~ 连接；百分比数值与 % 之间不留空格。

## 评估与自检
- MFU 是否注明模型/代际/并行配置？
- 经验区间是否标注非官方基准？
- 容错三项是否齐备？
- 消融是否单变量？
- 是否残留非标准占位符？

4.2. SKILL.md Specification

4.2.1. SKILL.md (AI Infra Group · Training Efficiency Ablation and Fault Orchestration)

---
name: training-efficiency-ablation-and-fault-sop
description: AI Infra 组训练框架方向标准技能——MFU 消融诊断、并行策略设计、
  checkpoint 与故障恢复编排。当需要提升 MFU、设计 TP/PP/DP/EP 组合、
  优化 checkpoint、验证容错流水线、对标 MLPerf 口径时使用。
  触发词：MFU、并行策略、Megatron、DeepSpeed、FSDP、checkpoint、ETTR、MLPerf。
version: 1.0
created: 2026-09-12
---

# AI Infra 组 · 训练效率消融与容错编排

> 本文为基于行业实践提炼的建议稿，非官方行业标准原文。
> 上位规范：01-AI-Infra组/AGENTS.md。

## 适用场景
- MFU 低于经验区间，需定位计算/通信/数据加载/调度碎片瓶颈。
- 并行策略（TP/PP/DP/EP/SP/CP）组合设计与跨域约束校验。
- checkpoint 策略设计与两阶段异步改造。
- 容错流水线（检测-诊断-隔离-替换-恢复）建设与演练。
- 框架选型（Megatron-LM / DeepSpeed / FSDP2 / 自研栈）与版本升级评估。
- MLPerf 口径对标与固定基准作业集建设。

## 前置条件
- 已声明框架版本、并行策略、模型结构、checkpoint 带宽、热备池。
- 存在基线 MFU 与中断记录；固定基准作业集可用。
- 影子队列或非生产分区可用于消融与演练。

## 输入
| 输入项 | 说明 | 必需 |
|---|---|---|
| 作业配置 | 并行策略、批量、序列长度、checkpoint 频率 | 是 |
| 集群拓扑 | Scale-up 域边界、跨机带宽、分区 | 是 |
| 基线指标 | MFU、吞吐、中断频率、恢复耗时 | 是 |
| 监控数据 | 通信占比、straggler、心跳/日志路径 | 是 |
| 框架变更记录 | 版本升级、开关变更历史 | 否 |

## 输出
| 输出项 | 说明 | 必需 |
|---|---|---|
| 可判定结论 | 通过 / 不通过 / 部分通过 + 阻塞项 | 是 |
| 消融报告 | 每项开关的 MFU 变化、规模条件、来源对照 | 是 |
| 并行策略方案 | TP/PP/DP/EP 映射与跨域校验结论 | 是 |
| 容错报告 | 检测时延、隔离粒度、恢复耗时 + 演练记录 | 是 |
| checkpoint 方案 | 频率/异步化设计 + 写带宽占用评估 | 条件必需 |
| 证据表 | 工具、版本、命令、日志路径、数值、时间窗口 | 是 |

## 执行步骤
1. 环境校验：框架版本、拓扑、checkpoint 带宽、影子队列可用性。
2. 基线确认：默认配置 MFU、通信占比、中断频率、恢复耗时。
3. 瓶颈分解：计算/通信/数据加载/调度碎片四类（CUDA Event 类低开销
   剖析 + 3D 并行可视化）。
4. 消融设计：按预期收益排序（通信重叠 → 算子 → 数据管线 → 批量），
   单变量执行，小规模复现后灰度放大。
5. 并行校验：TP 不跨 Scale-up 域、EP 带宽需求测算、PP 规模适用性。
6. 容错演练：注入故障（杀节点、断链模拟），测量检测时延、隔离粒度、
   恢复耗时；验证两阶段异步 checkpoint。
7. 对标：与公开拆解（MegaScale 各项收益）与 MLPerf 同代结果对照，
   注明规模与代际差异。
8. 交付：消融表 + 容错报告 + 回滚方案 + 待人工确认项。

## 质量标准（DoD）
- MFU 结论注明模型/硬件代际/并行配置/精度格式。
- 消融单变量；经验区间标注"非官方基准"。
- 容错三项齐备且有演练记录支撑。
- checkpoint 方案给出写带宽占用与恢复损失测算。
- 参照口径（不得直接套用）：MegaScale 12,288 GPU MFU 55.2% vs
  Megatron-LM 41.2%（A级）；Llama 3 有效训练时间占比 >90%、419 次
  意外中断、GPU 相关 58.7%（A级）；MLPerf Training v5.0 = 20 机构
  201 项、首设 Llama 3.1 405B 基准（A/B 级）。

## 常见失败与处理
| 失败模式 | 现象 | 处理 |
|---|---|---|
| TP 跨域 | 跨机 TP 使 all-reduce 掉速 20-40% | 重排并行映射，跨机改 PP/DP/EP |
| 消融多变量 | 同时改多个开关无法归因 | 强制单变量，逐项回退 |
| checkpoint 拖慢 | 写入耗时逼近故障平均间隔 | 两阶段异步化或调频率 |
| 规模放大失效 | 小规模收益在万卡消失 | 停止灰度，检查通信拓扑假设 |
| 疑似 SDC | 收敛异常无告警 | 全舰队测试 + 同置测试，升级人工 |
| 恢复慢 | 恢复耗时占比高 | 单 worker 读取再广播；检查热备池 |

## 示例
任务：某 8,000 卡作业 MFU 42%，目标提升至 50% 以上。
1. 环境校验：Megatron-Core 版本、TP/PP/DP 配置、checkpoint 带宽。
2. 基线：MFU 42%，通信占比 26%，数据管线等待 6%。
3. 瓶颈分解：通信重叠缺失为主因。
4. 消融：tp-comm-overlap 单项在 512 卡复现 +5.8%，灰度至 8,000 卡
   +6.1%；Zero-Bubble 流水线 +2.2%；数据管线预取 +0.9%。
5. 并行校验：TP=8 限制在 Scale-up 域内，PP=16 确认规模适用。
6. 容错演练：杀节点场景检测时延 40 秒内、DP 组级隔离、恢复 8 分钟；
   异步 checkpoint 写带宽占用达标。
7. 对标：拆解结构与 MegaScale 公开数据一致（规模不同，标注差异）。
8. 交付：通过——MFU 提升至 51.2%；消融表、容错报告、回滚方案齐备。
（示例数值为演示用假设值，非实测数据；方法论参照 MegaScale 与 Llama 3
公开实践。）

4.3. Implementation Checklist

5. Summary

In the Harness six-layer model, this direction's focus is L3 orchestration (parallel strategies and fault-tolerant recovery) and L5 evaluation (effective training time). Three conclusions:

1. Improving MFU is a decomposable engineering problem. Every gain that took MegaScale from 41.2% to 55.2% is publicly reusable, with communication overlap the largest single item; framework selection should shift from "which framework is better" to "which switches work at my scale and topology," verified through single-variable ablations.
2. Beyond ten thousand GPUs, fault-tolerant orchestration matters more than peak MFU. Llama 3's 419 unexpected interruptions and "only 3 manual interventions" together show that hitting an effective training time above 90% depends on a systematic combination of asynchronous checkpoints, hot-standby nodes, the flight recorder, and the SDC-governance triad — not on any single-point optimization.
3. A unified benchmark is the prerequisite for cross-system dialogue. MLPerf Training v5.0 (which first introduced a Llama 3.1 405B benchmark) together with MFU/ETTR/effective-training-time-ratio forms a two-tier evaluation system of "industry metric + project metric"; after any framework or Harness change, regression must be run on a fixed regression job set.

Information-Gap Statement

The following items have not been confirmed by Level A/B sources and are marked [To be verified] in the main text:

1. The MFU empirical ranges of the three major schools (Megatron 45-55%+ / FSDP2 35-45% / DeepSpeed 30-40%) and the selection rules of thumb (TP cross-domain slowdown 20-40%, PP 128-GPU threshold) are technical-community compilations, Level C, empirical ranges, not official benchmarks.
2. The DeepSeek-V3 engineering optimizations (256 experts split into 16 communication groups, bidirectional pipeline overlap rate 98%, adaptive gradient compression saving 40% bandwidth) are restated figures, Level C; citing the DeepSeek-V3 technical report directly is recommended.
3. Epoch AI's checkpoint overhead of 2.1% and optimal interval of 4 minutes are model estimates, Level C.
4. The inventory of domestic in-house stacks (HAI-LLM, InternEvo, KunlunMirror) is a community-compiled figure, Level C.
5. No new training-side results after MLPerf v5.1 have been confirmed as of 2026-09-12; the GB200-to-Hopper 2.6x/3.4x cross-cited figures are Level B/C restatements.
6. No public AGENTS.md/SKILL.md standard template in this direction was found; this document is a proposed draft.

6. References

1. MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs — ByteDance, arXiv 2402.15627, 2024. https://arxiv.org/abs/2402.15627
2. The Llama 3 Herd of Models — Meta, 2024. https://arxiv.org/abs/2407.21783
3. MLPerf Training v5.0 results release — MLCommons/PR Newswire, 2025-06-04. https://prn.to/3Zlln3j
4. Megatron-LM — NVIDIA GitHub. https://github.com/NVIDIA/Megatron-LM
5. DeepSpeed — Microsoft GitHub. https://github.com/microsoft/DeepSpeed
6. torchtitan / FSDP reference implementation — PyTorch official. https://github.com/pytorch/torchtitan
7. NCCL (including the flight recorder and 2.27 features) — NVIDIA GitHub. https://github.com/NVIDIA/nccl
8. AI Infra (design side, cross-reference for the MFU/ETTR metric system) — 02-行业赋能/04-硬件研发组/05-ai-infra.md, 2026-09-12.
9. MLCommons official site (benchmark metric) — MLCommons. https://mlcommons.org/
10. DualPipe / HAI-LLM related open source — DeepSeek GitHub. https://github.com/deepseek-ai