Training Framework Ecosystem: Megatron-LM / DeepSpeed / FSDP / Colossal-AI

1. Introduction

1.1. Positioning of This Integrated Chapter

This chapter consolidates the open-source training framework ecosystem into a single study, covering four representative frameworks: Megatron-LM (NVIDIA), DeepSpeed (Microsoft), FSDP/torchtitan (PyTorch native/Meta), and Colossal-AI (HPC-AI Tech). They are the "parallel-orchestration layer" of large-model pretraining and post-training — determining the actual compute utilization of ten-thousand-GPU clusters.

Together with chapter 09 (inference serving), this chapter forms the two wings of the open-source AI Infra ecosystem: the training side determines model production cost, and the inference side determines model serving cost.

1.2. Basic Information Cards of the Four Frameworks

1.3. Division of Labor with Chapter 03

This chapter overlaps with 03-huawei-ascend.md on MindSpeed (Ascend training acceleration library, exposing Megatron-style APIs) and MindSpore. Division of labor: the Ascend-side background of MindSpore/MindSpeed belongs to chapter 03; this chapter provides the complete description of the open-source training framework landscape, with cross-references in the form "see chapter 03 / 10". The MFU indicator system in section 4.2 of this chapter is aligned with chapter 03 (DeepSeek-V4-Pro post-training MFU above 30%) and chapter 05 (Kuae Dense MFU 60%, company-reported), forming a unified group-wide reference for training-efficiency metrics.

1.4. Position in the AI Harness System

Training frameworks are the "Harness of the training domain": parallelism strategy selection, pipeline scheduling, resumable training, and precision governance correspond respectively to L3 (orchestration), L4 (state), L5 (observability), and L6 (governance) of the six-layer Harness. The difference from the agent Harness is that the "task" of the training Harness is fixed (forward–backward–optimizer loop); the variability lies in how the model and data are sharded across thousands of GPUs — hence its L3 is a mathematical problem (parallel topology) rather than a planning problem. This is an analogy mapping; see the declaration in section 5.6.

---

2. Glossary

---

3. Feature Description and Product Lines

3.1. Megatron-LM / Megatron-Core

1. Origin and evolution: originated from the 2019 paper (complete industrial TP implementation); 2021 interleaved 1F1B → 2022 Sequence Parallelism → 2023 Megatron-Core became an independent embeddable library → 2024–2025 Zero-Bubble Pipeline, Context Parallelism, MoE, FP8, Mamba/SSM merged (A-grade paper series);
2. Standing: the benchmark of the "model sharding" school among the "three parallelism philosophies"; industrial default for pretraining at first-tier companies (continuing the GPT-3/OPT/LLaMA era);
3. Efficiency experience: empirical MFU of 45–55%+ for 70B-class Dense models (technical-community experiential range, not a benchmark conclusion, B-grade);
4. Synergy: works with Transformer Engine and FlashAttention; NeMo is its productized packaging.

3.2. DeepSpeed and the ZeRO Series

1. ZeRO 1/2/3: progressive sharding of optimizer state/gradients/parameters, cutting memory by 4/8× respectively with linear scaling;
2. ZeRO-Offload / NVMe: trains 13B+ models on a single GPU — the accessible starting point for small and mid-sized teams training large models;
3. Other capabilities: MoE, Ulysses long context, 1-bit Adam (5× less communication);
4. Governance affiliation: joined the PyTorch Foundation alongside vLLM in 2025-05 (first cohort of the umbrella foundation);
5. Momentum: some of its momentum is being caught up by FSDP2 (community-reported, B/C grade).

3.3. FSDP / FSDP2 / torchtitan

1. FSDP2: rewritten on top of DTensor — deterministic memory management, communication-free sharded checkpoint; throughput about +1.5% vs. FSDP1; Llama2-7B on 128×A100 reaches 3700 tokens/s/GPU (official figures);
2. torchtitan (accepted at ICLR 2025): production reference implementation of 4D parallelism (FSDP2 + TP + PP + CP); Llama 3.1 8B on 128 GPUs sped up 65%, 70B on 256 GPUs 12.6%, 405B on 512 GPUs 30% (official figures);
3. Positioning: the PyTorch-native route, default for the research community and mid-scale training; Meta's in-house route.

3.4. Colossal-AI and the Hybrid Schools

1. Colossal-AI: unified parallelism interface (1D/2D/2.5D/3D tensor parallelism) + Gemini heterogeneous memory management (trains 13B on a single consumer GPU) + Booster auto-parallelism injection; up to 2.76× training speedup (paper-reported); representative of the domestic easy-to-use school (HPC-AI Tech);
2. Hybrid schools: the Megatron-DeepSpeed hybrid stack (mainstream in the BLOOM and GPT-NeoX era: Megatron for TP/PP + DeepSpeed ZeRO-1);
3. Domestic in-house development: DeepSeek HAI-LLM (DualPipe already open-sourced), Shanghai AI Lab InternEvo, Baidu Kunlunjing/PaddleNLP (limited public details; see the information-gap declaration).

---

4. Platform Architecture

4.1. Parallelism Strategy Spectrum

Figure 10-1 | Parallelism strategy spectrum: DP / TP / PP / EP / SP (CP) and framework mapping

Data sources: NVIDIA Megatron paper series (2019–2024), arXiv survey (2025), Juejin technical column review (2026-04); the diagram is drawn based on this chapter's analysis, and all MFU values are experiential/company-reported.

4.2. MFU Experience Ranges and Indicator System

Empirical MFU ranges for 70B Dense models (technical-community experiential values; must be labeled as non-benchmark conclusions, B/C grade):

Supporting indicators: effective training time ratio (chapter 05, Kuae basis, 90%+), linear scaling efficiency (Kuae basis, 95%), resumable-training time — these four form a unified indicator system for training-cluster efficiency (aligned with the bases of chapters 03/05).

---

5. Harness Design

5.1. Overview of the Six-Layer Capabilities

The six-layer mapping between the training domain and the agent domain is an analogy mapping (see the declaration in section 5.6):

5.2. L3 Training Orchestration: Parallelism Strategy as Orchestration

The "Harness" on the training side is parallelism strategy orchestration: choosing the combination of TP/SP/PP/CP/DP, Zero-Bubble pipeline scheduling, and 1F1B/interleaved pipeline selection — essentially the training-domain counterpart of the "plan–execute–feedback" loop: plan (parallelism configuration) → execute (training steps across thousands of GPUs) → feedback (MFU/throughput observability) → re-plan (configuration tuning). Megatron-Core is currently the de facto standard for this layer.

5.3. L4 State Management: Checkpoint Evolution

1. FSDP2 communication-free sharded checkpoint: saving requires no extra communication to re-shard, reducing checkpoint overhead on large clusters;
2. Megatron distributed checkpoint conversion tool: migrates checkpoints across parallelism configurations;
3. Resumable-training capability aligns with chapter 03 (ten-second-scale checkpoint recovery) and chapter 05 (effective training time 90%+) — state continuity is the decisive variable in the availability of ten-thousand-GPU training.

5.4. L5 Evaluation and Observability Layer

MFU, effective training time ratio, linear scaling efficiency, and resumable-training time form the standard indicator system; it must be emphasized that each framework's MFU is an experiential range rather than a unified benchmark — cross-framework/cross-hardware comparisons must verify model scale, precision (FP8/BF16), and cluster configuration.

5.5. L6 Governance and Security Layer

Precision governance is the core of L6 in the training domain: the quality thresholds for FP8/quantization precision (MLPerf training tasks must satisfy accuracy goals) define the compliance boundary of "low-precision training" — the tradeoff between compute efficiency and model quality is institutionalized.

5.6. Analogy Declaration

This section maps the mechanisms of training frameworks onto the six Harness layers; this is a structural analogy rather than a direct correspondence: the "task" in the training domain is a deterministic training loop, and "orchestration" is a mathematized parallel-topology problem, which differs in nature from open-ended task orchestration in the agent domain (planning, tool invocation, human interruption). When citing this section's conclusions, preserve the analogy qualification.

---

6. Practical Cases

6.1. Framework Adoption Landscape

6.2. Hybrid Schools and Domestic In-House Development

1. Megatron-DeepSpeed hybrid stack: mainstream in the BLOOM and GPT-NeoX era — Megatron for TP/PP, DeepSpeed ZeRO-1;
2. Domestic in-house development: DeepSeek HAI-LLM (DualPipe already open-sourced), Shanghai AI Lab InternEvo, Baidu Kunlunjing/PaddleNLP (closed-source/limited public info, summarized only);
3. Domestic-hardware integration: Ascend MindSpeed exposes Megatron-style APIs (see chapter 03); Huawei CloudMatrix 384 trained stably for 40 days on the MindSpore/MindSpeed stack.

6.3. Selection Experience

Community selection ladder (B-grade review):

---

7. Summary

Strengths (ecosystem consolidation):

1. Maturity of evolution: from the 2019 Megatron paper to 2025 torchtitan 4D parallelism, the industrial practice of the parallel-orchestration layer has converged substantially;
2. Governance neutralization: DeepSpeed joined the PyTorch Foundation in the same cohort as vLLM, forming neutral governance for open-source training infrastructure;
3. Open indicator system: MFU/scaling efficiency/effective training time form a communicable efficiency language; domestic vendors (Kuae, Hetao) have adopted the same discourse.

Weaknesses:

1. No unified benchmark: all MFU figures are experiential ranges or company-reported; cross-framework comparison relies on self-investigation and self-testing;
2. Blackwell-generation data gap: official FP8/FP4 training MFU figures for each framework on GB200 are missing;
3. Low public openness of domestic stacks: HAI-LLM/InternEvo details are limited, so ecosystem portability cannot be assessed.

Applicability boundary: ten-thousand-GPU pretraining → Megatron-Core; small/mid-sized teams and single-machine large models → DeepSpeed/Colossal-AI; research and mid-scale → FSDP2/torchtitan; Ascend clusters → MindSpeed (chapter 03).

Selection suggestion: follow the selection ladder by model scale, avoiding the obsession with "one framework for everything"; when deploying at ten-thousand-GPU scale, prioritize "effective training time ratio" and "resumable-training time" above peak MFU (stability loss is the number-one cost of ten-thousand-GPU training); FP8 training must establish a precision regression set (L6 precision governance) before scaling up.

信息缺口声明

1. Official FP8/FP4 training MFU figures for each framework on Blackwell/GB200 are missing (NVIDIA has not separately published a Megatron-B200 MFU whitepaper);
2. Customer and revenue data for Colossal-AI's commercialization (HPC-AI Tech) are missing;
3. HAI-LLM/InternEvo details are closed-source/limited public, allowing only a summary;
4. All MFU data in this chapter are experiential ranges or company-reported, not unified-benchmark measurements; citations must carry the basis qualification.

---

8. References

1. [Large Model Infrastructure Engineering] 07: Megatron-LM and DeepSpeed — Juejin technical column, 2026-04. https://juejin.cn/post/7633318137166823487
2. Model Parallelism — AI Wiki, 2025. https://aiwiki.ai/wiki/model_parallelism
3. From Code Foundation Models to Agents and Applications (academic survey chapter on DeepSpeed/FSDP/TorchTitan/Colossal-AI) — arXiv, 2025. https://arxiv.org/pdf/2511.18538v1.pdf
4. Panorama of Merging the Three Parallelism Approaches: A Practical Guide to Megatron × DeepSpeed × Colossal-AI Configuration Comparison — CSDN, 2025. https://blog.csdn.net/sinat_28461591/article/details/147168750
5. Deep Learning Frameworks, Training Frameworks, and Inference Frameworks as of 2025 — DevPress/CSDN, 2025. https://devpress.csdn.net/awstech/6a9274d23bda720d4b371d5a.html
6. Megatron-LM open-source repository — NVIDIA GitHub, 2025. https://github.com/NVIDIA/Megatron-LM
7. DeepSpeed open-source repository — Microsoft GitHub, 2025. https://github.com/microsoft/DeepSpeed
8. PyTorch FSDP official tutorial — PyTorch, 2025. https://pytorch.org/tutorials/intermediate/FSDP_tutorial.html
9. PyTorch Foundation Welcomes vLLM (first cohort of the umbrella foundation includes DeepSpeed as evidence) — PyTorch, 2025. https://pytorch.org/blog/pytorch-foundation-welcomes-vllm/