Servers and SuperPods

1. Introduction

1.1. Background

SuperPod is the dominant product form of today's AI infrastructure. The authoritative formulation given by Huawei Connect 2025 is: "SuperPods have become the dominant product form and are becoming the new normal of AI infrastructure construction" and "a SuperPod is in fact a computer that can learn, think, and reason — physically composed of multiple machines, but logically a single machine" (Grade A). In terms of industry evolution, the SuperPod concept was proposed by NVIDIA; the GB200 NVL72 released in March 2024 is widely regarded as the defining SuperPod server product (Grade B, academic-review framing).

The direct driver behind the emergence of SuperPods is the bandwidth bottleneck of the traditional "one server, eight GPUs" PCIe topology: peer-to-peer interconnect bandwidth is below 64 GB/s, which cannot meet the high-bandwidth, low-latency needs of MoE large models. Aggregating hundreds or thousands of accelerators into a single logical machine via a memory-semantic network is the common choice of both current technology roadmaps.

1.2. Definition and Scope

The Server & SuperPod direction covers, on the operations side:

Scope note: The interconnect-protocol standard race (NVLink vs UALink vs ESUN/SUE and domestic protocols) is the territory of 03-cluster.md; this document covers only in-domain SuperPod implementations. Single-chip parameters are in 01-chip.md. The full hardware specification table of the GB200 NVL72 has already been collected on the design side (see 02-行业赋能/04-硬件研发组/05-ai-infra.md), and is not repeated here.

1.3. Positioning in the AI Harness System

Within the six-layer Harness model, the SuperPod is the "single-machine-ization of the L2 execution layer": it aggregates hundreds of accelerators into one logical execution unit, producing three layers of transmission:

Bottleneck-layer assessment: The center of gravity of this direction lies in L2 (the scale and efficiency of the single-machine execution unit) and the physical precondition of L1 (memory-semantic interconnect). When SuperPod in-domain interconnect bandwidth is insufficient, the first thing to degrade is not L3 scheduling but L1's KV sharing capability — hence "whether memory-semantic interconnect is in place" is the first criterion in SuperPod selection.

1.4. Current Status

Three generational lines are advancing in parallel (as of 2026-09-12):

• NVIDIA line: GB200 NVL72 in service (72 Blackwell GPUs + 36 Grace CPUs, specs in the design-side document) → Vera Rubin NVL72 (2026 H2 delivery) → Rubin Ultra NVL576 (2027 H2 announced, 576 GPUs in a single NVLink domain).
• Huawei line: Atlas 900 SuperPod (CM384 hardware foundation, official figure of over 300 deployed serving 20+ customers) → Atlas 950 SuperPoD (8192 GPUs, 2026 Q4) → Atlas 960 SuperPoD (15488 GPUs, 2027 Q4).
• Conflicting figures listed in parallel: there is also a vendor-disseminated figure of "over 750 Ascend 384 SuperPods commercially deployed" (Grade C). This document adopts the official HC2025 figure of 300+ (Grade A); the 750 figure is marked [To be verified].
• Extreme deployment case: xAI Colossus 1 validated the engineering limit of liquid-cooled racks + Ethernet Scale-out at ~100k-GPU scale (built in 122 days, Grade A/B official case-study PDF); Colossus 2 is reported by the media as the world's first gigawatt-class AI training cluster (Grade B/C, see the data-center side of 03-cluster.md).

The four SuperPod elements (interconnect protocol, interconnect technology, system management software, power & cooling) come from the academic-review framework of the Chinese Society of Thermal Physics (Grade B); this document organizes the solution descriptions in the cases according to this framework.

Figure 2-1 | CloudMatrix 384 vs Vera Rubin NVL72 architecture comparison

Schematic: drawn based on arXiv 2506.12708 (Huawei/SiliconFlow, Grade A) and the NVIDIA Vera Rubin NVL72 product page (Grade A, preliminary specs); the CM384 optical-module count follows the SemiAnalysis figure (Grade B).

2. Glossary

3. Case Studies

3.1. CloudMatrix 384 and Atlas 900 SuperPod

Evidence level: A (official paper arXiv 2506.12708 and the HC2025 release)

3.1.1. Background

Traditional "one server, eight GPUs" servers interconnect via PCIe, with peer-to-peer bandwidth below 64 GB/s — insufficient for the high-bandwidth, low-latency needs of MoE models. Under the constraint of process-node limitations, Huawei chose the "SuperPod + cluster" route: using a larger Scale-up domain and systems-level engineering to compensate for the single-chip gap.

3.1.2. Solution

• Scale and cabinets: 384 Ascend 910C + 192 Kunpeng CPUs, 16 cabinets = 12 compute cabinets (48 nodes) + 4 communication cabinets; node form is 8 NPU + 4 Kunpeng CPUs + 7 on-board UB switch chips.
• UB network, three-layer planes: UB (Scale-up memory semantics), RDMA (parameter and data plane), VPC (service plane); the L2 layer is non-blocking, divided into 7 redundant sub-planes.
• Optical interconnect scale: 6912 400G optical modules per Pod, GPU-to-optical-module ratio 1:18 (5376 scale-up + 1536 scale-out, SemiAnalysis/Industrial Securities figure, Grade B).
• Conflicting latency figures listed in parallel: the paper's figure is "cross-node bandwidth close to on-chip bandwidth, single-hop latency close to 1 microsecond" (Grade A); there is also a forum figure of "200ns UB latency" (Grade C). The two figures are listed side by side, each noting the nature of its source.

3.1.3. Effects

• System dense BF16 is about 300 PFLOPS (Huawei's 300P figure and SemiAnalysis's figure coincide); SemiAnalysis separately offers a "about 1.7–2x the GB200 NVL72" comparison (Grade B — note that the NVL72 comparison dimension is BF16 compute, not overall performance).
• Inference side (CloudMatrix-Infer, see 05-inference-framework.md Case 1): Prefill 6688 tokens/s/NPU, Decode 1943 tokens/s/NPU.
• Deployment: the Atlas 900 SuperPod (CM384 hardware foundation) has over 300 units cumulatively deployed, serving 20+ customers (HC2025, Grade A).
• Harness interpretation: CM384 expands the L2 execution unit from "one server, 8 GPUs" to "16 cabinets, 384 GPUs", allowing L3's TP/EP orchestration to complete in-domain without crossing slow boundaries — this is the source of system-level reachability under conditions where the 910C's single-card parameters are insufficient to independently support a large MoE.

3.2. Atlas 950/960 SuperPoD: Moving Toward 10k-GPU SuperPods

Evidence level: A (official release at Huawei Connect 2025)

3.2.1. Background

As SuperPod scale evolves from hundreds of GPUs to thousands and even tens of thousands, three problems must be solved: interconnect-topology scalability, optical-interconnect cost, and cooling. Huawei provides its official answer to 10k-GPU SuperPods with the Atlas 950/960.

3.2.2. Solution

Companion cluster-level form: Atlas 950 SuperCluster = 64 × Atlas 950 SuperPoD, over 520k GPUs, 524 EFLOPS FP8; the Atlas 960 SuperCluster is planned at 2 ZFLOPS FP8 (2027).

3.2.3. Effects

• Performance-release figures: 4.91M tokens/s training, 19.6M tokens/s inference; system-level latency 2.1 microseconds (media figure, Grade B).
• Real-hardware progress: the first real-machine showcase at WAIC 2026 was the 1024-GPU version (1 EFLOPS FP8, 256 TB memory, 3-microsecond RTT, Grade C).
• Harness interpretation: UB-Mesh's "recursive expansion in 64-GPU steps" turns the mapping of L3 orchestration parallel strategies from manual design into a structured choice — the 64-GPU sub-domain naturally corresponds to a stable granularity for TP/EP, and SuperPod scale expansion no longer requires rebuilding the in-domain topology. The 1152 TB-class shared memory provides the physical precondition for L1's ultra-long context and KV pooling.

3.3. NVIDIA Vera Rubin NVL72 and NVL576

Evidence level: A (NVIDIA product page, GTC 2026 press release, and official blog, preliminary specs)

3.3.1. Background

After the GB200 NVL72 defined the rack-scale SuperPod, NVIDIA advanced two directions in parallel in the Vera Rubin generation: increasing single-cabinet density (NVL72) and expanding the multi-cabinet NVLink domain (NVL576).

3.3.2. Solution

• Rack structure: 18 compute trays + 9 NVLink trays; PCB midplane design with "no cables, no water lines, no fans"; the NVLink spine embeds 4 pre-integrated cable boxes (5000 copper cables, total length over 2 miles).
• Maintainability: the rack is about 4,000 lbs with about 1.3 million parts; compute-tray assembly time drops from 2 hours to 5 minutes; NVLink switch trays support hot swap in maintenance mode and continued operation with multiple failed trays.
• Parameters: 72 Rubin GPUs + 36 Vera CPUs; NVLink 6 rack total bandwidth 260 TB/s; HBM4 total capacity 20.7 TB (two conflicting figures for total HBM4 bandwidth — 1,400/1,580 TB/s — are listed in parallel, see 01-chip.md).
• NVL576: 8 NVL racks, a two-layer all-to-all topology, 576 GPUs in a single NVLink domain (copper + direct-attach optical), 2027 H2.

3.3.3. Effects

• Engineering effect: assembly time from 2 hours → 5 minutes and tray hot swap directly improve the "recovery time" metric in L5 observability — for the first time, SuperPod maintainability design has become a first-class metric in official marketing.
• Ecosystem effect: Microsoft Azure completed the first validation on 2026-03-14, with 2026 H2 delivery.
• New-form supplement: the NVIDIA Groq 3 LPX rack (256 LPUs, 128 GB SRAM, 40 PB/s memory bandwidth, 640 TB/s scale-up), co-designed with NVL72, indicates that SuperPods are beginning to be combined by division of labor into "training domains + inference domains" (performance multipliers follow vendor figures).
• Harness interpretation: NVL576 pulls 576 GPUs into a single memory-semantic domain, raising the ceiling of L1's cross-machine KV sharing and L3's in-domain orchestration by another order of magnitude; the "copper + direct-attach optical" hybrid physical layer heralds that the SuperPod physical form will select media by distance layer.

4. Practice Standards

Nature statement: The following is a proposed draft distilled from public industry practice, not the original text of an official industry standard. No mandatory OCP specification text for AI SuperPod racks was found (the fact that the MGX design was contributed to the OCP and ESUN work groups stands, but the original specification text could not be linked), and none should be fabricated. The group-level higher-level specification is in AGENTS.md in the same directory.

4.1. AGENTS.md Specification

4.1.1. AGENTS.md (AI Infra Group · Servers and SuperPod Direction)

# AGENTS.md —— AI Infra 组 · 服务器与超节点方向

> 本文为基于行业实践提炼的建议稿，非官方行业标准原文。
> 上位规范：01-AI-Infra组/AGENTS.md（不可逆动作禁止清单、变更窗口、可观测要求）。

## 角色与边界
- 角色：超节点运行侧智能体，负责超节点选型比对、域内拓扑与并行映射评估、
  互联带宽预算、散热与供电符合性核验、可维护性与交付节奏评估。
- 不负责：机房土建与供配电设计（转交基础设施团队）、芯片设计问题
  （转交 04-硬件研发组）、采购决策。
- 所有超节点参数标来源级别与口径性质（实测/发布/初步规格）。

## 环境假设
执行前必须显式声明：
- 超节点型号与代际、机柜数、卡数、CPU 配比。
- Scale-up 互联体系（NVLink / UB / 其他）、域内拓扑、单跳时延口径。
- 物理层介质（DAC/AEC/光）与光模块数量级。
- 散热形态与单机柜功率密度、设计 PUE。
- 目标模型与并行策略（TP/PP/DP/EP）。
- 未声明超节点型号与互联体系时，禁止产出域内带宽、时延、并行映射结论。

## 上下文加载顺序（Context Budget）
1. 超节点规格表（含来源级别）。
2. 目标模型并行策略与带宽需求。
3. 机房约束：功率密度、散热、PUE 红线。
4. 历史部署口径（部署套数、交付节奏，区分官方与传播口径）。

## 工具契约
- 拓扑与带宽测算输出结构化表格，每行附来源与级别。
- 时延口径冲突必须并列（如 CM384 的 1 微秒 vs 200ns 两口径）。
- 部署套数等商业数据区分官方（HC2025 300+ 套）与传播口径（750+ 套，C 级）。

## 任务执行流程（SOP）
- S1 需求解析：把模型需求换算为域内带宽、显存池、机柜功率三类约束。
- S2 口径统一：算力对比同精度同密稀口径。
- S3 候选生成：候选超节点组合 + 并行映射建议。
- S4 预算测算：域内带宽、光模块比例、机柜功率、散热负荷。
- S5 合规核验：PUE 红线（1.25/1.2）与液冷要求核对。
- S6 证据打包与人工确认。

## 验证与证据要求
- 域内时延必须注明口径来源与测量层次（芯片/交换/系统）。
- 算力对比注明精度与密集/稀疏。
- 可维护性结论基于官方工程数据（组装时间、热替换能力），不用转述估算。

## 失败与升级策略
- 关键规格仅 C 级来源 → 标 [待核实] 暂停比较。
- 机房功率或 PUE 无法满足候选方案 → 输出降级方案并升级人工。
- 交付日期仅为预告口径 → 明确标注为预告，不做排期承诺。

## 安全与合规红线
- 不引用未核实的 OCP 规范文本；如需引用注明"规范原文未获链接"。
- 液冷与供配电数据注明站点与代际，禁止跨代混比 PUE。
- 禁止残留非标准占位符。

## 输出格式
- 首行：可判定结论 + 阻塞项。
- 对照表：规模/互联/散热/可维护性/交付，附来源级别。
- 预算表：域内带宽、功率、散热负荷测算。
- 数值规范：参数带单位；范围用 ~ 连接；百分比数值与 % 之间不留空格。

## 评估与自检
- 时延与部署口径是否并列标注？
- 是否存在跨代 PUE 混比？
- 预告日期是否标注为预告？
- 是否残留非标准占位符？

4.2. SKILL.md Specification

4.2.1. SKILL.md (AI Infra Group · SuperPod Selection and In-Domain Budget Estimation)

---
name: supernode-selection-and-domain-budget
description: AI Infra 组超节点方向标准技能——超节点选型比对与域内带宽/功率
  预算测算。当需要进行超节点（NVL72/CM384/Atlas 950 等）选型、测算 Scale-up
  域内带宽预算、核验机柜功率与散热符合性、评估并行映射时使用。
  触发词：超节点、NVL72、CloudMatrix、Atlas 950、Scale-up、光模块、液冷。
version: 1.0
created: 2026-09-12
---

# AI Infra 组 · 超节点选型与域内预算测算

> 本文为基于行业实践提炼的建议稿，非官方行业标准原文。
> 上位规范：01-AI-Infra组/AGENTS.md。

## 适用场景
- 超节点代际/路线选型（NVIDIA 线 vs 华为线 vs 其他）。
- Scale-up 域内带宽预算：目标并行策略（TP/EP）是否在域内可完成。
- 机柜功率密度与散热负荷测算，PUE 红线符合性核验。
- 可维护性与交付节奏评估（组装时间、热替换、上市时间表）。
- 部署口径核验（官方套数 vs 传播口径）。

## 前置条件
- 已声明目标模型、并行策略意向、上下文长度需求。
- 已具备候选超节点的 A/B 级规格来源。
- 已知机房约束：单机柜功率上限、散热形态、PUE 红线。

## 输入
| 输入项 | 说明 | 必需 |
|---|---|---|
| 模型与并行需求 | 参数量、专家数、TP/EP 映射意向 | 是 |
| 候选超节点规格表 | 含来源级别（实测/发布/初步规格） | 是 |
| 机房约束 | 功率密度、散热、PUE、面积 | 是 |
| 交付时间窗 | 计划上线时间 | 条件必需 |

## 输出
| 输出项 | 说明 | 必需 |
|---|---|---|
| 可判定结论 | 通过 / 不通过 / 部分通过 + 阻塞项 | 是 |
| 选型对照表 | 规模/互联/散热/可维护性/交付，附来源级别 | 是 |
| 域内带宽预算 | 并行策略带宽需求 vs 域内供给 | 是 |
| 功率散热预算 | 机柜数、总功率、散热负荷、PUE 核验 | 是 |
| 口径冲突清单 | 时延/部署套数等两口径并列 | 条件必需 |
| 待人工确认项 | 选型建议、机房改造需求 | 是 |

## 执行步骤
1. 需求解析：模型 → 域内带宽、显存池、功率三类硬约束。
2. 口径统一：算力同精度同密稀；时延注明测量层次。
3. 选型对照：结构化表格逐项标注来源级别。
4. 域内预算：TP/EP 映射到域内拓扑，测算带宽需求与供给差值。
5. 功率散热测算：机柜数 × 单柜功率，对照机房上限与 PUE 红线。
6. 可维护性评估：组装/热替换/交付节奏对照运维目标（L5 恢复耗时）。
7. 风险与口径汇总：冲突并列 + [待核实]。
8. 交付：结论 + 表格 + 待确认项。

## 质量标准（DoD）
- 每项参数有来源与级别；口径冲突并列；预告日期标注为预告。
- 带宽与功率预算给出计算过程。
- 参照口径（不得直接套用）：CM384 = 384 昇腾 910C + 192 鲲鹏、16 机柜、
  6912 个 400G 光模块、约 300 PFLOPS 密集 BF16（A/B 级）；Atlas 950
  SuperPoD = 8192 卡、8 EFLOPS FP8、1152 TB、16.3 PB/s、160 机柜、
  2026 Q4（A级发布口径）；Vera Rubin NVL72 = 72 Rubin + 36 Vera、
  NVLink 6 260 TB/s、20.7 TB HBM4、2026 H2（A级初步规格）。

## 常见失败与处理
| 失败模式 | 现象 | 处理 |
|---|---|---|
| PUE 跨代混比 | 把 Colossus 2 的 1.12 与其他站点直接比较 | 注明站点与代际，禁止混比 |
| 传播口径当官方 | 把 750+ 套写成官方数字 | 区分 HC2025 官方 300+ 与 C 级传播口径 |
| 时延单口径 | 只引用 200ns 或 1 微秒之一 | 两口径并列 + 来源性质 |
| 预算漏光模块 | 忽略 GPU:光模块比例成本 | 预算表单列光模块项 |
| 预告当排期 | 把 NVL576 2027 H2 写成确定交付 | 标注"预告口径" |

## 示例
任务：为专家数 320 的 MoE 推理服务选择超节点形态并测算域内预算。
1. 需求解析：EP320 映射需要域内带宽覆盖全到全专家路由；TTFT 目标
   决定 Prefill 子系统规模。
2. 口径统一：CM384 用论文实测口径（PDC/EP320 实证），NVL72 用初步
   规格口径，分别标注。
3. 选型对照：CM384（384 卡、UB、光互联、PDC/EP320 有公开实证）vs
   NVL72（72 卡、NVLink 6、铜缆、托盘热替换）；各标级别。
4. 域内预算：EP320 全路由带宽测算对照 UB 与 NVLink 6 供给。
5. 功率散热：两方案机柜数与功率对比，对照机房 PUE 红线。
6. 交付：部分通过——EP320 场景 CM384 有实证支撑（Prefill 6688
   tokens/s/NPU，A级），NVL72 需待正式规格与基准；口径冲突清单与
   待确认项齐备。

4.3. On-the-Ground Checklist

5. Summary

The Servers and SuperPod direction is the "single-machine-ization" ring of the L2 execution layer in the six-layer Harness model. Three conclusions:

1. SuperPods have become the dominant form, and scale is still accelerating. From the GB200 NVL72 (72 GPUs) to the Atlas 950 SuperPoD (8192 GPUs) to the NVL576 (576 GPUs in a single domain, 8 racks), the boundary of the Scale-up domain has expanded from the rack to the data center, and the space for in-domain simplification of L3 orchestration has grown accordingly.
2. The two routes make different engineering tradeoffs but share the same goal. Huawei compensates for single-point process-node limits with optical interconnect + UB + system scale (CM384 has public evidence), while NVIDIA wins through process + rack-integration density (copper-cable dominant, hot-swappable trays); selection should follow the target model's parallel mapping and data-center constraints, not route allegiance.
3. Maintainability has become a first-class metric. The NVL72's "assembly from 2 hours → 5 minutes and hot-swappable trays" shows that SuperPod competition has shifted from peak parameters to full-lifecycle metrics that include recovery time — precisely the mapping of Harness's L5 observability metrics (detection latency, isolation granularity, recovery time) onto hardware form.

Information Gap Statement

The following items have not been confirmed by a Grade A/B source and are already marked [To be verified] in the body text:

1. The OCP specification text for AI SuperPod racks has not been linked (the fact that MGX was contributed to the OCP and ESUN work groups stands); no specification number should be fabricated.
2. CloudMatrix 384 UB latency: the paper figure "single-hop close to 1 microsecond" (Grade A) and the forum figure "200ns" (Grade C) are listed in parallel.
3. Atlas 900 SuperPod deployment count: official 300+ (HC2025, Grade A) and the disseminated figure of 750+ (Grade C) are listed in parallel.
4. Atlas 950 SuperPoD system-level latency of 2.1 microseconds is a media figure (Grade B); the WAIC 2026 real-machine 1024-GPU-version parameters are Grade C.
5. Atlas 950/960 training 4.91M and inference 19.6M tokens/s are release figures, with no independent measurement.
6. Vera Rubin NVL72 total HBM4 bandwidth figures of 1,400/1,580 TB/s are listed in parallel (see 01-chip.md).
7. xAI Colossus 2's scale (about 555k GB200/GB300) and PUE 1.12 are encyclopedia/media figures (Grade B/C).
8. NVIDIA Groq 3 LPX's "35x inference improvement per watt" is a vendor-claimed figure.
9. No public AGENTS.md/SKILL.md standard template for this direction was found; this document is a proposed draft.

6. References

1. Serving Large Language Models on Huawei CloudMatrix384 — Huawei/SiliconFlow, arXiv 2506.12708, 2025-06. https://arxiv.org/pdf/2506.12708
2. Leading a New Paradigm of AI Infrastructure with Pioneering SuperPod Interconnect Technology (Xu Zhijun's HC2025 keynote speech) — Huawei, 2025-09-18. https://www.huawei.com/cn/news/2025/9/hc-xu-keynote-speech
3. NVIDIA Vera Rubin NVL72 product page (preliminary specs) — NVIDIA, 2026-03. https://www.nvidia.com/en-us/data-center/vera-rubin-nvl72/
4. NVIDIA Vera Rubin Opens Agentic AI Frontier (GTC 2026 press release) — NVIDIA, 2026-03. https://investor.nvidia.com/news/press-release-details/2026/NVIDIA-Vera-Rubin-Opens-Agentic-AI-Frontier/
5. NVIDIA Vera Rubin POD: Seven Chips, Five Rack-Scale Systems, One AI Supercomputer — NVIDIA Developer Blog, 2026-03. https://developer.nvidia.com/blog/nvidia-vera-rubin-pod-seven-chips-five-rack-scale-systems-one-ai-supercomputer/
6. Research on Key Technologies and Industrial Development Trends of SuperPods — Chinese Society of Thermal Physics (reprint), 2025-2026. https://www.zgcsswdx.cn/info/24724.html
7. Success Story: xAI Colossus Cluster — Supermicro (written by ServeTheHome), 2024. https://www.supermicro.com/CaseStudies/Success_Story_xAI_Colossus_Cluster.pdf
8. Multiple AI Scale-Up Options Emerge — SemiEngineering. https://semiengineering.com/multiple-ai-scale-up-options-emerge/
9. NVIDIA GB200 NVL72 product page (design-side cross reference) — NVIDIA. https://www.nvidia.com/en-us/data-center/gb200-nvl72/
10. AI Infra (design side) — 02-行业赋能/04-硬件研发组/05-ai-infra.md, 2026-09-12.