集群互联与机房
1. 介绍
1.1. 背景
集群互联与机房是 AI Infra 中"看得见机器"的一层:对内承接超节点的 Scale-up 域,对外提供 Scale-out 网络与物理承载(供配电、散热、承重、占地)。这层正在同时经历两场变革:一是互联标准的多路线竞赛(NVLink、UALink、ESUN/SUE 与国内 EthLink/Eth-X/OISA/灵衢并行),二是机房约束从"配套专业"升格为政策红线——国家发改委等四部门《数据中心绿色低碳发展专项行动计划》(2024-07)规定到 2025 年底新建及改扩建大型/超大型数据中心 PUE 不超过 1.25、国家枢纽节点项目不超过 1.2、国家枢纽新建数据中心绿电占比超过 80%(A级,政策原文)。
需求侧的推力同样具体:AI 训练负载的功率摆动约 15 MW(对比传统非 AI 负载约 1.5 MW,B/C 级转述口径),Llama 3 训练中数万 GPU 同步功率变化产生数十 MW 级波动,逼近数据中心供电极限。互联与供电第一次成为与芯片同级的一阶约束。
1.2. 定义与范围
集群互联与机房方向在运行侧覆盖:
| 环节 | 内容 | 典型对象 |
|---|---|---|
| Scale-up 互联 | 域内高带宽低时延网络与标准 | NVLink 5/6、UALink 1.0/2.0、ESUN/SUE、灵衢 2.0 |
| Scale-out 互联 | 跨节点/跨机柜网络 | InfiniBand(Quantum-X800)、RoCE(Spectrum-X800) |
| 国内互联协议 | 国产开放标准 | EthLink(字节)、Eth-X(腾讯)、OISA(中国移动)、灵衢(华为) |
| 光互联 | 物理层演进 | DAC/AEC、NPO/CPO、全光互联 |
| 供配电 | 供电架构与波动治理 | 储能缓冲、800V 高压直流、电源架削峰 |
| 散热与能耗 | 液冷路线与 PUE 治理 | 冷板式、浸没式、喷淋式、iCooling 类 AI 调优 |
边界说明:超节点域内实现见 02-supernode.md;PUE 政策的完整条文与能耗统计见本组 03-cluster.md 与 04-硬件研发组的机房相关内容;GB300 电源架电容削峰参数(约 65 J/GPU、电网峰值降低最高 30%,A/B 级厂商口径)在 1.4.3 节概述。
1.3. 在 AI Harness 体系中的定位
集群网络是 L2 执行层的"总线",供配电与 PUE 是 L6 治理层的物理投影:
| Harness 层 | 本方向的作用 |
|---|---|
| L1 上下文工程 | Scale-up 网络承载内存语义访问,支撑跨卡 KV 共享 |
| L2 工具与执行 | 集群网络 = 分布式执行总线 |
| L3 编排与控制 | 拓扑感知调度依赖网络层级信息(TP 不跨 Scale-up 域) |
| L5 评估与观测 | 网络利用率(如 Colossus 1 的 95% 口径,C 级 )与通信效率 |
| L6 治理与安全 | PUE/绿电考核、能耗预算是 L6 护栏的物理形态 |
瓶颈层判断:本方向的重心在 L2(总线能力)与 L6(能耗治理)。AI 负载功率波动 + 储能缓冲(如特斯拉 Megapack)的组合是 L3 编排与物理基础设施协同的典型案例——算力调度必须与电力调度联合设计,这是本方向独有的跨层问题。
1.4. 发展现状
1.4.1. Scale-up 三线竞赛
| 路线 | 主导方 | 现状(截至 2026-09-12) |
|---|---|---|
| NVLink / NVLink Fusion | NVIDIA | NVLink 6 单 GPU 3.6 TB/s(GTC 2026);NVLink Fusion 允许第三方定制 CPU/GPU 接入,单端口最高 900 GB/s,但不开放 NVLink Switch 芯片与 PHY,每个 XPU 须连接 NVIDIA 产品入网(封闭 + 半开放) |
| UALink | UALink 联盟(AMD、AWS、Google、Meta、Microsoft 等发起) | 1.0(2025-04):单 Pod 最多 1024 加速器、每通道 200 GT/s、内存语义访问;2.0(2026-04)新增 In-Network Compute(交换节点内嵌 all-reduce)、Chiplet 1.0 与 UCIe 3.0 兼容;成员超 85 家(博通 2025-10 退出董事会转向 OCP ESUN) |
| ESUN / SUE | OCP(博通主导) | 以太网基 Scale-up;博通 Tomahawk 6(102 Tbps、2048 端口、时延小于 400ns)为芯片基础;与 UALink 并行的开放路线 |
撰写纪律:UALink 原生交换芯片截至 2026-09 未量产,AMD Helios 机架以 800GbE 物理层承载 UALink 协议(UALoE)过渡——必须写明"原生硅未就绪、以太网桥接过渡",禁止写成"UALink 已规模商用"。
1.4.2. 国内互联协议
- 灵衢(Unified Bus)2.0:华为 HC2025 宣布作为开放标准发布技术规范;OSI 各层内置可靠性,光路径 100ns 级故障检测与保护倒换(A级)。
- EthLink(字节):1-4 个以太网接口、单域最大 1024 GPU(B 级学术综述口径)。
- Eth-X(腾讯) 与 OISA(中国移动):OISA 主张统一报文、多语义融合、多级流控重传(B 级综述口径)。
- 国产 IP 进展:芯动科技 2026-05 发布国产全套 UALink IP(112G 量产、224G 已流片);ODCC 2026-04 发布 UALink 测试验证服务(C 级,逐条引用需再核来源)。
1.4.3. Scale-out、光互联与机房
- Scale-out 双栈:InfiniBand(Quantum-X800)与以太网(Spectrum-X800 及 CPO 交换机,宣称 5 倍能效、10 倍网络韧性,厂商口径);ConnectX-9 SuperNIC 每 GPU 1.6 Tb/s(GTC 2026)。
- 光互联演进:NPO(光电芯片下放 GPU 板,米级 → 厘米级)已落地;CPO(光电共封装,厘米级 → 毫米级)演进中;曦智 LightSphere X 超节点采用机柜内 DAC + 跨柜 AEC(5-7 m)以减少光模块(B/C 级)。
- 供配电与波动治理:GB300 NVL72 电源架内置电解电容(约 65 J/GPU),NVIDIA 宣称相同 GPU 输出下电网侧峰值功率降低最高 30%(A/B 级厂商口径);《促进数字化绿色化协同转型发展实施方案(2026—2030年)》把 800V 高压直流、超百千瓦单机柜、液冷 + 余热回收、"算电协同"调度首次量化落地(B 级,新华社转述)。
- 液冷:冷板式成熟度最高(据行业估计约占 65% 份额,C 级);液冷渗透率约 8%(2024)→ 据行业预测 35%(2026/2027,C 级);贵安新区喷淋液冷样板单机柜算力 +30%、能耗 -40%、PUE 常年小于 1.1(B 级);华为云贵安年均 PUE 1.1、iCooling AI 调优能耗下降 8~15%(B 级)。
- 能耗大盘:全球数据中心用电 415 TWh(2024)→ 945 TWh(2030 预测,IEA,B 级);中国算力中心用电 1960 亿 kWh(2025)→ 5000 亿(2030E,信通院,B 级)。
图 3-1|Scale-up 互联标准三线竞赛与国内协议格局
示意图:基于 UALink 联盟、OCP、NVIDIA GTC 2026 与华为 HC2025 公开信息(A/B 级)及《数据中心绿色低碳发展专项行动计划》(A 级)绘制;三态(已量产/已发布/过渡)为本文分析口径。
2. 名词解释
| 术语 | 英文 / 缩写 | 释义 |
|---|---|---|
| 纵向扩展 | Scale-Up | 提升单域内互联密度(更多卡进一个内存语义域) |
| 横向扩展 | Scale-Out | 增加跨节点/跨机柜网络规模(集群级扩展) |
| 统一加速器互联 | UALink | 开放加速器互联标准:1.0 支持 1024 加速器/Pod,2.0 新增 In-Network Compute |
| 以太网纵向扩展 | ESUN(Ethernet for Scale-Up Networking) | OCP 下博通主导的以太网基 Scale-up 工作组路线 |
| SUE | Scale-Up Ethernet | 以太网基 Scale-up 协议(博通体系),与 ESUN 关联 |
| UALoE | UALink over Ethernet | 以 800GbE 物理层承载 UALink 协议的过渡方案(原生交换芯片未量产) |
| 网络内计算 | In-Network Compute | 在交换节点内嵌集合通信计算(如 all-reduce)的技术 |
| 远程直接内存访问 | RDMA | 绕过内核的远端内存读写,Scale-out 训练网络的基础能力 |
| 基于融合以太网的 RDMA | RoCE | 在以太网上实现 RDMA,需 QoS/ECN/PFC 配合 |
| 无损以太网参数 | QoS/ECN/PFC | 显式拥塞通知与优先级流控,RoCE 无损网络的三大配置面 |
| 直连铜缆 | DAC | 短距(机柜内)铜互连,无光电器件、成本与功耗最低 |
| 近封装光学 | NPO | 光电芯片下放 GPU 板级,缩短电通道 |
| 共封装光学 | CPO | 光引擎与交换芯片共封装,能效与集成度进一步提升 |
| 电源使用效率 | PUE | 数据中心总能耗与 IT 设备能耗之比,政策红线指标 |
| 算电协同 | Compute-Power Coordination | 算力调度与电力调度联合优化,2026 方案首次量化落地 |
| 高压直流供电 | 800V HVDC | 数据中心高压直流供电架构,2026 方案推广方向 |
| 功率摆动 | Power Oscillation | 训练负载同步启停造成的功率波动,AI 机房特有供电难题 |
| 储能缓冲 | Battery Buffer | 以电池储能平抑功率波动,如特斯拉 Megapack(单机 3.9 MWh 口径) |
| 故障检测与保护倒换 | Fault Detection & Protection Switching | 灵衢 2.0 的光路径 100ns 级检测与倒换能力 |
| 内存语义访问 | Memory-Semantic Access | 以访存语义(load/store)而非消息语义使用远端资源 |
3. 案例
3.1. UALink 1.0 到 2.0:开放互联路线的推进与过渡态
证据级别:B(官方规范发布事实经多源交叉证实,部分国产进展为 C 级)
3.1.1. 背景
NVLink 生态的封闭性(不开放 Switch 芯片与 PHY,接入方必须连接 NVIDIA 产品入网)促使 AMD、AWS、Google、Meta、Microsoft 于 2024-10 成立 UALink 联盟,寻求开放的 Scale-up 标准。
3.1.2. 方案
- 1.0(2025-04):UALink 200G——单 Pod 最多 1024 加速器、每通道 200 GT/s,支持内存语义访问、显存共享与 Switch 组网;64B flit/640B 包结构,平均时延低于以太网数百 ns(B 级)。
- 2.0(2026-04,含三份配套规范:200G 物理层、管理框架、Chiplet 接口):新增 In-Network Compute(交换节点内嵌 all-reduce);Chiplet 1.0 与 UCIe 3.0 兼容;成员超 85 家。
- 过渡态:原生 UALink 交换芯片截至 2026-09 未量产;AMD 2026 H2 的 Helios 机架以 800GbE 物理层承载 UALink 协议(UALoE)。
- 生态动作:博通 2025-10 退出 UALink 董事会转向 OCP ESUN;ODCC 2026-04 发布 UALink 测试验证服务;芯动科技 2026-05 发布国产全套 UALink IP(112G 量产、224G 已流片,C 级 )。
3.1.3. 效果
- 规范层效果:1024 加速器单域 + In-Network Compute 的组合,为非 NVIDIA 体系提供了与 NVLink 5/6 对位的开放选项;In-Network Compute 把集合通信下沉到交换层,直接改善 L3 编排中的同步开销。
- 现实约束:截至信息截止日,"开放标准已发布"与"原生硅未量产"并存——开放路线的真实可用性取决于 2026-2027 年的首批芯片交付(1.0 首批芯片预计 2026 年面世,B 级)。
- Harness 解读:In-Network Compute 是 L2 执行层向网络层下沉的标志——执行原语(all-reduce)从 GPU 迁移到交换机,改变的是 L2 的实现边界而非六层结构本身。
3.2. 华为灵衢(UnifiedBus)2.0 开放:国产互联的标准输出
证据级别:A(华为全联接大会 2025 官方发布)
3.2.1. 背景
华为在 HC2025 明确"中国半导体制造工艺将长期落后、可持续算力只能基于实际可获得的制程"的判断后,互联成为国产路线的战略支点:单芯片受限,就把互联做到极致,并把互联规范本身开放为标准,扩大体系边界。
3.2.2. 方案
- 将 UnifiedBus 2.0 技术规范作为开放标准发布。
- 可靠性内置:OSI 各层内置可靠性机制;光路径 100ns 级故障检测与保护倒换。
- 与超节点产品线联动:灵衢是 CloudMatrix 384、Atlas 950/960 SuperPoD 的 UB 网络规范基础(超节点实现见 02-supernode.md)。
3.2.3. 效果
- 把国产 Scale-up 从"私有实现"升级为"可对外遵循的规范",为第三方接入昇腾体系(配套 CANN 开放、Mind 套件与 openPangu 开源)提供接口基础。
- 100ns 级光路径故障检测直接服务 L5 观测(故障检测时延指标)与 L3 容错编排(保护倒换)。
- 效果数据均为发布口径,生态采纳度(第三方芯片/整机跟进数量)截至信息截止日未获独立统计,标注为缺口。
3.3. xAI Colossus:十万卡集群的机房工程
证据级别:A/B(Supermicro 官方案例 PDF);Colossus 2 部分为 B/C(百科与媒体口径)
3.3.1. 背景
十万卡级训练集群把互联与机房约束推到极限:网络选型、供电波动、散热密度都必须在建成时间压力下同时解决。Colossus 1 是公开度最高的极端案例(10 万 H100,122 天建成,首批 1 万卡 19 天上线;后 92 天扩展至 20 万卡,约 300 MW)。
3.3.2. 方案
- 网络:弃用 InfiniBand,采用 Spectrum-X 400G 以太网;SN5600 交换机(51.2 Tbps);每台服务器 9 个 400GbE 口(8 个 BlueField-3 + 1 个 ConnectX-7),共 3.6 Tbps;案例口径网络吞吐效率 95%(A/B 级官方案例)。
- 供电:特斯拉 Megapack 电池缓冲(单个 3.9 MWh)应对训练负载 10-20 MW 级功率突变;约 300 MW 站点规模。
- 散热:Supermicro 4U 液冷机架(8 台 4U 服务器 + CDU),单机柜 64 GPU;EB 级全 NVMe 存储(VAST Data → DDN EXAScaler/Infinia 两阶段)。
- Colossus 2(B/C 级口径):2026-01-17 宣布投运,全球首个吉瓦级 AI 训练集群;约 55.5 万 GB200/GB300(等效 140 万 H100),1 GW 起步、目标 2 GW;41 台天然气涡轮机 + 接近 1.2 GWh Megapack;PUE 1.12(全液冷口径,C 级);伴随环保争议(EPA 违规裁定、NAACP 诉讼)。
3.3.3. 效果
- 网络路线效果:证明 RoCE 系以太网在十万卡训练场景可替代 IB,但代价是"激进 QoS 与 flow 级遥测"的运维复杂度(C 级运维口径 )。
- 口径冲突并列(必须保留):Colossus 1 MFU 约 11% 的口径(C 级)与"网络利用率 95%+"口径(C 级)并存——两者定义不同(训练效率 vs 网络效率),易被混用,须分别注明指标定义;PUE 存在 1.05/1.1(Colossus 1,R03 口径)与 1.12(Colossus 2 全液冷口径)多个数值,引用 PUE 必须注明站点与代际,禁止跨代混比。
- Harness 解读:Colossus 展示了 L6 治理层的物理化——功率波动治理(Megapack)与能耗口径(PUE)本质上是把 L6 的"预算与护栏"翻译成电力与散热设备;网络利用率 95% 口径则是 L5 观测在 Scale-out 网络上的直接映射。
4. 实践标准
性质声明:以下为基于行业公开实践提炼的建议稿,非官方行业标准原文。本方向以政策文件(PUE 红线)与事实标准(IB/RoCE 配置实践、互联规范)为主,无 IEEE/ISO/JEDEC 专用集群标准。组级上位规范见同目录 AGENTS.md。
4.1. AGENTS.md 规范
4.1.1. AGENTS.md(AI Infra 组 · 集群互联与机房方向)
# AGENTS.md —— AI Infra 组 · 集群互联与机房方向
> 本文为基于行业实践提炼的建议稿,非官方行业标准原文。
> 上位规范:01-AI-Infra组/AGENTS.md(不可逆动作禁止清单、变更窗口、可观测要求)。
## 角色与边界
- 角色:集群互联与机房运行侧智能体,负责网络拓扑与配置分析、
Scale-up/Scale-out 选型比对、PUE 与能耗核验、功率波动治理方案评估。
- 不负责:网络全局配置(RoCE QoS/ECN/PFC、IB 子网管理器)的直接修改
(属禁止清单)、供配电工程设计、互联标准组织的立场声明。
- PUE 等政策条文只引用有 A 级原文的条目;仅媒体转述的政策标注 [待核实]。
## 环境假设
执行前必须显式声明:
- Scale-up 互联体系与代际(NVLink/UB/UALink/其他)与单域规模。
- Scale-out 网络形态(InfiniBand / RoCE / Spectrum-X / 其他)、交换层级、
跨机带宽、QoS/ECN/PFC 配置现状。
- 机房参数:单机柜功率上限、散热形态、设计 PUE 与实测 PUE、绿电占比。
- 训练负载画像:同步作业数、功率摆动幅度、储能/削峰设施。
- 未声明网络形态与机房参数时,禁止产出通信效率、PUE、扩展性结论。
## 上下文加载顺序(Context Budget)
1. 网络拓扑与配置现状(层级、带宽、QoS)。
2. 机房参数与能耗台账。
3. 目标负载的通信模式(集合通信规模、频率)。
4. 互联标准动态(区分已量产/已发布/过渡三态)。
## 工具契约
- 网络诊断通过受控工具(IB 探测、RoCE 遥测、nccl-tests)输出结构化结果。
- nccl-tests 输出必须含 busbw、buffer size、拓扑范围。
- 互联标准动态引用必须注明三态与来源级别。
## 任务执行流程(SOP)
- S1 需求解析:把负载需求换算为域内带宽、跨节点带宽、时延、功耗四类约束。
- S2 现状盘点:拓扑、QoS、机房参数、能耗台账。
- S3 候选生成:网络配置调整、拓扑调整、散热/供电改造候选方案。
- S4 受控验证:在维护分区或影子链路验证,记录 busbw 与时延。
- S5 合规核验:PUE 对照红线(1.25/1.2)、绿电占比对照要求。
- S6 证据打包:变更方案附回滚命令,进入审批队列。
## 验证与证据要求
- 通信结论附 nccl-tests 或遥测原始数据;网络利用率注明指标定义
(吞吐效率 vs 链路利用率,禁止混用)。
- PUE 结论注明站点、代际、测量周期;禁止跨代/跨站点混比。
- 互联标准结论注明三态;过渡方案(如 UALoE)不得写成规模商用。
## 失败与升级策略
- RoCE 网络出现不可归因的拥塞或丢包 → 停止自动调优,升级网络工程师。
- 功率摆动接近供电上限 → 触发削峰预案并上报,不得自行调整作业调度策略。
- 实测 PUE 逼近红线 → 输出能耗报告并启动散热优化评估。
## 安全与合规红线
- 网络全局配置修改属禁止清单动作,只产出待审批方案。
- 政策引用限于 A 级原文;转述条文标 [待核实]。
- 出口管制语境下保持中性表述。
- 禁止残留非标准占位符。
## 输出格式
- 首行:可判定结论 + 阻塞项。
- 拓扑与配置表:层级、带宽、QoS 参数、来源。
- 能耗报告:PUE、实测功耗、红线差值、绿电占比。
- 标准三态表:各路线的量产/发布/过渡状态与来源级别。
- 数值规范:参数带单位;范围用 ~ 连接;百分比数值与 % 之间不留空格。
## 评估与自检
- 网络利用率与 MFU 等指标定义是否分清?
- PUE 是否注明站点与代际?
- 互联标准是否标注三态?
- 是否产出过禁止清单内的直接修改动作?(必须为否)
- 是否残留非标准占位符? 4.2. SKILL.md 规范
4.2.1. SKILL.md(AI Infra 组 · 集群网络诊断与机房能耗核验)
---
name: cluster-network-and-facility-audit
description: AI Infra 组集群互联与机房方向标准技能——网络配置诊断、互联
选型三态比对、PUE 与能耗红线核验。当需要诊断 RoCE/IB 网络问题、评估
Scale-up/Scale-out 路线、核验 PUE 政策符合性、评估功率波动治理时使用。
触发词:RoCE、InfiniBand、UALink、NVLink Fusion、PUE、液冷、功率摆动。
version: 1.0
created: 2026-09-12
---
# AI Infra 组 · 集群网络诊断与机房能耗核验
> 本文为基于行业实践提炼的建议稿,非官方行业标准原文。
> 上位规范:01-AI-Infra组/AGENTS.md。
## 适用场景
- 训练作业通信异常(掉速、超时、集合通信失败)的网络侧诊断。
- Scale-up 互联路线选型(NVLink / UALink / ESUN / 灵衢,三态比对)。
- Scale-out 网络选型(IB vs RoCE)与 QoS 配置评审。
- PUE 政策符合性核验(1.25/1.2 红线)与能耗预算评估。
- 功率摆动治理方案评估(削峰、储能缓冲、算电协同)。
## 前置条件
- 已声明网络形态、交换层级、QoS 现状与机房参数。
- nccl-tests、网络遥测工具可用;变更窗口机制可用。
- 能耗台账(PUE、分项功耗)可查询。
## 输入
| 输入项 | 说明 | 必需 |
|---|---|---|
| 网络拓扑与配置 | 层级、带宽、QoS/ECN/PFC、固件版本 | 是 |
| 负载通信画像 | 集合通信规模、频率、作业同步结构 | 是 |
| 机房参数 | 功率上限、散热形态、PUE 实测、绿电占比 | 是 |
| 故障现象 | 告警、日志路径、时间窗口 | 条件必需 |
| 互联标准动态 | 各路线三态与来源级别 | 否 |
## 输出
| 输出项 | 说明 | 必需 |
|---|---|---|
| 可判定结论 | 通过 / 不通过 / 部分通过 + 阻塞项 | 是 |
| 网络诊断报告 | busbw、时延、拥塞/丢包定位,附原始数据 | 条件必需 |
| 三态比对表 | 各互联路线量产/发布/过渡状态 | 条件必需 |
| 能耗报告 | PUE、实测功耗、红线差值、绿电占比 | 是 |
| 变更方案 | 配置调整建议 + 影响面 + 回滚命令 | 条件必需 |
| 证据表 | 工具、版本、命令、日志路径、数值、时间窗口 | 是 |
## 执行步骤
1. 环境校验:核对网络形态、QoS 现状、机房参数;确认不触及禁止清单。
2. 基线确认:nccl-tests 基准 busbw、能耗台账基线。
3. 问题定位(诊断场景):分层定位——链路(光模块/线缆)、交换
(拥塞/ECN)、配置(QoS/PFC)、软件(NCCL 超时)。
4. 选型比对(选型场景):按三态(已量产/已发布规范/规划过渡)比对
候选路线,标注来源级别。
5. 能耗核验:PUE 对照红线,分项功耗对照预算,绿电占比对照要求。
6. 受控验证:配置类建议先在维护分区验证并记录 busbw 差异。
7. 交付:诊断/比对/核验结论 + 变更方案(含回滚)+ 待人工确认项。
## 质量标准(DoD)
- 网络结论附 busbw 等原始数据;指标定义明确(吞吐效率 vs 链路利用率)。
- PUE 注明站点与代际,无跨代混比;红线核验有计算过程。
- 互联路线标注三态;过渡方案不写成规模商用。
- 变更方案含回滚命令;未执行禁止清单动作。
- 参照口径(不得直接套用):PUE 红线 1.25/1.2(A级政策);八大枢纽平均
PUE 约 1.3、最先进 1.04(B级,国家能源局);Tomahawk 6 = 102 Tbps /
2048 端口 / 小于 400ns(B级);灵衢 2.0 光路径 100ns 级检测(A级发布)。
## 常见失败与处理
| 失败模式 | 现象 | 处理 |
|---|---|---|
| 指标混用 | 把网络利用率 95% 当训练效率引用 | 分别注明指标定义与来源级别 |
| 过渡态误判 | 把 UALoE 桥接写成 UALink 规模商用 | 标注三态与过渡事实 |
| QoS 盲调 | 未测基线直接改 ECN/PFC | 先测基线,变更走审批窗口 |
| PUE 混比 | 跨站点/跨代比较 PUE | 注明站点与代际 |
| 红线误引 | 引用仅媒体转述的政策条文 | 标 [待核实] 或仅引 A 级原文 |
| 波动治理越权 | 自行调整全局作业调度应对功率摆动 | 升级人工,输出预案建议 |
## 示例
任务:某 8 千卡训练集群作业频繁出现集合通信超时,怀疑 RoCE 网络配置。
1. 环境校验:确认 Spectrum-X 类以太网、交换层级、QoS/ECN/PFC 现状、
固件版本;确认网络全局配置修改不在本轮直接执行。
2. 基线确认:nccl-tests 分层实测 busbw(节点内/跨机/跨柜),记录时间窗口。
3. 分层定位:跨柜 busbw 显著低于节点内,ECN 标记率在跨柜段异常升高。
4. 候选方案:ECN 阈值调整、PFC 策略收敛、故障光模块替换,各附影响面
与回滚命令。
5. 受控验证:维护分区验证 ECN 调整,busbw 恢复至基线的 97% 以上。
6. 能耗核验:同步核对 PUE 实测与红线差值(本例无变化)。
7. 交付:结论"部分通过,跨柜段需按方案调整 ECN 并更换 2 个光模块";
变更方案进入审批队列;证据表含命令、日志路径、时间窗口。 4.3. 落地检查清单
| 序号 | 检查项 | 判定标准 | 必需 |
|---|---|---|---|
| 1 | 环境声明 | 网络形态、QoS 现状、机房参数已声明 | 是 |
| 2 | 基线数据 | busbw 基准与能耗台账已记录 | 是 |
| 3 | 指标定义 | 网络利用率/吞吐效率定义明确不混用 | 是 |
| 4 | 三态标注 | 互联路线标注量产/发布/过渡 | 是 |
| 5 | PUE 口径 | 注明站点与代际,无跨代混比 | 是 |
| 6 | 红线核验 | 对照 1.25/1.2 红线与绿电要求 | 是 |
| 7 | 政策来源 | 仅引 A 级原文,转述条文标 | 是 |
| 8 | 变更回滚 | 网络配置方案含回滚命令与窗口 | 条件必需 |
| 9 | 禁止清单 | 未直接修改全局网络配置 | 是 |
| 10 | 波动治理 | 功率摆动预案经人工确认 | 条件必需 |
| 11 | 证据齐备 | 工具、命令、日志路径、时间窗口齐备 | 是 |
| 12 | 占位符清理 | 无 XX、___ 等非标准占位符 | 是 |
5. 总结
集群互联与机房方向覆盖 Harness 六层模型中 L2 的"总线"与 L6 的"物理投影"。三条结论:
- 互联标准进入多路线长期并存格局。 NVLink(封闭 + 半开放)、UALink(开放、In-Network Compute)、ESUN/SUE(以太网基)三线竞赛,加上国内 EthLink/Eth-X/OISA/灵衢 2.0,短期不存在单一胜者;选型纪律是按"已量产/已发布规范/规划过渡"三态评估,UALoE 等过渡方案不得写成规模商用。
- 机房约束已从配套升格为红线。 PUE 1.25/1.2 与绿电占比 80% 是有 A 级政策原文的硬约束;AI 负载 15 MW 级功率摆动使供电治理(储能缓冲、削峰、算电协同)成为互联之外的第二约束,L3 算力调度必须与电力调度联合设计。
- 口径纪律在本方向尤其重要。 网络利用率与训练效率、不同站点不同代际的 PUE、发布与传播口径的部署数据,都是本方向的高发混用点;AGENTS.md 的指标定义分离与站点代际标注是强制要求。
信息缺口声明
以下条目未获 A/B 级来源确认,已在正文标注 :
- 本方向未检索到 IEEE/ISO/JEDEC 面向 AI 集群互联与机房的专用标准;以政策文件 + 互联规范 + 事实标准表述,不得虚构标准编号。
- 《促进人工智能与能源双向赋能行动方案》(2026-04 四部门,"新建大型 AI 数据中心 100% 液冷、2028 年前存量风冷改造")仅见媒体转述,引用前须核实原文。
- Colossus 1 MFU 约 11% 与网络利用率 95%+ 均为 C 级且定义不同,已分别注明,。
- Colossus 2 精确规模(约 55.5 万卡)、PUE 1.12、41 台燃气轮机与 1.2 GWh Megapack 均为百科/媒体口径(B/C 级)。
- 芯动科技 UALink IP、瀚博 ODCC 互通测试为 C 级,逐条引用需再核来源。
- EthLink/Eth-X/OISA 参数(1024 GPU 域、统一报文等)来自学术综述转述(B 级),未获各公司一手规范文本。
- 液冷渗透率 8% → 35%、冷板式 65% 份额为 C 级行业观察,正文已改写为"据行业估计/预测"。
- GB300 电源架 65 J/GPU 与电网峰值 -30% 为厂商口径(A/B 级),无独立复测。
- 未检索到本方向公开 AGENTS.md/SKILL.md 标准范本,本文为建议稿。
6. 参考资料
- 数据中心绿色低碳发展专项行动计划 — 国家发展改革委等四部门,2024-07。https://www.ndrc.gov.cn/
- 以开创的超节点互联技术,引领AI基础设施新范式(灵衢 2.0 开放)— 华为,2025-09-18。https://www.huawei.com/cn/news/2025/9/hc-xu-keynote-speech
- UALink 1.0/2.0 规范发布 — UALink Consortium,2025-04 / 2026-04。https://ualinkconsortium.org/
- Multiple AI Scale-Up Options Emerge — SemiEngineering。https://semiengineering.com/multiple-ai-scale-up-options-emerge/
- 超节点关键技术与产业发展态势研究(互联协议与国内协议综述)— 中国热物理学会(转载),2025-2026。https://www.zgcsswdx.cn/info/24724.html
- Success Story: xAI Colossus Cluster — Supermicro(ServeTheHome 撰写),2024。https://www.supermicro.com/CaseStudies/Success_Story_xAI_Colossus_Cluster.pdf
- Energy and AI — 国际能源署(IEA),2025-04。https://www.iea.org/reports/energy-and-ai
- 促进数字化绿色化协同转型发展实施方案(2026—2030年)— 中央网信办等七部门,2026-09(新华社转述)。https://www.gov.cn/
- NVIDIA Spectrum-X / ConnectX-9 相关发布(GTC 2026)— NVIDIA。https://www.nvidia.com/en-us/networking/
- The Llama 3 Herd of Models(功率波动与训练负载背景)— Meta,2024。https://arxiv.org/abs/2407.21783
Cluster Interconnect and Data Centers
1. Introduction
1.1. Background
Cluster interconnect and data centers are the layer of AI Infra where "the machines are visible": inward, they carry the Scale-up domain of supernodes, while outward they provide the Scale-out network and physical hosting (power distribution, cooling, load-bearing, and floor space). This layer is undergoing two transformations at once: first, a multi-route competition in interconnect standards (NVLink, UALink, and ESUN/SUE running in parallel with the domestic EthLink/Eth-X/OISA/LingQu), and second, data-center constraints being elevated from a "supporting discipline" to a policy red line — the "Special Action Plan for Green and Low-Carbon Development of Data Centers" issued by the NDRC and three other ministries (2024-07) requires that by the end of 2025, newly built and expanded large/extra-large data centers have a PUE no higher than 1.25, projects at national hub nodes no higher than 1.2, and the green-power share of newly built data centers at national hubs exceeding 80% (A-level, policy original text).
The demand-side push is equally concrete: AI training workloads exhibit power oscillations of roughly 15 MW (versus roughly 1.5 MW for traditional non-AI workloads, B/C-level relayed framing), and in Llama 3 training, synchronised power changes across tens of thousands of GPUs produce oscillations on the order of tens of MW, approaching the power-supply limits of a data center. For the first time, interconnect and power supply have become first-order constraints on par with the chips themselves.
1.2. Definition and Scope
The Cluster Interconnect and Data Center track covers, on the operational side:
| Stage | Content | Typical Objects |
|---|---|---|
| Scale-up interconnect | In-domain high-bandwidth, low-latency networks and standards | NVLink 5/6, UALink 1.0/2.0, ESUN/SUE, LingQu 2.0 |
| Scale-out interconnect | Cross-node / cross-rack networks | InfiniBand (Quantum-X800), RoCE (Spectrum-X800) |
| Domestic interconnect protocols | Domestic open standards | EthLink (ByteDance), Eth-X (Tencent), OISA (China Mobile), LingQu (Huawei) |
| Optical interconnect | Physical-layer evolution | DAC/AEC, NPO/CPO, all-optical interconnect |
| Power distribution | Power-supply architecture and oscillation governance | Battery buffer, 800V HVDC, power-shell peak shaving |
| Cooling and energy | Liquid-cooling routes and PUE governance | Cold plate, immersion, spray, iCooling-type AI tuning |
Boundary notes: for in-supernode (domain) implementations see 02-supernode.md; for the full text of the PUE policy and the energy statistics see 03-cluster.md in this group and the data-center-related content in Group 04 - Hardware Development; the GB300 power-shell capacitive peak-shaving parameters (about 65 J/GPU, grid peak reduction of up to 30%, A/B-level vendor framing) are outlined in Section 1.4.3.
1.3. Positioning in the AI Harness Architecture
The cluster network is the "bus" of the L2 execution layer, while power distribution and PUE are the physical projection of the L6 governance layer:
| Harness Layer | Role of This Track |
|---|---|
| L1 Context Engineering | The Scale-up network carries memory-semantic access, supporting cross-GPU KV sharing |
| L2 Tools and Execution | Cluster network = distributed execution bus |
| L3 Orchestration and Control | Topology-aware scheduling depends on network-level information (TP does not cross Scale-up domains) |
| L5 Evaluation and Observability | Network utilization (e.g. the 95% figure for Colossus 1, C-level) and communication efficiency |
| L6 Governance and Security | PUE/green-power assessment and energy budgets are the physical form of the L6 guardrails |
Bottleneck layer assessment: the focus of this track lies in L2 (bus capability) and L6 (energy governance). The combination of AI-load power oscillation plus battery buffering (e.g. Tesla Megapack) is a typical case of coordination between L3 orchestration and physical infrastructure — computing scheduling must be co-designed with power scheduling, a cross-layer problem unique to this track.
1.4. Current State of Development
1.4.1. The Three-Way Scale-up Race
| Route | Leader | Current State (as of 2026-09-12) |
|---|---|---|
| NVLink / NVLink Fusion | NVIDIA | NVLink 6 single-GPU 3.6 TB/s (GTC 2026); NVLink Fusion allows third-party custom CPU/GPU access at up to 900 GB/s per port, but does not open the NVLink Switch chip and PHY, and every XPU must connect through an NVIDIA product to join the network (closed + semi-open) |
| UALink | UALink Consortium (initiated by AMD, AWS, Google, Meta, Microsoft, etc.) | 1.0 (2025-04): up to 1024 accelerators per Pod, 200 GT/s per channel, memory-semantic access; 2.0 (2026-04) adds In-Network Compute (all-reduce embedded in switch nodes), Chiplet 1.0 and UCIe 3.0 compatibility; over 85 members (Broadcom left the board in 2025-10, moving to OCP ESUN) |
| ESUN / SUE | OCP (led by Broadcom) | Ethernet-based Scale-up; Broadcom Tomahawk 6 (102 Tbps, 2048 ports, latency under 400 ns) is the chip foundation; an open route running in parallel with UALink |
Writing discipline: as of 2026-09 the native UALink switch chips were not yet in mass production, and AMD’s Helios racks carry the UALink protocol over an 800GbE physical layer (UALoE) as a transitional measure — it must be stated that "native silicon is not yet ready, bridged over Ethernet", and it is forbidden to write that "UALink is already deployed at scale commercially".
1.4.2. Domestic Interconnect Protocols
- LingQu (Unified Bus) 2.0: announced by Huawei at HC2025 to release the technical specification as an open standard; built-in reliability at every OSI layer, with 100 ns-class fault detection and protection switching on the optical path (A-level).
- EthLink (ByteDance): 1-4 Ethernet interfaces, up to 1024 GPUs per domain (B-level academic-survey framing).
- Eth-X (Tencent) and OISA (China Mobile): OISA advocates unified packets, multi-semantic fusion, and multi-level flow-control retransmission (B-level survey framing).
- Domestic IP progress: XinDong Technologies released a complete domestic UALink IP suite in 2026-05 (112G in production, 224G taped out); ODCC released a UALink test-and-verification service in 2026-04 (C-level; re-verify sources for item-by-item citation).
1.4.3. Scale-out, Optical Interconnect and Data Centers
- Scale-out dual stack: InfiniBand (Quantum-X800) and Ethernet (Spectrum-X800 and CPO switches, claiming 5x energy efficiency and 10x network resilience — vendor framing); ConnectX-9 SuperNIC at 1.6 Tb/s per GPU (GTC 2026).
- Optical interconnect evolution: NPO (optoelectronic chips moved down to the GPU board, meter-scale → centimeter-scale) is already deployed; CPO (co-packaged optics, centimeter-scale → millimeter-scale) is evolving; the Lightmatter LightSphere X supernode uses in-rack DAC plus cross-rack AEC (5-7 m) to reduce optical modules (B/C-level).
- Power distribution and oscillation governance: the GB300 NVL72 power shell has built-in electrolytic capacitors (about 65 J/GPU), and NVIDIA claims a peak grid-side power reduction of up to 30% at the same GPU output (A/B-level vendor framing); the "Implementation Plan for Facilitating the Synergistic Digital-Green Transformation (2026-2030)" quantitatively operationalizes for the first time 800V HVDC, over-100-kW single racks, liquid cooling plus waste-heat recovery, and "compute-power coordination" scheduling (B-level, relayed by Xinhua News Agency).
- Liquid cooling: cold-plate cooling has the highest maturity (roughly 65% share by industry estimate, C-level); liquid-cooling penetration is about 8% (2024) → about 35% by industry forecast (2026/2027, C-level); the Gui’an New Area spray-liquid-cooling pilot delivers +30% compute per rack, -40% energy, and a PUE persistently below 1.1 (B-level); Huawei Cloud Gui’an has an average annual PUE of 1.1, with iCooling AI tuning cutting energy by 8-15% (B-level).
- Energy overview: global data-center electricity use of 415 TWh (2024) → 945 TWh (2030 forecast, IEA, B-level); China’s compute-center electricity use of 196 billion kWh (2025) → 500 billion (2030E, CAICT, B-level).
Figure 3-1 | The three-way Scale-up interconnect-standards race and the domestic protocol landscape
Diagram: drawn from public information from the UALink Consortium, OCP, NVIDIA GTC 2026 and Huawei HC2025 (A/B-level) and the "Special Action Plan for Green and Low-Carbon Development of Data Centers" (A-level); the three states (in production / published / transitional) are this document’s analysis framing.
2. Glossary
| Term | English / Abbreviation | Definition |
|---|---|---|
| Scale-up | Scale-Up | Raising the interconnect density within a single domain (more cards in one memory-semantic domain) |
| Scale-out | Scale-Out | Increasing the cross-node / cross-rack network scale (cluster-level expansion) |
| Unified accelerator interconnect | UALink | Open accelerator interconnect standard: 1.0 supports 1024 accelerators/Pod, 2.0 adds In-Network Compute |
| Ethernet scale-up | ESUN (Ethernet for Scale-Up Networking) | The Ethernet-based Scale-up working-group route under OCP, led by Broadcom |
| SUE | Scale-Up Ethernet | Ethernet-based Scale-up protocol (Broadcom family), associated with ESUN |
| UALoE | UALink over Ethernet | Transitional scheme carrying the UALink protocol over an 800GbE physical layer (native switch chips not yet in mass production) |
| In-network compute | In-Network Compute | Technology that embeds collective-communication computation (e.g. all-reduce) in switch nodes |
| Remote direct memory access | RDMA | Kernel-bypassing remote memory read/write, the foundational capability of a Scale-out training network |
| RDMA over converged Ethernet | RoCE | Implements RDMA over Ethernet, requiring QoS/ECN/PFC cooperation |
| Lossless-Ethernet parameters | QoS/ECN/PFC | Explicit congestion notification and priority flow control, the three configuration planes of a lossless RoCE network |
| Direct-attach copper | DAC | Short-range (in-rack) copper interconnect with no optoelectronic components; lowest cost and power |
| Near-package optics | NPO | Optic chips moved down to the GPU board level, shortening the electrical channel |
| Co-packaged optics | CPO | Optical engines co-packaged with the switch chip, further improving energy efficiency and integration |
| Power usage effectiveness | PUE | Ratio of a data center’s total energy to its IT-equipment energy; a policy red-line metric |
| Compute-power coordination | Compute-Power Coordination | Joint optimization of computing scheduling and power scheduling, first quantitatively operationalized by the 2026 plan |
| High-voltage DC power | 800V HVDC | High-voltage DC power-supply architecture for data centers, a direction promoted by the 2026 plan |
| Power oscillation | Power Oscillation | Power fluctuation caused by the synchronized start/stop of training workloads, a power-supply challenge unique to AI data centers |
| Battery buffer | Battery Buffer | Using battery storage to smooth power fluctuations, e.g. Tesla Megapack (single unit 3.9 MWh framing) |
| Fault detection and protection switching | Fault Detection & Protection Switching | The 100 ns-class optical-path detection and switching capability of LingQu 2.0 |
| Memory-semantic access | Memory-Semantic Access | Using remote resources with memory-access semantics (load/store) rather than message semantics |
3. Case Studies
3.1. UALink 1.0 to 2.0: The Advance of the Open Interconnect Route and Its Transitional State
Evidence level: B (publication facts of official specifications are cross-confirmed by multiple sources; some domestic progress is C-level)
3.1.1. Background
The closed nature of the NVLink ecosystem (it does not open up the Switch chip and PHY, and entrants must connect through NVIDIA products to join the network) prompted AMD, AWS, Google, Meta, and Microsoft to found the UALink Consortium in 2024-10, seeking an open Scale-up standard.
3.1.2. Solution
- 1.0 (2025-04): UALink 200G — up to 1024 accelerators per Pod and 200 GT/s per channel, supporting memory-semantic access, memory sharing, and Switch-based networking; a 64B flit/640B packet structure with average latency hundreds of ns lower than Ethernet (B-level).
- 2.0 (2026-04, including three companion specifications: the 200G physical layer, the management framework, and the Chiplet interface): adds In-Network Compute (all-reduce embedded in switch nodes); Chiplet 1.0 and UCIe 3.0 compatibility; over 85 members.
- Transitional state: native UALink switch chips were not yet in mass production as of 2026-09; AMD’s Helios racks in 2026 H2 carry the UALink protocol over an 800GbE physical layer (UALoE).
- Ecosystem moves: Broadcom left the UALink board in 2025-10, moving to OCP ESUN; ODCC released a UALink test-and-verification service in 2026-04; XinDong Technologies released a complete domestic UALink IP suite in 2026-05 (112G in production, 224G taped out, C-level).
3.1.3. Effects
- Specification-level effects: the combination of a 1024-accelerator single domain plus In-Network Compute gives non-NVIDIA ecosystems an open option on a par with NVLink 5/6; In-Network Compute sinks collective communication down to the switch layer, directly improving the synchronization overhead in L3 orchestration.
- Real-world constraint: as of the information cutoff, "the open standard is published" and "native silicon is not yet in mass production" coexist — the true usability of the open route depends on the first chip deliveries in 2026-2027 (first 1.0 chips expected in 2026, B-level).
- Harness interpretation: In-Network Compute marks the L2 execution layer being pushed down toward the network layer — the execution primitive (all-reduce) moves from GPUs to switches, changing L2’s implementation boundary rather than the six-layer structure itself.
3.2. Huawei LingQu (UnifiedBus) 2.0 Goes Open: Standard Output of Domestic Interconnect
Evidence level: A (official release at Huawei Connect 2025)
3.2.1. Background
After Huawei stated at HC2025 that "China’s semiconductor-manufacturing processes will lag for a long time, and sustainable compute can only be built on actually available process nodes", interconnect became the strategic fulcrum of the domestic route: with single chips constrained, push interconnect to the extreme and open the interconnect specification itself as a standard, expanding the boundary of the ecosystem.
3.2.2. Solution
- Release the UnifiedBus 2.0 technical specification as an open standard.
- Built-in reliability: reliability mechanisms built into every OSI layer; 100 ns-class fault detection and protection switching on the optical path.
- Linkage with the supernode product line: LingQu is the UB network-specification foundation of the CloudMatrix 384 and the Atlas 950/960 SuperPoD (supernode implementation see 02-supernode.md).
3.2.3. Effects
- It upgrades domestic Scale-up from a "private implementation" to a "specification others can comply with", providing an interface foundation for third parties to join the Ascend ecosystem (with the open CANN, Mind suite, and openPangu open-sourcing).
- The 100 ns-class optical-path fault detection directly serves L5 observability (fault-detection latency metric) and L3 fault-tolerant orchestration (protection switching).
- The effect data are all release framing; ecosystem adoption (the number of third-party chips/systems following suit) had not received independent statistics as of the information cutoff and is marked as a gap.
3.3. xAI Colossus: The Data-Center Engineering of a 100,000-GPU Cluster
Evidence level: A/B (Supermicro official case-study PDF); part of Colossus 2 is B/C (encyclopedia and media framing)
3.3.1. Background
A 100,000-GPU-scale training cluster pushes interconnect and data-center constraints to the limit: network selection, power-supply fluctuation, and cooling density must all be solved simultaneously under time pressure for build completion. Colossus 1 is the highest-publicity extreme case (100,000 H100s, built in 122 days, first 10,000 GPUs live in 19 days; then expanded to 200,000 GPUs over a further 92 days, about 300 MW).
3.3.2. Solution
- Network: dropped InfiniBand in favor of Spectrum-X 400G Ethernet; SN5600 switches (51.2 Tbps); nine 400GbE ports per server (8 BlueField-3 + 1 ConnectX-7), 3.6 Tbps in total; case-state network throughput efficiency of 95% (A/B-level official case study).
- Power: Tesla Megapack battery buffers (3.9 MWh each) handle 10-20 MW-class sudden power changes in training workloads; about 300 MW site scale.
- Cooling: Supermicro 4U liquid-cooled racks (8 4U servers + CDU), 64 GPUs per rack; EB-scale all-NVMe storage (VAST Data → DDN EXAScaler/Infinia in two stages).
- Colossus 2 (B/C-level framing): announced operational on 2026-01-17, the world’s first gigawatt-class AI training cluster; about 555,000 GB200/GB300 (equivalent to 1.4 million H100s), starting at 1 GW with a 2 GW target; 41 natural-gas turbines plus close to 1.2 GWh of Megapack; PUE 1.12 (all-liquid-cooled framing, C-level); accompanied by environmental controversy (EPA violation ruling, NAACP lawsuit).
3.3.3. Effects
- Network-route effects: demonstrates that RoCE-family Ethernet can replace IB in 100,000-GPU training scenarios, but at the cost of the operational complexity of "aggressive QoS and flow-level telemetry" (C-level operations framing).
- Framing conflicts presented side by side (must be retained): the Colossus 1 MFU of about 11% (C-level) and the "network utilization 95%+" framing (C-level) coexist — the two definitions differ (training efficiency vs network efficiency) and are easily conflated, so each metric definition must be noted separately; multiple PUE values exist (1.05/1.1 for Colossus 1, R03 framing, and 1.12 for the Colossus 2 all-liquid-cooled framing), and any PUE citation must note the site and generation; cross-generation comparison is forbidden.
- Harness interpretation: Colossus demonstrates the physicalization of the L6 governance layer — power-fluctuation governance (Megapack) and the energy metric (PUE) are essentially translating L6’s "budgets and guardrails" into power and cooling equipment; the 95% network-utilization framing is the direct mapping of L5 observability onto the Scale-out network.
4. Practice Standards
Nature statement: the following is a draft proposal distilled from publicly disclosed industry practices, not the original text of official industry standards. This track is based primarily on policy documents (the PUE red line) and de-facto standards (IB/RoCE configuration practices, interconnect specifications), with no IEEE/ISO/JEDEC standards specific to clusters. The group-level higher-order specification is in AGENTS.md in the same directory.
4.1. AGENTS.md Specification
4.1.1. AGENTS.md (AI Infra Group · Cluster Interconnect and Data Center Track)
# AGENTS.md —— AI Infra 组 · 集群互联与机房方向
> 本文为基于行业实践提炼的建议稿,非官方行业标准原文。
> 上位规范:01-AI-Infra组/AGENTS.md(不可逆动作禁止清单、变更窗口、可观测要求)。
## 角色与边界
- 角色:集群互联与机房运行侧智能体,负责网络拓扑与配置分析、
Scale-up/Scale-out 选型比对、PUE 与能耗核验、功率波动治理方案评估。
- 不负责:网络全局配置(RoCE QoS/ECN/PFC、IB 子网管理器)的直接修改
(属禁止清单)、供配电工程设计、互联标准组织的立场声明。
- PUE 等政策条文只引用有 A 级原文的条目;仅媒体转述的政策标注 [待核实]。
## 环境假设
执行前必须显式声明:
- Scale-up 互联体系与代际(NVLink/UB/UALink/其他)与单域规模。
- Scale-out 网络形态(InfiniBand / RoCE / Spectrum-X / 其他)、交换层级、
跨机带宽、QoS/ECN/PFC 配置现状。
- 机房参数:单机柜功率上限、散热形态、设计 PUE 与实测 PUE、绿电占比。
- 训练负载画像:同步作业数、功率摆动幅度、储能/削峰设施。
- 未声明网络形态与机房参数时,禁止产出通信效率、PUE、扩展性结论。
## 上下文加载顺序(Context Budget)
1. 网络拓扑与配置现状(层级、带宽、QoS)。
2. 机房参数与能耗台账。
3. 目标负载的通信模式(集合通信规模、频率)。
4. 互联标准动态(区分已量产/已发布/过渡三态)。
## 工具契约
- 网络诊断通过受控工具(IB 探测、RoCE 遥测、nccl-tests)输出结构化结果。
- nccl-tests 输出必须含 busbw、buffer size、拓扑范围。
- 互联标准动态引用必须注明三态与来源级别。
## 任务执行流程(SOP)
- S1 需求解析:把负载需求换算为域内带宽、跨节点带宽、时延、功耗四类约束。
- S2 现状盘点:拓扑、QoS、机房参数、能耗台账。
- S3 候选生成:网络配置调整、拓扑调整、散热/供电改造候选方案。
- S4 受控验证:在维护分区或影子链路验证,记录 busbw 与时延。
- S5 合规核验:PUE 对照红线(1.25/1.2)、绿电占比对照要求。
- S6 证据打包:变更方案附回滚命令,进入审批队列。
## 验证与证据要求
- 通信结论附 nccl-tests 或遥测原始数据;网络利用率注明指标定义
(吞吐效率 vs 链路利用率,禁止混用)。
- PUE 结论注明站点、代际、测量周期;禁止跨代/跨站点混比。
- 互联标准结论注明三态;过渡方案(如 UALoE)不得写成规模商用。
## 失败与升级策略
- RoCE 网络出现不可归因的拥塞或丢包 → 停止自动调优,升级网络工程师。
- 功率摆动接近供电上限 → 触发削峰预案并上报,不得自行调整作业调度策略。
- 实测 PUE 逼近红线 → 输出能耗报告并启动散热优化评估。
## 安全与合规红线
- 网络全局配置修改属禁止清单动作,只产出待审批方案。
- 政策引用限于 A 级原文;转述条文标 [待核实]。
- 出口管制语境下保持中性表述。
- 禁止残留非标准占位符。
## 输出格式
- 首行:可判定结论 + 阻塞项。
- 拓扑与配置表:层级、带宽、QoS 参数、来源。
- 能耗报告:PUE、实测功耗、红线差值、绿电占比。
- 标准三态表:各路线的量产/发布/过渡状态与来源级别。
- 数值规范:参数带单位;范围用 ~ 连接;百分比数值与 % 之间不留空格。
## 评估与自检
- 网络利用率与 MFU 等指标定义是否分清?
- PUE 是否注明站点与代际?
- 互联标准是否标注三态?
- 是否产出过禁止清单内的直接修改动作?(必须为否)
- 是否残留非标准占位符? 4.2. SKILL.md Specification
4.2.1. SKILL.md (AI Infra Group · Cluster Network Diagnosis and Data-Center Energy Verification)
---
name: cluster-network-and-facility-audit
description: AI Infra 组集群互联与机房方向标准技能——网络配置诊断、互联
选型三态比对、PUE 与能耗红线核验。当需要诊断 RoCE/IB 网络问题、评估
Scale-up/Scale-out 路线、核验 PUE 政策符合性、评估功率波动治理时使用。
触发词:RoCE、InfiniBand、UALink、NVLink Fusion、PUE、液冷、功率摆动。
version: 1.0
created: 2026-09-12
---
# AI Infra 组 · 集群网络诊断与机房能耗核验
> 本文为基于行业实践提炼的建议稿,非官方行业标准原文。
> 上位规范:01-AI-Infra组/AGENTS.md。
## 适用场景
- 训练作业通信异常(掉速、超时、集合通信失败)的网络侧诊断。
- Scale-up 互联路线选型(NVLink / UALink / ESUN / 灵衢,三态比对)。
- Scale-out 网络选型(IB vs RoCE)与 QoS 配置评审。
- PUE 政策符合性核验(1.25/1.2 红线)与能耗预算评估。
- 功率摆动治理方案评估(削峰、储能缓冲、算电协同)。
## 前置条件
- 已声明网络形态、交换层级、QoS 现状与机房参数。
- nccl-tests、网络遥测工具可用;变更窗口机制可用。
- 能耗台账(PUE、分项功耗)可查询。
## 输入
| 输入项 | 说明 | 必需 |
|---|---|---|
| 网络拓扑与配置 | 层级、带宽、QoS/ECN/PFC、固件版本 | 是 |
| 负载通信画像 | 集合通信规模、频率、作业同步结构 | 是 |
| 机房参数 | 功率上限、散热形态、PUE 实测、绿电占比 | 是 |
| 故障现象 | 告警、日志路径、时间窗口 | 条件必需 |
| 互联标准动态 | 各路线三态与来源级别 | 否 |
## 输出
| 输出项 | 说明 | 必需 |
|---|---|---|
| 可判定结论 | 通过 / 不通过 / 部分通过 + 阻塞项 | 是 |
| 网络诊断报告 | busbw、时延、拥塞/丢包定位,附原始数据 | 条件必需 |
| 三态比对表 | 各互联路线量产/发布/过渡状态 | 条件必需 |
| 能耗报告 | PUE、实测功耗、红线差值、绿电占比 | 是 |
| 变更方案 | 配置调整建议 + 影响面 + 回滚命令 | 条件必需 |
| 证据表 | 工具、版本、命令、日志路径、数值、时间窗口 | 是 |
## 执行步骤
1. 环境校验:核对网络形态、QoS 现状、机房参数;确认不触及禁止清单。
2. 基线确认:nccl-tests 基准 busbw、能耗台账基线。
3. 问题定位(诊断场景):分层定位——链路(光模块/线缆)、交换
(拥塞/ECN)、配置(QoS/PFC)、软件(NCCL 超时)。
4. 选型比对(选型场景):按三态(已量产/已发布规范/规划过渡)比对
候选路线,标注来源级别。
5. 能耗核验:PUE 对照红线,分项功耗对照预算,绿电占比对照要求。
6. 受控验证:配置类建议先在维护分区验证并记录 busbw 差异。
7. 交付:诊断/比对/核验结论 + 变更方案(含回滚)+ 待人工确认项。
## 质量标准(DoD)
- 网络结论附 busbw 等原始数据;指标定义明确(吞吐效率 vs 链路利用率)。
- PUE 注明站点与代际,无跨代混比;红线核验有计算过程。
- 互联路线标注三态;过渡方案不写成规模商用。
- 变更方案含回滚命令;未执行禁止清单动作。
- 参照口径(不得直接套用):PUE 红线 1.25/1.2(A级政策);八大枢纽平均
PUE 约 1.3、最先进 1.04(B级,国家能源局);Tomahawk 6 = 102 Tbps /
2048 端口 / 小于 400ns(B级);灵衢 2.0 光路径 100ns 级检测(A级发布)。
## 常见失败与处理
| 失败模式 | 现象 | 处理 |
|---|---|---|
| 指标混用 | 把网络利用率 95% 当训练效率引用 | 分别注明指标定义与来源级别 |
| 过渡态误判 | 把 UALoE 桥接写成 UALink 规模商用 | 标注三态与过渡事实 |
| QoS 盲调 | 未测基线直接改 ECN/PFC | 先测基线,变更走审批窗口 |
| PUE 混比 | 跨站点/跨代比较 PUE | 注明站点与代际 |
| 红线误引 | 引用仅媒体转述的政策条文 | 标 [待核实] 或仅引 A 级原文 |
| 波动治理越权 | 自行调整全局作业调度应对功率摆动 | 升级人工,输出预案建议 |
## 示例
任务:某 8 千卡训练集群作业频繁出现集合通信超时,怀疑 RoCE 网络配置。
1. 环境校验:确认 Spectrum-X 类以太网、交换层级、QoS/ECN/PFC 现状、
固件版本;确认网络全局配置修改不在本轮直接执行。
2. 基线确认:nccl-tests 分层实测 busbw(节点内/跨机/跨柜),记录时间窗口。
3. 分层定位:跨柜 busbw 显著低于节点内,ECN 标记率在跨柜段异常升高。
4. 候选方案:ECN 阈值调整、PFC 策略收敛、故障光模块替换,各附影响面
与回滚命令。
5. 受控验证:维护分区验证 ECN 调整,busbw 恢复至基线的 97% 以上。
6. 能耗核验:同步核对 PUE 实测与红线差值(本例无变化)。
7. 交付:结论"部分通过,跨柜段需按方案调整 ECN 并更换 2 个光模块";
变更方案进入审批队列;证据表含命令、日志路径、时间窗口。 4.3. Landing Checklist
| No. | Check Item | Pass Criterion | Required |
|---|---|---|---|
| 1 | Environment declaration | Network form, QoS status, and data-center parameters declared | Yes |
| 2 | Baseline data | busbw baseline and energy ledger recorded | Yes |
| 3 | Metric definitions | Network utilization / throughput efficiency clearly defined and not conflated | Yes |
| 4 | Three-state labeling | Interconnect routes labeled in production/published/transitional | Yes |
| 5 | PUE framing | Site and generation noted; no cross-generation conflation | Yes |
| 6 | Red-line verification | Checked against the 1.25/1.2 red lines and green-power requirements | Yes |
| 7 | Policy source | Only A-level original text cited; relayed provisions marked [To be verified] | Yes |
| 8 | Change rollback | Network-configuration plan includes rollback commands and window | Conditionally required |
| 9 | Prohibited list | No direct modification of global network configuration | Yes |
| 10 | Oscillation governance | Power-oscillation contingency plan confirmed by a human | Conditionally required |
| 11 | Evidence completeness | Tools, commands, log paths, and time windows all in place | Yes |
| 12 | Placeholder cleanup | No non-standard placeholders such as XX, ___ | Yes |
5. Summary
The Cluster Interconnect and Data Center track covers the L2 "bus" and the L6 "physical projection" in the six-layer Harness model. Three conclusions:
- Interconnect standards have entered a long-term pattern of multiple routes coexisting. The three-way race of NVLink (closed + semi-open), UALink (open, In-Network Compute), and ESUN/SUE (Ethernet-based), plus the domestic EthLink/Eth-X/OISA/LingQu 2.0, leaves no single winner in the short term; the selection discipline is to assess against the three states of "in production / published specification / planned or transitional", and transitional schemes such as UALoE must not be written as commercially deployed at scale.
- Data-center constraints have been elevated from supporting to red-line. A PUE of 1.25/1.2 and an 80% green-power share are hard constraints with A-level policy original text; the 15 MW-class power oscillation of AI workloads makes power-supply governance (battery buffering, peak shaving, compute-power coordination) the second constraint after interconnect, and L3 computing scheduling must be co-designed with power scheduling.
- Metric discipline matters especially in this track. Network utilization versus training efficiency, PUE across different sites and generations, and deployment data at release versus dissemination framings are all high-conflation points in this track; AGENTS.md’s separation of metric definitions and site/generation labeling are mandatory requirements.
Information-Gap Statement
The following items have not been confirmed by A/B-level sources and are marked [To be verified] in the body text:
- This track found no IEEE/ISO/JEDEC standards dedicated to AI cluster interconnect and data centers; it is expressed in terms of policy documents + interconnect specifications + de-facto standards, and no standard numbers may be fabricated.
- The "Action Plan for Facilitating the Two-Way Empowerment of AI and Energy" (four ministries, 2026-04, "100% liquid cooling for newly built large AI data centers, retrofitting existing air-cooled facilities by 2028") has only been seen in media relays; the original text must be verified before citation.
- Colossus 1’s MFU of about 11% and a network utilization of 95%+ are both C-level and have different definitions; they have been separately noted.
- Colossus 2’s exact scale (about 555,000 cards), PUE 1.12, 41 gas turbines, and 1.2 GWh of Megapack are all encyclopedia/media framing (B/C-level).
- XinDong Technologies’ UALink IP and Biren’s cooperation in the ODCC interoperability test are C-level; sources must be re-checked for item-by-item citation.
- The EthLink/Eth-X/OISA parameters (1024-GPU domain, unified packets, etc.) come from relays in academic surveys (B-level); no first-hand specification text from the companies was obtained.
- Liquid-cooling penetration of 8% → 35% and a 65% cold-plate share are C-level industry observations, rewritten in the body text as "by industry estimate/forecast".
- The GB300 power-shell 65 J/GPU and grid-peak -30% figures are vendor framing (A/B-level) with no independent retesting.
- No public AGENTS.md/SKILL.md standard templates were found for this track; this document is a draft proposal.
6. References
- Special Action Plan for Green and Low-Carbon Development of Data Centers — NDRC and three other ministries, 2024-07. https://www.ndrc.gov.cn/
- Leading a New Paradigm of AI Infrastructure with Pioneering Supernode Interconnect Technology (LingQu 2.0 open) — Huawei, 2025-09-18. https://www.huawei.com/cn/news/2025/9/hc-xu-keynote-speech
- UALink 1.0/2.0 specification release — UALink Consortium, 2025-04 / 2026-04. https://ualinkconsortium.org/
- Multiple AI Scale-Up Options Emerge — SemiEngineering. https://semiengineering.com/multiple-ai-scale-up-options-emerge/
- Research on Key Technologies and Industry Development Trends of Supernodes (survey of interconnect protocols and domestic protocols) — Chinese Society of Engineering Thermophysics (repost), 2025-2026. https://www.zgcsswdx.cn/info/24724.html
- Success Story: xAI Colossus Cluster — Supermicro (written by ServeTheHome), 2024. https://www.supermicro.com/CaseStudies/Success_Story_xAI_Colossus_Cluster.pdf
- Energy and AI — International Energy Agency (IEA), 2025-04. https://www.iea.org/reports/energy-and-ai
- Implementation Plan for Facilitating the Synergistic Digital-Green Transformation (2026-2030) — Office of the Central Cyberspace Affairs Commission and six other departments, 2026-09 (relayed by Xinhua News Agency). https://www.gov.cn/
- NVIDIA Spectrum-X / ConnectX-9 related releases (GTC 2026) — NVIDIA. https://www.nvidia.com/en-us/networking/
- The Llama 3 Herd of Models (background on power fluctuation and training workloads) — Meta, 2024. https://arxiv.org/abs/2407.21783