Inference Frameworks & Serving

1. Introduction

1.1. Background

The inference-framework-and-serving layer is the layer closest to the business in AI Infra: training is measured by MFU and effective training time, while inference is measured by TTFT (time to first token), TPOT (time per output token), and cost per token. Three main threads ran in this direction during 2025-2026:

First, KV Cache management has become a competitive focus. vLLM's PagedAttention (KV paging) and SGLang's RadixAttention (radix-tree KV reuse) represent the two routes of "memory paging" and "prefix reuse", respectively, directly mapping to L1 context engineering of the Harness six-layer model.

Second, Prefill/Decode disaggregation has become the default architecture for large-model serving. The two phases have completely different resource profiles (Prefill is compute-intensive, Decode is bandwidth-intensive), and once deployed separately each scales independently. NVIDIA's official form is Dynamo disaggregated serving orchestration (see 04-hardware-research-05-ai-infra.md), while Huawei's is the PDC disaggregation of CloudMatrix-Infer (arXiv 2506.12708).

Third, the domestic inference stack has gone open source. MindIE Turbo was officially open-sourced on 2025-11-30 and integrated with vLLM-Ascend, forming the Ascend ecosystem combination of "community framework + acceleration plugin + enterprise management service".

1.2. Scope & Definitions

The inference-framework-and-serving direction covers, on the runtime side:

Boundary note: The complete parameters of NVIDIA Dynamo disaggregated orchestration and NVFP4 have been captured on the design side (see 02-industry-empowerment/04-hardware-research/05-ai-infra.md); this document cross-references them without repeating. For training frameworks, see 04-training-framework.md.

1.3. Position in the AI Harness Stack

The inference framework is the most direct counterpart of the Harness six-layer model, and a layer-by-layer mapping can be established:

Bottleneck-layer assessment: This direction spans all six layers, but the center of gravity is L1 (KV management determines cost) and L5 (online SLO). KV Cache memory usage grows linearly with context length and concurrency, making it the primary driver of inference cost; therefore, the choice of KV management technology (paging/reuse/pooling) has a greater impact on cost than the engine's own operator optimization.

1.4. Current State

1.4.1. Technology Labels & Selection Verdicts for the Four Engines

Third-party benchmark figures (GPT-OSS-120B/H100, grade C): at high concurrency (100), vLLM has the highest throughput at 4,741 tokens/s; at medium concurrency (50), SGLang at 3,108 tokens/s; for single requests, TensorRT-LLM is best; on B200, TensorRT-LLM leads across all metrics (the result of deep binding to new hardware). Selection verdict: there is no all-scenario winner; select according to "concurrency pattern × hardware generation" — this conclusion is already written into this direction's checklist.

1.4.2. Architecture & Quantization

• PD disaggregation: Prefill and Decode are deployed and scaled independently; on the Huawei side this extends to the PDC three-subsystem (Prefill/Decode/Caching) disaggregation.
• Quantization: The INT8 route (adaptive scale, outlier suppression, efficient GEMM, pruning, error compensation) approaches FP8 benefits on non-FP8 hardware; W8A8 quantized attention (MindIE Turbo); NVFP4 on the NVIDIA side (see design-side documents).
• Scale validation: SiliconFlow, based on the CloudMatrix architecture "TengYun", achieved single-card DeepSeek-R1 inference at 1,920 tokens/s, matching NVIDIA deployments (press-conference figures, grade-B relay).

Figure 5-1 | PDC disaggregation: three-subsystem separation architecture of Prefill / Decode / Caching

Schematic: drawn based on arXiv 2506.12708 (Huawei/SiliconFlow, grade A); the performance figures are that paper's measured numbers for DeepSeek-R1.

2. Glossary

3. Case Studies

3.1. CloudMatrix-Infer: A Full MoE Serving Stack Outside the NVIDIA Ecosystem

Evidence level: A (official paper, arXiv 2506.12708)

3.1.1. Background

Ultra-large MoE models of the DeepSeek-R1 type (320 activated experts) pose three challenges for inference systems: communication storms in expert routing, memory pressure from the KV Cache, and maintaining precision on hardware that does not support FP8. This paper is the industry's first public disclosure of the full details of a multi-thousand-card (10,000-card class) MoE serving stack outside the NVIDIA ecosystem.

3.1.2. Approach

• PDC disaggregation: The Prefill, Decode, and Caching subsystems scale independently; a peer-to-peer serving architecture in which NPUs access the KV Cache and weights through UB directly to the DRAM pool, differing from the traditional "KV cache centric" architecture.
• EP320: DeepSeek-R1's 320 experts are mapped 1:1 to 160 cards/320 dies, eliminating load imbalance in expert routing.
• INT8 quantization: Adaptive scale, outlier suppression, efficient GEMM, pruning, and error compensation make up for the hardware's lack of FP8 support.
• Micro-batch pipeline + native operators: Pipeline parallelism across subsystems with operator-level fused optimization.

3.1.3. Results

• Prefill 6,688 tokens/s/NPU (4.45 tokens/s/TFLOPS), versus 3.75 for SGLang@H100 Perfect EPLB and 3.96 for DeepSeek@H800 Profile.
• Decode 1,943 tokens/s/NPU (1.29 tokens/s/TFLOPS).
• Output token latency below 50 ms; maintains 538 tokens/s while TPOT is below 15 ms.
• Precision: after INT8 quantization, 16 benchmarks are on par with the official DeepSeek API.
• Harness reading: This case verifies the mapping table in section 1.3 layer by layer — PDC and EP320 are L3 orchestration, direct UB access to the DRAM pool is L1/L4 KV management, "precision on par across 16 benchmarks" is L5 evaluation, and quantization and routing strategies together lower the cost per token.

3.2. MindIE & vLLM-Ascend: Open Source & Enterprise Serving Stack for Ascend

Evidence level: A/B (official documentation and release facts); comparison figures are grade-C vendor figures

3.2.1. Background

The Ascend ecosystem needs an inference stack that is both compatible with the community ecosystem (vLLM) and offers enterprise-grade capabilities. The open-sourcing of MindIE Turbo and the maturity of vLLM-Ascend have formed a three-layer combination of "community framework + acceleration plugin + management service".

3.2.2. Approach

• MindIE Turbo (Ascend LLM inference acceleration plugin library, officially open-sourced on 2025-11-30): W8A8/W8A8QuantAttention quantization (MindStudio output can be enabled in vLLM with one click); automatic operator patching; integrates with vLLM-Ascend without changing business code.
• vLLM-Ascend: The adaptation backend of community vLLM on Ascend, maintaining API compatibility.
• MindIE MS management service: Multi-tenancy and failure rescheduling (claims failure recovery reduced from minutes to roughly 30 seconds, grade-C vendor figure).
• Memory optimization (community-post figures, grade C): activation checkpointing + ZeRO-3-style distribution, claiming single-card maximum sequence 2048 → 4096 and 42% lower memory usage.

3.2.3. Results

• Third-party comparison figures (grade C): official measurements claim that on the same hardware, MindIE's peak throughput is 2.3× that of community vLLM and P99 latency drops by 62%; another account states "on Ascend, vLLM-Ascend is slightly better for single-card low-concurrency, while MindIE overtakes it for multi-card high-concurrency large-model/MoE scenarios".
• Writing discipline: All comparison figures above are vendor/third-party analysis figures (grade C) and may only be cited with the "official measured" label, not treated as independent conclusions; the only certain facts are the open-source date, the Apache 2.0 license, and one-click enablement in vLLM (grade A).
• Harness reading: MindIE MS's multi-tenancy and failure rescheduling are the direct implementation of L6 governance (quota, recovery) in the serving stack; "one-click enablement in vLLM" lowers the switching cost of the L2 runtime and is a key design of the ecosystem strategy.

3.3. Four-Framework Concurrency Selection Benchmark: No All-Scenario Winner

Evidence level: B/C (compilation of third-party benchmarks, deploybase.ai, etc.; grade C must be labeled)

3.3.1. Background

The most common question during team selection is "which engine is fastest". Controlled third-party benchmarks on GPT-OSS-120B provide an answer sliced by "concurrency pattern".

3.3.2. Approach

• Fix the hardware (H100) and model (GPT-OSS-120B); vary only the engine and concurrency (1/50/100), and uniformly measure throughput and latency; additionally collect TensorRT-LLM results on B200 for a hardware-generation comparison.

3.3.3. Results

• High concurrency (100): vLLM has the highest throughput at 4,741 tokens/s.
• Medium concurrency (50): SGLang is best at 3,108 tokens/s.
• Single request: TensorRT-LLM has the lowest latency (precompilation cost: ~2-4 hours for a 70B model).
• Hardware-generation impact: on B200, TensorRT-LLM leads across all metrics — the engine-hardware binding grows deeper with each generation.
• Harness reading: This is the standard paradigm of "controlled-variable benchmarking" at the evaluation layer L5 — the conclusion is not "who is fastest" but "who is fastest under what conditions"; this conclusion is turned into a prerequisite for concurrency profiling in the checklist.

4. Practice Standards

Nature Statement: The following is a draft proposal distilled from public industry practices (arXiv 2506.12708, MindIE official documentation, and third-party benchmarks), not the original text of an official industry standard. For the authoritative group-level specification, see AGENTS.md in the same directory; the following is an abridged version for the inference-frameworks-and-serving direction.

4.1. AGENTS.md Specification

4.1.1. AGENTS.md (AI Infra Group · Inference Frameworks & Serving Direction)

# AGENTS.md —— AI Infra 组 · 推理框架与 serving 方向

> 本文为基于行业实践提炼的建议稿，非官方行业标准原文。
> 上位规范：01-AI-Infra组/AGENTS.md（不可逆动作禁止清单、变更窗口、可观测要求）。

## 角色与边界
- 角色：推理 serving 智能体，负责引擎选型、PD 分离与批量参数设计、
  量化方案评估、KV Cache 管理策略、SLO 与成本核算。
- 不负责：多租户配额的最终分配（属 L6 治理人工决策）、
  训练侧问题（转交 04-training-framework）、硬件采购。
- 判定器为压测与线上指标：TTFT/TPOT/吞吐/每 token 成本必须来自
  受控压测或生产监控，禁止估算。

## 环境假设
执行前必须显式声明：
- 硬件代际与互联体系（决定引擎适配与量化格式可用集）。
- 引擎与版本（vLLM / SGLang / TensorRT-LLM / MindIE / vLLM-Ascend）、
  量化格式（INT8/W8A8/NVFP4 类）。
- 流量画像：并发分布、输入/输出长度分布、前缀重复率、峰值/均值比。
- SLO：TTFT/TPOT 目标、可用性要求、每 token 成本预算。
- 未声明流量画像与 SLO 时，禁止产出选型与容量结论。

## 上下文加载顺序（Context Budget）
1. 流量画像与 SLO。
2. 引擎版本与量化配置现状。
3. 压测基线与线上指标摘要。
4. 已知引擎问题与版本兼容清单。

## 工具契约
- 压测必须记录：并发数、输入/输出长度分布、量化格式、硬件代际、
  引擎版本——五要素缺一不可，否则结果不可比。
- 配置变更（批量、并行度、量化）可回滚，先在影子流量验证。
- PD 分离参数（Prefill/Decode 配比、路由策略）变更走变更窗口。

## 任务执行流程（SOP）
- S1 需求解析：流量画像 → 并发模式归类（单请求/中并发/高并发）。
- S2 基线压测：当前引擎在目标画像下的 TTFT/TPOT/吞吐/成本。
- S3 候选生成：引擎/量化/PD 配比候选，各附适用条件与预期收益。
- S4 受控压测：影子流量验证，五要素齐备记录。
- S5 精度核验：量化方案须过固定精度基准集（参照 16 项基准口径）。
- S6 成本核算：每 token 成本与预算差值。
- S7 证据打包与人工确认（生产切换走变更窗口）。

## 验证与证据要求
- 性能结论注明：并发模式、硬件代际、量化格式、引擎版本。
- 精度结论来自固定基准集，禁止只用困惑度单一指标。
- 厂商对比口径（如吞吐 2.3 倍）只能标"官方实测口径"引用。
- 每 token 成本给出计算过程（卡时成本 × 卡数 / 吞吐）。

## 失败与升级策略
- TPOT 超标 → 复核 PD 配比与量化格式，必要时扩容并记录。
- 量化后精度基准不达标 → 回退精度或换量化方案，禁止带病上线。
- 前缀命中率低于预期 → 检查 RadixAttention 类复用配置与路由亲和。
- 多租户争抢 → 升级治理人工，输出配额建议。

## 安全与合规红线
- 多租户严格隔离：禁止跨租户读取会话与 KV 数据。
- KV Cache 持久化数据按敏感等级管理。
- 生产切换与回滚走变更窗口，留审计记录。
- 禁止残留非标准占位符。

## 输出格式
- 首行：可判定结论 + 阻塞项。
- 压测报告：五要素 + TTFT/TPOT/吞吐/成本 + 与基线差值。
- 选型矩阵：引擎 × 并发模式 × 硬件代际。
- 待人工确认项：生产切换、配额调整。
- 数值规范：参数带单位；范围用 ~ 连接；百分比数值与 % 之间不留空格。

## 评估与自检
- 压测是否五要素齐备？
- 精度是否过了固定基准集？
- 厂商口径是否正确标注？
- 成本是否有计算过程？
- 是否残留非标准占位符？

4.2. SKILL.md Specification

4.2.1. SKILL.md (AI Infra Group · Inference Serving Selection & SLO Tuning)

---
name: inference-serving-selection-and-slo
description: AI Infra 组推理框架方向标准技能——serving 引擎选型、PD 分离
  参数设计、量化方案评估与 SLO/成本核算。当需要选型 vLLM/SGLang/
  TensorRT-LLM/MindIE、设计 PD 分离、评估 KV Cache 策略、压测 SLO 时使用。
  触发词：推理框架、PD 分离、KV Cache、TTFT、TPOT、量化、vLLM、MindIE。
version: 1.0
created: 2026-09-12
---

# AI Infra 组 · 推理 serving 选型与 SLO 调优

> 本文为基于行业实践提炼的建议稿，非官方行业标准原文。
> 上位规范：01-AI-Infra组/AGENTS.md。

## 适用场景
- serving 引擎选型（按并发模式 × 硬件代际矩阵）。
- PD 分离（或 PDC 三子系统）参数设计与路由策略。
- 量化方案评估（INT8/W8A8/NVFP4 类）与精度基准核验。
- KV Cache 管理策略（分页/前缀复用/池化/持久化）。
- 多租户与故障重调度能力评估（MindIE MS 类）。
- SLO 未达标或每 token 成本超预算的调优。

## 前置条件
- 已声明硬件代际、引擎版本、流量画像、SLO 与成本预算。
- 固定精度基准集可用；影子流量环境可用。
- 线上监控可采集 TTFT/TPOT/吞吐分布。

## 输入
| 输入项 | 说明 | 必需 |
|---|---|---|
| 流量画像 | 并发分布、输入/输出长度、前缀重复率 | 是 |
| SLO 与预算 | TTFT/TPOT 目标、可用性、每 token 成本上限 | 是 |
| 引擎现状 | 版本、量化配置、PD 配比 | 是 |
| 压测基线 | 当前画像下的四指标 | 是 |
| 精度基准集 | 固定评测集与达标线 | 是 |

## 输出
| 输出项 | 说明 | 必需 |
|---|---|---|
| 可判定结论 | 通过 / 不通过 / 部分通过 + 阻塞项 | 是 |
| 选型矩阵 | 引擎 × 并发模式 × 硬件代际，附口径 | 条件必需 |
| 压测报告 | 五要素 + TTFT/TPOT/吞吐/成本与差值 | 是 |
| 精度核验报告 | 固定基准集结果与达标判定 | 条件必需 |
| 变更方案 | PD 配比/量化/引擎切换 + 回滚命令 | 条件必需 |
| 待人工确认项 | 生产切换、配额调整 | 是 |

## 执行步骤
1. 需求解析：流量画像归类并发模式；明确 SLO 与成本红线。
2. 基线压测：五要素齐备记录当前引擎四指标。
3. 候选生成：引擎、量化、PD 配比候选，各附适用条件与预期收益
   （参照公开口径：vLLM 高并发、SGLang 中并发与前缀复用、
   TensorRT-LLM 单请求与最新硬件）。
4. 受控压测：影子流量执行，记录五要素与四指标。
5. 精度核验：量化候选过固定基准集（参照 CM384 案例 16 项基准口径）。
6. 成本核算：卡时成本 × 卡数 / 吞吐 → 每 token 成本对照预算。
7. 变更打包：生产切换方案 + 回滚命令，进入审批队列。
8. 交付：结论 + 报告 + 待确认项。

## 质量标准（DoD）
- 压测五要素齐备；性能数字注明并发模式/硬件代际/量化格式/引擎版本。
- 精度结论来自固定基准集，非单一困惑度。
- 厂商口径标"官方实测口径"，C 级数据标 [待核实]。
- 成本有计算过程。
- 参照口径（不得直接套用）：CM384 Prefill 6,688 / Decode 1,943
  tokens/s/NPU、输出延迟 <50 ms（A级论文实测）；MindIE Turbo 开源
  2025-11-30、Apache 2.0、vLLM 一键使能（A级）；MindIE 吞吐 2.3 倍/
  P99 -62% 为厂商口径（C 级，仅限标注引用）。

## 常见失败与处理
| 失败模式 | 现象 | 处理 |
|---|---|---|
| 压测不可比 | 缺并发分布或量化格式记录 | 补齐五要素重测 |
| 引擎错配 | 高并发场景用了单请求优化引擎 | 按并发模式矩阵重选 |
| 精度滑坡 | 量化后基准集不达标 | 回退精度或换方案，禁止上线 |
| KV 膨胀 | 长上下文导致显存溢出 | 分页/复用/池化策略复核 |
| 前缀失效 | 复用命中率低 | 检查路由亲和与前缀稳定性 |
| SLO 背离 | TPOT 或 TTFT 超标 | 调 PD 配比、扩容或降级并记录 |
| 跨租户越界 | 读取他租户会话 | 立即中断并上报 |

## 示例
任务：DeepSeek-R1 类 MoE 模型上线 serving，TTFT 目标 2 秒内、
每 token 成本预算给定，在昇腾超节点体系选型。
1. 需求解析：320 专家 → EP 并行；输入长、前缀重复率中高 → Prefill
   计算密集为主。
2. 基线压测：社区 vLLM-Ascend 默认配置记录四指标。
3. 候选：PDC 解耦 + EP320 + INT8（参照 arXiv 2506.12708 实证），
   对照 MindIE Turbo W8A8 一键使能路线。
4. 受控压测：影子流量验证 PD 配比，Prefill 侧吞吐对齐公开口径量级。
5. 精度核验：INT8 方案过 16 项基准集，与官方 API 口径一致。
6. 成本核算：每 token 成本低于预算，给出计算过程。
7. 交付：通过——采用 PDC + EP320 + INT8；压测报告、精度报告、
   回滚方案与待确认项齐备。
（示例流程为方法论演示；具体数值以实际压测为准。）

4.3. Deployment Checklist

5. Summary

The inference-framework-and-serving direction is the Infra direction with the most complete correspondence to the Harness six-layer model. Three conclusions:

1. KV Cache management is the first lever on cost. PagedAttention and RadixAttention compress KV overhead from the two directions of paging and prefix reuse respectively; PDC disaggregation further makes Caching an independent subsystem. The choice of KV strategy has a greater impact on cost per token than engine operator optimization.
2. The selection conclusion is "conditional" rather than "one unique optimum". Third-party benchmarks show: choose vLLM for high concurrency, SGLang for medium concurrency and prefix reuse, and TensorRT-LLM for single requests and the newest hardware; any "fastest engine" conclusion that detaches from the concurrency profile and hardware generation is untrustworthy.
3. The domestic stack follows an "open-source compatibility + enterprise capability" combination route. MindIE Turbo is open source with one-click enablement in vLLM (grade-A fact), combined with the multi-tenancy and failure rescheduling of the management service (L6), making the Ascend ecosystem's serving migration cost controllable; however, most of its comparison performance figures are vendor figures (grade C) and must be labeled when cited.

Information Gap Statement

The following items have not been confirmed by grade-A/B sources and are marked [To be verified] in the text:

1. The four-framework GPT-OSS-120B/H100 benchmark figures (4,741 / 3,108 tokens/s, etc.) are a compilation of third-party benchmarks (grade C); we recommend re-benchmarking yourself before key decisions.
2. MindIE's same-hardware peak-throughput 2.3× and P99 latency reduction of 62% are vendor figures (grade C) and may only be cited with the "official measured" label.
3. MindIE MS failure recovery of roughly 30 seconds, single-card sequence 2048 → 4096, and 42% lower memory usage are vendor/community-post figures (grade C).
4. SiliconFlow's single-card DeepSeek-R1 inference at 1,920 tokens/s is a press-conference figure (grade-B relay) with no independent reproduction.
5. MLPerf Inference v5.1 (Nebius GB200 submission) is the most recently confirmed figure on the inference side (grade-B relay); no confirmed results after v5.1.
6. No public AGENTS.md/SKILL.md standard template was found for this direction; this document is a draft proposal.

6. References

1. Serving Large Language Models on Huawei CloudMatrix384 — Huawei/SiliconFlow, arXiv 2506.12708, 2025-06. https://arxiv.org/pdf/2506.12708
2. MindIE Turbo Open-Source Repository — Ascend (gitcode/Ascend). https://gitcode.com/Ascend/MindIE-Turbo
3. vLLM Official Documentation and Repository — vLLM Project. https://github.com/vllm-project/vllm
4. SGLang Official Repository — SGLang Project. https://github.com/sgl-project/sglang
5. TensorRT-LLM — NVIDIA GitHub. https://github.com/NVIDIA/TensorRT-LLM
6. vLLM-Ascend — vLLM Ascend Backend. https://github.com/vllm-project/vllm-ascend
7. NVIDIA Dynamo (Disaggregated Serving Orchestration, cross-referenced on the design side) — NVIDIA. https://developer.nvidia.com/
8. Ascend Community MindIE Documentation — Huawei Ascend. https://www.hiascend.com/
9. DeepSeek-R1 Official API and Technical Report — DeepSeek. https://arxiv.org/abs/2501.12948
10. AI Infra (design side, Dynamo/NVFP4 cross-references) — 02-industry-empowerment/04-hardware-research/05-ai-infra.md, 2026-09-12.