Open-Source Inference Serving Ecosystem: vLLM / SGLang / TensorRT-LLM / MindIE

1. Introduction

1.1. Positioning of This Integration Article

This article integrates the open-source inference serving ecosystem into a single study, covering four representative engines/frameworks — vLLM, SGLang, TensorRT-LLM, and MindIE — and also incorporates NVIDIA Dynamo (the scheduling layer). The rationale for choosing these four: together they form the "de-facto standard layer" for LLM inference serving in 2025–2026 — almost all domestic (Chinese) compute platforms (Ascend, Moore Threads, etc.) use compatibility with or integration into this ecosystem as a selling point of their software stack.

This article is the most direct of the group in terms of six-layer mapping: the core innovations of inference engines (PagedAttention, RadixAttention, KV management, PD disaggregation) almost all fall within the Harness L1/L4 layers.

1.2. Basic Info Card of the Four Engines

Supplementary components: NVIDIA Dynamo (announced at GTC 2025-03, successor to Triton) is included in this article as a multi-engine scheduling layer; llm-d (out-of-the-box solution for vLLM + K8s, version 0.3) is included in the L3 discussion as a clusterized deployment form.

1.3. Division of Work with Article 01

This article overlaps with 01-nvidia.md on TensorRT-LLM and Dynamo. Division convention: Article 01 takes "NVIDIA full-stack competitiveness" as its narrative main line and only mentions these briefly; this article is the primary technical home of TensorRT-LLM and Dynamo, with Article 01's related content handled via cross-references (see 09-inference-serving.md). Likewise, the hardware-side background of MindIE belongs to Article 03 (Huawei Ascend), while this article provides the complete description of its engine capabilities.

1.4. Position in the AI Harness System

The inference serving ecosystem is the standard-component supply layer of Harness L1/L2: upper-layer agent frameworks (Claude Agent SDK, LangGraph, etc.) integrate with this ecosystem through the OpenAI-compatible API, and this ecosystem determines their KV reuse efficiency, first-token latency, and throughput cost. It can be summarized as: the inference engine to the agent Harness is like the database to the application server.

2. Glossary

3. Feature Description and Product Line

3.1. vLLM: PagedAttention and Foundation-Neutral Governance

1. Technology: PagedAttention raises memory utilization above 90% (naive implementation around 20%), SOSP 2023 paper; V1 architecture (2025-01 alpha); supports 100+ LLM architectures;
2. Hardware coverage: NVIDIA / AMD / TPU / AWS Neuron / Intel all-hardware backends + plug-in mechanism — Huawei Ascend vLLM-Ascend and IBM Spyre are both official plug-in forms;
3. Governance: joined the PyTorch Foundation in 2025-05 (one of the umbrella foundation's first hosted projects) — neutral governance is its strategic advantage over vendor engines; 46,500+ GitHub Stars, 1000+ contributors (repo figures, no third-party audit);
4. Community dynamics: Meta migrated its internal inference engine to vLLM in 2025 (per the maintainers' year-end summary, C grade); DeepSeek large-scale expert parallelism (DS-vLLM); the maintainers founded the company Inferact (a16z/lightspeed seed round over $150M).

3.2. SGLang: RadixAttention and Structured Output

1. Technology: RadixAttention prefix reuse; community-reported multi-turn throughput can exceed vLLM by 2–5× (community figures, need to be measured against load); FlashInfer operator library;
2. Structured output: regex-constrained decoding — a direct must-have for agent scenarios (tool call JSON, Schema constraints);
3. Deployment figures: officially claims over 300K GPUs deployed and trillions of tokens processed daily (official figures, no third-party audit).

3.3. TensorRT-LLM: v1.0 and PyTorch-first

1. v1.0 (2025-09): moved to a PyTorch-first architecture + trtllm-serve OpenAI-compatible server, greatly lowering the barrier to entry — previously its "complex compiled development experience" was a main community complaint;
2. Quantization depth: deepest FP8/FP4/NVFP4 quantization; the default backend for MLPerf NVIDIA submissions;
3. Performance: leading throughput on NVIDIA hardware (community benchmarks: H100 7B scenario in the 7000–10000 tok/s range, C grade); the cost is deep coupling to NVIDIA hardware.

3.4. NVIDIA Dynamo and MindIE

Dynamo (scheduling layer, see section 5.4): PD disaggregation scheduling, GPU Planner, KV-aware routing, KV offloading; compatible with TensorRT-LLM/vLLM/SGLang three backends; official figures show DeepSeek-R1 on Blackwell up to 30× request handling; the enterprise version ships with NIM/AI Enterprise.

MindIE (Ascend official engine): MindIE LLM / Turbo (quantization, pruning, operator fusion); underlying CANN; supports FA/PA (Paged Attention), W8A8/W8A16 quantization, MindIE Service, TGI compatibility; model list covers Baichuan/ChatGLM/Qwen/LLaMA/DeepSeek, etc. (1.0.RC2 docs 2025-03, latest 3.0.0 — new-feature list missing); integrates into the vLLM ecosystem via the vLLM-Ascend plug-in and contributes upstream features including Chunked Prefill, Prefix Caching, Speculative Decoding, Multi-LoRA Function Call.

4. Platform Architecture

4.1. PD Disaggregation Architecture

Figure 9-1 | PD disaggregation inference architecture: taking Dynamo orchestration + multi-engine backends as an example

Data sources: NVIDIA Dynamo official blog (2025-03), MLPerf Inference v5.1 (as relayed in NVIDIA technical blogs), Ascend community MindIE documentation; the architecture is a schematic diagram.

4.2. Multi-Engine, Multi-Hardware Backend Matrix

Matrix interpretation: vLLM's "plug-in mechanism + foundation neutrality" makes it the greatest common denominator across hardware ecosystems; TensorRT-LLM and MindIE are respectively the optimal performance solutions for NVIDIA/Ascend.

5. Harness Design

5.1. Overview of Six-Layer Capabilities

This article is the most direct of the group in terms of six-layer mapping (all facts are A grade):

5.2. L1 Context Engineering Layer

The core innovations of the three engines all land in the KV Cache management layer: PagedAttention (paged memory management), RadixAttention (prefix reuse), prefix caching and Chunked Prefill, and KV offloading — this layer is the main battleground of inference engine competition and the cost determinant of agent long-context workloads. For Harness, the prefix cache hit strategy directly determines the token cost curve for multi-turn tool-call scenarios.

5.3. L2 Tools and Execution Layer

The OpenAI-compatible API is already the unified interface contract of the four engines — the "model-agnosticism" of upper-layer agent frameworks is in fact built on this layer. Function Call support (the vLLM-Ascend Multi-LoRA Function Call feature contributing on the Ascend side) lets inference engines directly serve agent tool-call workloads.

5.4. L3 Orchestration and Control Layer

1. Dynamo's four components: GPU Planner (dynamically adds/removes GPUs), Smart Router (LLM-aware routing, avoiding KV recomputation), low-latency communication library, Memory Manager;
2. llm-d: open-source orchestration solution for vLLM + K8s (out-of-the-box in 0.3) — brings inference serving into the cloud-native orchestration system;
3. PD disaggregation orchestration: the scheduling of requests across the Prefill/Decode pools is the core problem of current inference orchestration.

5.5. L4 Memory and State Layer

Persistence and reuse of KV Cache across memory tiers (HBM → CPU → local disk → object storage) turns "session state" from a volatile GPU-memory asset into a manageable persistent asset — this is the deepest contribution of the inference side to Harness L4: an agent's session continuity can be achieved through KV reuse rather than full recomputation.

5.6. L5 Evaluation and Observability Layer

TTFT/TPOT/TPS form the standard metrics system (defined by the MLPerf Interactive scenario); NIM provides request-level structured JSON logs (request ID / token count / latency / inference profile) that can be forwarded to Elasticsearch/Splunk — supplying request-granularity raw data for upper-layer agent trajectory observation.

5.7. L6 Governance and Security Layer

Three governance models coexist: vLLM's foundation-neutral governance (PyTorch Foundation, avoiding single-vendor lock-in), NVIDIA AI Enterprise commercial support/security edition (enterprise-grade CVE response and stable release channel), and the MindIE commercial edition (in the Ascend compliance context). When selecting, governance model and performance requirements must be evaluated separately.

6. Case Studies

6.1. Ecosystem Adoption

6.2. Combined Form: Official Reference Architecture of Three-Engine Collaboration

Three-engine collaboration within the NVIDIA ecosystem is already the official reference architecture: the Exemplar Cloud inference benchmark designates a TensorRT-LLM or SGLang backend + Dynamo multi-node orchestration (A grade) — i.e., the combination pattern of "optimal-performance engine + flexible engine + unified scheduling layer," rather than a single engine doing everything. This is the current best-practice form for enterprise-grade inference deployment.

6.3. Selection Conclusions

Third-party comprehensive comparison (Alibaba Cloud developer community report, data as of 2025-10, B grade):

7. Summary

Strengths (ecosystem combined):

1. Highest density of L1/L4 innovation: KV Cache management is currently the most active innovation layer in AI Infra, and it spreads quickly in open-source form;
2. Interface standardization complete: the OpenAI-compatible API + MLPerf Interactive metrics system constitutes a replaceable de-facto standard;
3. Diverse governance models: foundation-neutral (vLLM) and vendor enterprise editions (TensorRT-LLM/MindIE) coexist, selectable by lock-in risk preference.

Weaknesses:

1. All scale data is official/repo-reported: "300K GPUs" and "46,500 stars" have no third-party audit;
2. High engine-selection complexity: the combination of three engines + a scheduling layer requires the team to have strong operations and tuning capabilities;
3. Insufficient transparency of domestic (Chinese) engines: MindIE 3.0.0's new-feature list is missing and the release cadence is opaque.

Applicability boundary: every team that self-hosts inference serving — this ecosystem is currently the only mature "multi-hardware backend + standard interface" option; for agent workloads (structured output, Function Call, multi-turn prefix reuse), SGLang and vLLM-Ascend provide the most direct support.

Selection recommendation: choose the engine by target hardware (NVIDIA → TensorRT-LLM, multi-backend → vLLM, agent-intensive → SGLang, Ascend → MindIE + vLLM-Ascend); use Dynamo/llm-d for unified scheduling when combining multiple engines; include TTFT/TPOT alongside model quality in the service SLA.

Information Gap Declaration

1. No independent statistics on the production cluster scale of vLLM/SGLang ("300K GPUs" and "46,500 stars" are official/repo-reported respectively, no third-party audit);
2. MindIE 3.0.0's new-feature list is missing (this round only confirmed the version exists);
3. Secondary engines such as PopS/TGI/LocalAI are out of scope of this article, only briefly mentioned in their ecosystem niche;
4. SGLang's "throughput exceeding vLLM by 2–5×" is community-reported and needs to be verified against actual load.

8. References

1. Comprehensive Comparison and Analysis Report of vLLM, SGLang and TensorRT-LLM (data as of 2025-10) — Alibaba Cloud Developer Community, 2025. https://developer.aliyun.com/article/1686693
2. PyTorch Foundation Welcomes vLLM as a Hosted Project — PyTorch Official Blog, 2025. https://pytorch.org/blog/pytorch-foundation-welcomes-vllm/
3. Introducing NVIDIA Dynamo — NVIDIA Developer, 2025. https://developer.nvidia.com/zh-cn/blog/introducing-nvidia-dynamo-a-low-latency-distributed-inference-framework-for-scaling-reasoning-ai-models/
4. MindIE Supported Model List (1.0.RC2) — Ascend Community, 2025-03. https://www.hiascend.com/document/detail/zh/mindie/10RC2/whatismindie/mindie_what_0002.html
5. Deep Insight: LLM Inference Capabilities and the MindIE Engine (vLLM-Ascend Ecosystem) — Volcano Engine ADG Community, 2025. https://adg.csdn.net/695251d05b9f5f31781b844f.html
6. Serving LLMs for Inference on Huawei Ascend NPU with MindIE and OpenAI-API Service Compatibility — CSDN Technical Practice, 2025-01. https://blog.csdn.net/kelxLZ/article/details/145228443
7. vLLM Official Documentation — vLLM Project, 2025. https://docs.vllm.ai
8. SGLang Official Documentation — SGLang Project, 2025. https://docs.sglang.ai
9. vLLM Core Maintainers' Year-End Summary (PyTorch Foundation governance / Meta migration / Inferact) — WeChat public account "开源大模型主理人", 2026 (community first-hand account, no stable URL for the original).
10. NVIDIA Blackwell Ultra MLPerf Debut (TensorRT-LLM/NVFP4 production figures) — NVIDIA, 2025. https://developer.nvidia.com/blog/nvidia-blackwell-ultra-sets-new-inference-records-in-mlperf-debut/