NVIDIA: GB200 NVL72 Supernode and CUDA Full-Stack Ecosystem

1. Introduction

1.1. Vendor Positioning

NVIDIA is a global accelerated computing platform vendor whose tightly integrated "GPU + networking + full software stack" ecosystem constitutes the de facto standard of the AI compute market. Among the ten platforms in this group, NVIDIA is the only vendor that simultaneously satisfies the following three conditions:

1. Fully public data scale: as a US-listed company, its financial reports (SEC filings) disclose data center revenue, networking revenue, and the major-customer structure quarterly — the highest-credibility data source in this group (A-grade);
2. Third-party verifiable benchmark results: MLPerf Training / Inference submission results are published uniformly by MLCommons, and NVIDIA is the vendor with the most frequent and most complete submissions;
3. Software stack spanning all six Harness layers vertically: from KV Cache management (L1) to enterprise-grade support services (L6), every layer has a corresponding first-class product.

This document serves as the benchmark piece of the 06-AI-Infra group; the comparison coordinate systems of the other nine documents all reference this one.

1.2. Basic Information Card

1.3. Financial and Business Scale

All of NVIDIA's financial data comes from SEC filings and the official CFO Commentary (A-grade); the key figures are as follows:

Two structural observations:

1. Networking is the second growth curve of the supernode era: the USD 7.3 billion, +98% growth directly corresponds to the ramp of NVLink / GB200 / GB300, XDR InfiniBand and Spectrum-X Ethernet — supernodes turn "buying GPUs" into "buying rack-scale systems plus interconnects".
2. China business is rigidly constrained by export controls: the FY26 earnings Outlook explicitly "does not include China H20 shipments", and no official details exist for a China-specific route (see information-gap statement for gaps).

1.4. Position in the AI Harness Ecosystem

Under the six-layer model in the project parameter card, NVIDIA's role is to provide the underlying compute and system-level runtime for the Harness six layers: it does not produce agent frameworks (L3 orchestration relies mainly on upper-layer partners), but through Dynamo, NIM, and TensorRT-LLM it has made L1 (KV Cache), L2 (standard containerization tooling), and L5 (cluster observability) into "inference-side Harness foundation pieces". In short: NVIDIA is the foundation supplier of the inference-side Harness, not the Harness itself.

2. Glossary of Terms

3. Feature Description and Product Lines

3.1. Hardware Product Lines

NVIDIA's product lines targeting AI data centers consist of three parts:

1. GPUs and superchips: Hopper (H100/H200) → Blackwell (B200/GB200) → Blackwell Ultra (GB300) → Vera Rubin (2026—2027 roadmap);
2. CPUs: Grace (Arm architecture), tightly coupled to GPUs in GB200 in a "1 Grace + 2 Blackwell" superchip form;
3. Networking: NVLink/NVLink Switch (within rack), InfiniBand XDR and Spectrum-X Ethernet (between racks); Q2 FY26 networking revenue USD 7.3 billion (+98%).

Product cadence (per the official roadmap): GB300 NVL72 has been shipping to customers since 2025-06; Vera Rubin NVL144 targets Agentic AI architectures, adds storage accelerators and the Vera CPU, and is planned to land in 2026—2027.

3.2. Supernode Products: GB200 / GB300 NVL72

The GB200 NVL72 is the benchmark product of the supernode form factor; key specifications (official claims, cross-checked via Baidu Baike):

In its debut at MLPerf Inference v5.1, the GB300 NVL72 (Blackwell Ultra) refreshed every new benchmark: single-GPU DeepSeek-R1 offline throughput is 45% higher than the GB200 NVL72 and ~5x over Hopper (A-grade, MLCommons results).

3.3. Software Stack and Inference Services

1. TensorRT-LLM: official LLM inference optimization library; in MLPerf v5.0, FP4 precision was delivered via TensorRT-LLM + TensorRT Model Optimizer, achieving 2x the throughput of FP8 while meeting precision thresholds; 2025-09 v1.0 moved to a PyTorch-first architecture and provides the trtllm-serve OpenAI-compatible server (see 09-inference-serving.md; not elaborated here);
2. NIM microservices: model containerization + OpenAI-compatible API + LoRA hot loading (loads fine-tuned adapters without restarting); request-level structured JSON logs (request ID / token count / latency / inference tier), forwardable to Elasticsearch / Splunk;
3. Dynamo: distributed inference framework (see Section 5.4);
4. NVIDIA Mission Control: cluster management and observability platform;
5. NVIDIA AI Enterprise: enterprise-grade security, support, and stable-version subscription for the components above; NVIDIA's vehicle for software monetization.

4. Platform Architecture

4.1. Four-Layer Stack: Chip—Machine—Cluster—Framework

Figure 1-1 | NVIDIA full-stack architecture: from the GB200 NVL72 supernode to the AI software ecosystem

Data sources: NVIDIA GTC 2024 official materials, Q2 FY26 CFO Commentary (SEC), and NVIDIA's official roadmap; the diagram is based on the analysis in this document.

4.2. GB200 NVL72 Supernode Architecture

The essence of a supernode is presenting "a collection of 72 GPUs" to software as "a single compute unit with 130 TB/s of interconnect bandwidth". This yields three architectural consequences:

1. Memory pooling: the 72 GPUs in the NVLink domain can share 30 TB of fast memory, allowing very large KV Caches to reside (directly relevant to L1; see Section 5.2);
2. System-level delivery: liquid cooling, power delivery, and interconnect become an integration capability barrier for system vendors and cloud providers — the early rack overheating problem (see Section 6.4) stems from exactly this;
3. Internalization of networking revenue: NVLink Switch, InfiniBand, and Spectrum-X ramp with the supernode, constituting the source of the 98% growth in networking revenue.

5. Harness Design

5.1. Six-Layer Capability Overview

NVIDIA does not directly provide an agent Harness, but its software stack covers all the underlying primitives of an inference-side Harness. The table below is the six-layer mapping (rating basis: depth of support for upper-layer agent workloads):

Structural characteristics: L1/L2/L5 are clearly strong layers, while L3 is strong in "inference orchestration" but blank in "task orchestration" — agent capabilities such as Plan-and-Execute and sub-agent dispatch depend entirely on upper-layer frameworks (LangGraph, Claude Agent SDK, etc.), which NVIDIA deliberately does not cross into.

5.2. L1 Context Engineering Layer

1. Very large KV Cache residency: the 72 GPUs in the NVLink domain share 30 TB of fast memory, the physical precondition for ultra-long contexts and large-scale concurrent sessions;
2. NVL72-aware sharding: TensorRT-LLM is aware of NVLink domain topology to do tensor/pipeline sharding, reducing cross-domain communication;
3. Dynamo distributed KV Cache Manager: supports offloading KV to CPU, local disk, and object storage, extending KV Cache from "a first-class citizen in GPU memory" to "a manageable asset across memory tiers" (A-grade, NVIDIA Dynamo official blog 2025-03).

5.3. L2 Tools and Execution Layer

1. NIM microservices: package models into standard containers exposing an OpenAI-compatible API — this lets any agent framework following the OpenAI protocol integrate without modification;
2. NIM Agent Blueprints (later renamed NVIDIA Blueprints): package "multiple NIMs + orchestration logic + example code" into enterprise application workflows; typical blueprints include digital human customer service, virtual drug screening, and multimodal PDF RAG, published on a monthly cadence since 2024-08.

5.4. L3 Orchestration and Control Layer

Dynamo's four major components constitute the inference-side orchestration primitives (A-grade, GTC 2025 press release):

1. GPU Planner: dynamically adds or removes GPUs based on load to deliver SLA-driven autoscaling;
2. Smart Router: LLM-aware routing that preferentially reuses existing KV Cache and avoids recomputation;
3. Low-latency communication library: an inference-specific communication path;
4. Memory Manager: KV Cache offload and restore.

Official effect claims: DeepSeek-R1 671B request processing throughput on Blackwell improved by up to 30x; Cohere and Together AI announced integration. It should be emphasized: what Dynamo orchestrates is inference requests and GPU resources, not agent tasks — this is the essential difference between it and LangGraph-class frameworks.

5.5. L4 Memory and State Layer

1. Persistence and reuse of KV Cache across memory tiers (HBM → CPU → storage);
2. NIM's LoRA hot loading: load fine-tuned adapters without restarting the service; together with NeMo fine-tuning and AI Foundry version management, it forms a closed "fine-tune—deploy—rollback" state management loop.

5.6. L5 Evaluation and Observability Layer

1. Mission Control: cluster-level management and observability; CoreWeave uses it as its AI cloud operations standard;
2. Dynamo runtime observability: SLA autoscaling, real-time observability, fault tolerance;
3. NIM request-level logs: structured JSON (request ID / token count / latency / inference tier), integrable with Elasticsearch / Splunk — this provides request-granularity raw data for the L5 (trajectory observability) of upper-layer agents.

5.7. L6 Governance and Security Layer

1. NVIDIA AI Enterprise: provides enterprise-grade security, CVE response, and stable-version channels for Dynamo/NIM;
2. Exemplar Cloud certification system: provides a unified benchmark validation for cloud providers (inference / fine-tuning / pretraining recipe types), externalizing "governance" into auditable certification; CoreWeave is the first vendor with dual certification (training + inference).

6. Real-World Cases

6.1. MLPerf Benchmark Results (Third-Party Verifiable)

NVIDIA is the vendor with the most complete submissions within the MLPerf ecosystem; the results below all come from official MLCommons results (A-grade):

Ecosystem-breadth corroboration: in MLPerf Inference v5.0, 15 partners submitted results on the NVIDIA platform (ASUS, Cisco, CoreWeave, Dell, Fujitsu, Google Cloud, HPE, Lambda, Lenovo, OCI, QCT, Supermicro, SMC, VMware, and others) — the benchmark ecosystem itself is the NVIDIA ecosystem.

6.2. Cloud Providers and Hyperscale Customers

6.3. Conflicting Vendor Performance Claims, Shown Side by Side

Comparison of GB200 vs. AMD MI355X has two sets of vendor claims; this group presents them side by side per discipline without choosing one (see 02-amd.md and the group README appendix):

The two claims differ in test model, precision, and system configuration, so they cannot be directly compared; when citing either, one must note "vendor-commissioned/self-tested".

6.4. Known Issues

1. Early rack overheating: early GB200 NVL72 deployments experienced rack overheating, leading to design revisions and delivery delays (industry media reports 2024-11, B/C grade);
2. Tooling feedback: users reported insufficient NVLink copper-backplane diagnostic/debugging tools (same source);
3. China business uncertainty: the FY26 Outlook does not include China H20 shipments, and there are no official details of a China-specific route.

7. Summary

Advantages:

1. Full-stack closure: from FP4 operators to Exemplar Cloud certification, all six layers have a corresponding product, and MLPerf results are third-party verifiable (unique in this group);
2. Supernode generation leadership: NVL72/GB300 already shipping at scale, a clear Vera Rubin roadmap, and networking revenue forming a second growth curve;
3. Ecosystem lock-in: the CUDA + TensorRT-LLM + NIM combination makes upper-layer agent frameworks default to NVIDIA as their first backend.

Disadvantages:

1. China business is rigidly constrained by export controls, with no official route for H20 going forward;
2. Deep binding of proprietary interconnect (NVLink) and software stack creates vendor lock-in, and every hyperscaler hedges with its own chips (see documents 06/07);
3. System-level delivery is highly complex, and the early overheating and diagnostic-tooling issues exposed the operational cost of the supernode form factor.

Applicable boundary: the first choice for training/inference scenarios pursuing extreme performance and ecosystem completeness with no hard domestic-production constraint; scenarios with strong domestic-production and compliance constraints should turn to the domestic solutions in documents 03–08 of this group.

Selection recommendation: for training and large-scale inference, prioritize the NVL72 form factor over piling up single GPUs — the "9x NVLink domain" breakdown in MLPerf shows that supernode gains come mainly from interconnect rather than single GPUs; also, fold the NVIDIA AI Enterprise subscription and enterprise support for Dynamo/NIM into TCO calculations.

Information Gap Statement

1. No official documents exist for GB200/GB300 rack unit pricing or Vera Rubin NVL144 pricing (USD 8.8 million/rack is a media citation, not adopted);
2. No official details exist for NVIDIA China's (post-H20, compliant cards) 2026 shipment strategy;
3. No official statistics exist for the number of NIM enterprise customers (only case-name mentions: Amazon, Microsoft, Oracle, DocuSign, Perplexity, Snap, etc.).

8. References

1. NVIDIA Q2 FY2026 CFO Commentary — NVIDIA Investor Relations, 2025. https://investor.nvidia.com/files/doc_financials/2026/Q226/Q2FY26-CFO-Commentary.pdf
2. NVIDIA CFO Commentary Q1 FY2026 — SEC EDGAR, 2025. https://www.sec.gov/Archives/edgar/data/1045810/000104581025000115/q1fy26cfocommentary.htm
3. NVIDIA Blackwell Delivers Massive Performance Leaps in MLPerf Inference v5.0 — NVIDIA Technical Blog, 2025. https://developer.nvidia.com/blog/nvidia-blackwell-delivers-massive-performance-leaps-in-mlperf-inference-v5-0/
4. NVIDIA Blackwell Ultra Sets New Inference Records in MLPerf Debut — NVIDIA Technical Blog, 2025. https://developer.nvidia.com/blog/nvidia-blackwell-ultra-sets-new-inference-records-in-mlperf-debut/
5. Blackwell Takes Pole Position in Latest MLPerf Inference Results — NVIDIA Blog, 2025. https://blogs.nvidia.com/blog/blackwell-mlperf-inference%20
6. CoreWeave, NVIDIA, and IBM Set MLPerf Record with Largest NVIDIA GB200 Blackwell Cluster — CoreWeave, 2025. https://coreweave.com/blog/coreweave-nvidia-and-ibm-set-mlperf-record-with-largest-nvidia-gb200-blackwell-cluster-achieving-over-2x-faster-training
7. CoreWeave Becomes One of the First Cloud Providers to Achieve NVIDIA Exemplar Cloud Validation (Inference) — CoreWeave, 2025. https://www.coreweave.com/blog/coreweave-becomes-one-of-the-first-cloud-providers-to-achieve-nvidia-exemplar-cloud-validation-for-inference-on-nvidia-gb200-nvl72
8. Introducing NVIDIA Dynamo: A Low-Latency Distributed Inference Framework — NVIDIA Developer, 2025. https://developer.nvidia.com/zh-cn/blog/introducing-nvidia-dynamo-a-low-latency-distributed-inference-framework-for-scaling-reasoning-ai-models/
9. NVIDIA Dynamo Open-Source Library Accelerates and Scales AI Reasoning Models — NVIDIA Newsroom, 2025-03. https://nvidianews.nvidia.com/news/nvidia-dynamo-open-source-library-accelerates-and-scales-ai-reasoning-models
10. NVIDIA GB200 NVL72 — Baidu Baike (cross-reference), 2024—2026. https://baike.baidu.com/item/NVIDIA%20GB200%20NVL72/64300681
11. NVIDIA GB200 NVL72 / NVIDIA NIM entries — AI Wiki, 2026. https://aiwiki.ai/wiki/nvidia_gb200
12. Nvidia Blackwell Statistics 2026 — Axis Intelligence (third-party statistical compilation; third-party data needs verification), 2026. https://axis-intelligence.com/nvidia-blackwell-statistics