HPC · High-Performance Computing

1. Introduction

1.1 Background

High-performance computing (HPC) is the direction in the data science group with the heaviest engineering discipline and the least tolerance for error. Jobs run on shared leadership-class clusters, where resources are billed and quota-limited by core-hour; a single incorrectly submitted job can occupy hundreds of nodes for hours, and a failure caused by a mismatched compilation environment wastes not minutes but an entire queuing cycle.

The Exascale era has arrived. According to official TOP500 data, as of the 67th list released on 2026-06-23, five systems worldwide have sustained measured HPL performance above 1 EFlop/s, and for the first time Exascale systems appeared simultaneously across three continents — Asia, North America, and Europe (see Section 3.1). The larger and more heterogeneous the system, the more complex job scheduling, parallel strategy, and environment management become — which is precisely where agents can be most helpful, and also most likely to cause trouble.

A consensus design insight in the industry comes from the practice of Argonne National Laboratory: "The LLM never touches a scheduler" (LLM 从不接触调度器). The agent's responsibility is to translate scientific goals into a job graph and environment description, while placement, concurrency, and fault tolerance are left to mature orchestration systems (Slurm, Parsl). This is not conservatism, but the best evidence of the separation of responsibilities between the L2 tool layer and the L3 orchestration layer.

1.2 Definition

The AI Harness for the high-performance computing direction is an engineering carrier layer that, around job-script authoring, compilation environment management, parallel strategy design, job submission and monitoring, and result verification, provides agents with environment-context assembly, restricted job-submission tools, scheduler reuse, run-status memory, and physical-level verification.

It does not replace schedulers such as Slurm/PBS/LSF, nor the MPI/OpenMP parallel model, nor domain solvers. Its responsibility is the layer that "enables the agent to correctly understand and use this mature system."

Three points on its boundaries need to be clarified:

• HPC Harness is not a "robot that runs jobs automatically." Its deliverables are auditable job scripts, environment declarations, and job graphs, and submission actions are constrained by quotas and approvals.
• HPC Harness does not reinvent the orchestration layer. Slurm itself is a mature orchestration system (job array, job dependency, backfill); the value of Harness lies in translating natural-language goals into forms these engines can digest.
• HPC Harness does not make real-time scheduling decisions. Studies show that scheduling 100 jobs via a cloud API service takes 1–2 hours (see Section 3.3); real-time scheduling is infeasible given inference latency; the viable path is the separation of reasoning from constraint enforcement.

1.3 Position in the AI Harness System

图 1-1｜HPC 六层能力模型：核心在 L2 工具层与 L3 编排层

数据来源：基于本文分析绘制的示意图。

High-performance computing emphasizes the following across the six-layer capability model (this emphasis analysis is based on public case facts, not a standard):

The core is in L2 and L3.

L2 is core because HPC's tool contract is the most special: an agent should not directly call sbatch, but should produce a job description that a workflow engine can carry. Argonne's evidence (Section 3.2) shows that the correct form of an MCP tool is to emit Parsl applications, with Parsl responsible for placement, concurrency, and fault tolerance on the cluster — the separation of the tool layer from the orchestration layer here is not a design preference, but an architectural fact validated by large-scale operation.

L3 is core because the scheduler itself is HPC's orchestration layer. Harness should reuse rather than reinvent: translate natural-language goals into a job graph that Slurm can digest (job array + dependency), rather than inventing another scheduling loop on top of Slurm.

Bottleneck layer: the intersection of L2 and L5. An agent can write syntactically correct job scripts, but cannot write them correctly without environment context — a missing module, a compiler version mismatch, or a wrong GPU architecture will make a job fail instantly after hours of queuing. At the same time, physical-level verification is extremely expensive, so a full regression cannot be run on every change, and a layered evaluation must be designed.

1.4 Value and Bottlenecks

Value is reflected in three places:

• Reducing job authoring and debugging cost: leave the "write job script + trial-and-error compile environment" loop to the agent as pre-processing, so people can focus on parallel strategy and the scientific problem itself.
• Turning scheduling knowledge into executable constraints: partition selection, resource estimation, dependency relations, and checkpoint strategy — things that used to live in the memory of senior users — become checkpoints the agent must verify before every submission.
• Exploring new optimization space: ORNL research shows that agent orchestration can improve time-to-solution by up to 1.7× over evolutionary algorithms, while lowering computational cost (see Section 3.3).

Bottlenecks also number three:

• Inference latency: LLM scheduling has high computational overhead; scheduling 100 jobs via a cloud API takes 1–2 hours, limiting real-time deployment — exactly the problem Harness addresses with "the separation of reasoning from constraint enforcement."
• Environment drift: the combination space of the module environment and compiler/library versions is huge, and model training data cannot cover the real state of a specific cluster; environment declarations and short-run validation must serve as the safety net.
• Evaluation cost: a single simulation takes 1,600–4,400 s (per the publicly reported Argonne case), making high-frequency regression impossible; a layered evaluation must be used, with cheap proxy metrics first and expensive validation triggered on demand.

2. Glossary

3. Case Studies

3.1 Case 1: The Exascale Era — TOP500's 67th List and LineShine Takes the Top Spot

3.1.1 Background

The TOP500 list ranks the world's supercomputers by the HPL benchmark and is the most authoritative public source of facts for the HPC direction. On the 65th (2025-06-10) and 66th (2025-11-17, SC25) lists, El Capitan held the No. 1 spot, three systems from U.S. DOE laboratories successively reached Exascale, and JUPITER became Europe's first Exascale system on the 66th list.

The 67th list, released on 2026-06-23, brought structural changes. The following is the top ten (data source: TOP500 official site, grade A):

Basis-of-calculation note (must be stated explicitly): TOP500's official table records LineShine's core count as 13,789,440; meanwhile the Chinese report from the CAS e-Infrastructure website states "more than 2.45 million CPU cores deployed across the machine." These are two different bases of calculation (the TOP500 basis typically counts the processing units used by HPL, while the Chinese report's basis may count physical CPU packages). This document takes TOP500's official figures as authoritative, and presents both bases side by side without conversion.

3.1.2 Approach

The key facts about LineShine (灵晟, Chinese name per the CAS official site) are as follows:

• The first supercomputer in the world to break through 2 EFlop/s of sustained measured HPL performance, and also the fifth Exascale system worldwide to be built and put into service.
• Pure CPU architecture: the global first system to exceed 2 exaflops of sustained double-precision performance on HPL using only CPUs; its platform is "LingKun", its processors are the self-developed LX2 (304 cores, 1.55 GHz), its interconnect is the self-developed LingQi, and its operating system is Kylin (麒麟).
• Efficiency about 80% of peak (2.198 / 2.736); energy efficiency 52.07 GFlops/Watt (comparison: El Capitan's energy efficiency is 60.94 GFlops/Watt).
• HPCG No. 1: 22.00 PFlop/s — showing it also leads on the sparse-computation benchmark that is closer to real applications.
• HPL-MxP No. 4: 7.92 EFlop/s, only about 3.6× acceleration relative to HPL — a direct reflection of the pure CPU design (no dedicated low-precision accelerator).
• This is the first time a Chinese system has topped the TOP500 list since Sunway TaihuLight in 2017.

The implication for Harness design lies in the contrast of the ranking itself: among systems at the same Exascale level, LineShine chose a pure CPU route without a low-precision accelerator, and its HPL-MxP/HPL ratio is significantly lower than the common level of GPU systems. Architecture choice determines the performance profile — when generating job strategies for different systems, an agent must use the target system's real architectural profile as context, rather than applying generic heuristics.

3.1.3 Results

The 67th list confirmed three structural facts:

1. Exascale has become the norm: five systems exceed 1 EFlop/s on HPL, and for the first time Exascale systems appeared simultaneously across Asia, North America, and Europe.
2. Technical routes have diversified: pure CPU (LineShine), GPU acceleration (El Capitan / Frontier / Aurora), and Booster-tiering (JUPITER) coexist; there is no single optimal architecture.
3. Energy consumption has become a first-class constraint: power at the scale of 42,220 kW (LineShine) makes energy efficiency (GFlops/Watt) a metric on par with peak performance, raising the value of energy-efficiency lists such as Green500.

The direct corollary for HPC-direction Harness is: the target-system profile (architecture, interconnect, precision capability, energy efficiency) must become a fixed component of L1 context, and the agent's parallel and precision strategy suggestions must be premised on that profile.

3.2 Case 2: Argonne Agentic Material Screening — "The LLM Never Touches a Scheduler"

3.2.1 Background

Material screening is a typical batch task on HPC: running physics simulations on a huge set of candidate structures one by one and ranking them. According to industry media reports (supercomputing.news, grade B; the original arXiv preprint was submitted on 2026-04-09 and was not directly fetched), Argonne National Laboratory completed an agent-driven screening of metal-organic frameworks (MOFs) on the Aurora leadership-class system, targeting 5,591 MOF structures from the CoRE MOF 2025 database, for atmospheric water harvesting applications.

3.2.2 Approach

This work's architecture design is the most valuable reference for this direction:

• Layered agents: a planner agent (decomposes goals and dynamically derives an executor pool) + executor agents (run each single simulation) + a data-analyst agent (aggregates and ranks).
• Two MCP Servers: the Chemistry MCP (exposes simulation-launch tools) and the DataTool MCP (ranking).
• Key design: the MCP tools do not directly run simulations; instead they emit Parsl applications, with Parsl responsible for placement, concurrency, and fault tolerance on Aurora — "The LLM never touches a scheduler."
• Local inference: uses the open-weight gpt-oss-120b served through an ALCF local-inference endpoint, avoiding API cost and data-governance issues.
• Each GCMC (grand canonical Monte Carlo) job runs on a single tile of an Intel Data Center GPU Max 1550, using the GPU Monte Carlo code gRASPA.

This case explicitly demonstrates that this is a real data point of MCP stepping out of the IDE and becoming the tool-binding layer between LLMs and scientific-computing infrastructure.

3.2.3 Results

Per the above report (grade B; the figures should ideally be cross-checked against the arXiv original before citing):

• Output scale: completed 11,182 GCMC simulations; the largest production run used 256 Aurora nodes simultaneously.
• Scalability: weak scaling (fixed 9 MOFs per node, 1→256 nodes) stayed flat; strong scaling was near-linear from 8→32 nodes, with efficiency dropping to 64.9% at 256 nodes.
• Controllable orchestration overhead: agent orchestration overhead was 60–90 s per run, small relative to a single GCMC job's 1,600–4,400 s.
• 84% reliability: 21 of 25 scaling experiments succeeded; all 4 failures stemmed from tool-call arguments with malformed LLM output, not the orchestration layer.
• Scientific output: the top 20% of screened MOFs achieved a water working capacity of 7.06 mol/kg over the range of 298 K and 60%→10% relative humidity.

Three directly transferable engineering conclusions:

1. Parameter-format errors are the main source of agent failures, and they can be intercepted before invocation via schema validation of tool parameters — this should be a mandatory item of the L2 tool contract.
2. The ratio of orchestration overhead to single-job duration (60–90 s vs. 1,600–4,400 s) shows that the agent layer is feasible for long-duration scientific jobs, provided orchestration frequency matches job granularity.
3. Known limitations must be recorded honestly: the v1 preprint is not peer-reviewed; 256 nodes are far from filling Aurora's 10,624 nodes; and there is no head-to-head comparison with hand-written Parsl workflows. This case proves architectural feasibility, not overall superiority.

3.3 Case 3: The Possible and Impossible of LLM Job Scheduling — ORNL's Evidence and Its Limits

3.3.1 Background

"Can an LLM directly do job scheduling?" is the most natural idea for HPC intelligence, and also the direction easiest to get wrong. A research team at ORNL and other institutions (DOE grant DE-SC0024387) has provided the most complete empirical study to date: the paper on using LLM reasoning for multi-objective HPC job scheduling was published on arXiv (2506.02025) and indexed by the ACM Digital Library (grade A source).

3.3.2 Approach

The study's system design emphasizes the separation of reasoning from constraint enforcement:

• A ReAct-style (Reason + Act) scheduling agent: includes scratchpad memory to track the scheduling history, and refines decisions through natural-language feedback.
• Constraint-enforcement module: a deterministic module independent of the LLM guarantees the feasibility and safety of scheduling decisions. The paper states its conclusion explicitly: "The separation of reasoning from constraint enforcement ensures system reliability."
• Experimental setup: 7 real HPC workload scenarios (heterogeneous mixes, burst patterns, adversarial cases, etc.) with job sizes of 10–100; baseline comparisons are FCFS, SJF, and Google OR-Tools; two models are used: OpenAI O4-Mini and Anthropic Claude 3.7.
• Real-trace validation: uses the public job log of 100 jobs submitted in 2024-11 on Argonne's Polaris system (560 compute nodes, 512 GB memory per node).

3.3.3 Results

The study's conclusion is a set of paired positive and negative results that must be cited together:

• Positive: the LLM scheduler markedly improves waiting time and turnaround time (comparable to an idealized SJF), while resource utilization and throughput are no worse than baseline; no significant outliers in the negative metrics.
• Negative (key limitation): high computational overhead — scheduling 100 jobs with a current cloud API service takes 1–2 hours, limiting real-time deployment.

Continuing work in the same direction is ORNL's "Toward Agentic HPC" presented at ISC 2026 (2026-06-23, Hamburg, Germany): it proposes a framework with persistent model serving, scalable agent orchestration, and general-purpose interfaces for HPC applications and schedulers, explicitly handling batch scheduling, failures, and heterogeneous accelerators; on two applications — materials design and weather modeling — across multiple frontier LLMs, it improves time-to-solution by up to 1.7× over evolutionary algorithms while lowering computational cost (source: official ISC agenda, grade A/B).

The three conclusions of this case for Harness design:

1. The bottleneck of LLM scheduling is not "intelligence" but inference latency — so the correct posture for Harness is to place LLM reasoning at low-frequency decision points (job-graph design, strategy selection) and leave high-frequency decisions to deterministic mechanisms.
2. Constraint enforcement must be an independent deterministic module and must not rely on the model's self-constraint — consistent with the L2/L6 layered governance design.
3. Real traces are a necessary validation means: synthetic scenarios struggle to expose a scheduler's real behavior, so internal evaluation should be built on historical job logs.

4. Practice Standards

4.1 AGENTS.md Specification

Standard source statement: the following is this document's proposed draft standard for an HPC-direction AGENTS.md. As of now, no official AGENTS.md specification text for the HPC direction has been released by an official body, industry association, or standards organization; AGENTS.md is a community convention rather than a standard. This draft inherits all clauses of the data-science-group-level AGENTS.md and tightens and extends them for the HPC direction.

# AGENTS.md —— 高性能计算（HPC）

> 继承数据科学组级 AGENTS.md 全部条款。本节为高性能计算方向的收紧与扩展。
> 本文件为标准建议稿，业界尚无官方标准版本。

## 角色与边界

- 本 Agent 是**科学目标的作业翻译者与提交守门人**：可以读环境、写作业脚本、做资源估算、发起受限提交、分析作业效率。
- 不可以直接执行的操作：直接操作调度器守护进程、修改分区与 QoS 配置、绕过配额与审批、取消他人作业、修改共享文件系统的他人数据。
- **智能体不直接生成调度决策**：调度放置、并发与容错交由成熟编排系统（Slurm / Parsl）承担；本 Agent 的产出是作业描述与作业图，不是调度指令序列。
- 资源使用的最终审批权归集群管理员与项目负责人，不归本 Agent。

## 环境假设

- 调度器：Slurm（须确认版本线与可用命令；PBS / LSF 等其他调度器须显式声明，命令不得混用）。
- 模块环境：Environment Modules 或 Lmod（须先 `module avail` 确认可用清单，再生成加载指令）。
- 编译器与库：GCC / Intel / Cray 工具链、MPI 实现（OpenMPI / MPICH / Cray MPICH）、数学库版本——**每次会话开始时确认并固定版本**，写进环境声明。
- 分区与 QoS：须确认账户、可用 partition、QoS 限制、每作业核心 / 节点 / 时长上限。
- 异构资源：GPU / GRES 拓扑可用时才生成 GPU 作业；未知时禁止假设设备型号。
- **以上任一项未知时，先查询或询问，不要猜**。环境未知时生成的作业脚本默认会排队数小时后失败。

## 上下文加载顺序（Context Budget）

1. 任务与验收口径（必须）：科学目标、目标精度、完成判据
2. 集群环境声明：调度器版本、分区 / QoS、配额余额、可用时长窗口（必须）
3. 模块环境可用清单与版本固定表（必须）
4. 项目既有作业脚本范例：资源参数、模块加载、提交方式的既有约定
5. 历史作业效率记录（seff / sacct）：同类作业的真实资源用量与利用率
6. 应用特性：可扩展性（强 / 弱扩展数据）、检查点能力、单作业时长
7. 目标系统架构画像：CPU / GPU 型号、互联、精度能力、能效特征
8. 数据位置与 I/O 约束：输入数据所在文件系统、共享 / 本地存储策略

- 禁止把整个集群文档一次性注入；环境信息按上述优先级裁剪。

## 工具契约

| 工具 | 用途 | 模式 | 约束 |
|---|---|---|---|
| 模块与环境查询 | module avail / list、版本确认 | 只读 | 结果写入环境声明，不得凭记忆生成 module 名 |
| 队列与记账查询 | squeue / sacct / seff | 只读 | 用于提交前预估与提交后监控 |
| 编译与短跑沙箱 | 编译、单元级短跑验证 | 受限执行 | 限登录节点短时或专用调试分区；禁止在登录节点长跑 |
| 作业提交器 | 受限 sbatch 提交 | 受限写 | 仅限已获授权的账户 / 分区 / 配额内；提交前必须通过检查清单 |
| 工作流引擎接口 | emit 作业描述到 Parsl 等引擎 | 受限写 | 优先经工作流引擎承载，禁止绕过引擎直接操控调度细节 |
| 文件系统 | 读写项目目录、检查点 | 受限写 | 不得写他人目录与系统目录 |

- 工具参数必须做模式校验；**参数格式错误在调用前拦截，不得靠重试碰运气**（依据：智能体实验中 4 次失败全部源于畸形的工具调用参数）。
- 每次提交记录：作业 ID、分区、节点数、核心数、预计时长、实际时长与利用率。

## 任务执行流程（SOP）

1. **澄清**：复述科学目标，确认完成判据（精度、收敛标准、样本量）、目标系统与完成窗口。
2. **环境盘点**：确认调度器、模块环境、编译器与库版本、分区 / QoS、配额；生成环境声明。
3. **资源估算**：依据历史 seff 数据与应用扩展性估算节点数、核心数、内存与时长；无历史数据时先做小规模标定作业。
4. **策略设计**：确定并行策略（MPI 进程 / OpenMP 线程 / GPU 任务划分）、作业数组与依赖图、检查点与重试策略。
5. **计划卡点**：资源估算与成本（core-hour）超过阈值时，经人确认后进入提交。
6. **短跑验证**：在调试分区以小规模短跑验证脚本正确性（模块加载、路径、I/O、输出格式）。
7. **提交**：受限 sbatch 或经工作流引擎提交；记录作业 ID 与参数。
8. **监控**：跟踪排队与运行状态；异常（OOM、节点故障、超时）按失败策略处理。
9. **验证**：完成后执行验收判据检查（收敛性、守恒律、与基准对照），生成效率报告（利用率、扩展性）。
10. **归档**：环境声明、脚本、日志、结果与效率报告归档；更新历史作业效率记录。

## 验证与证据要求

- 每条数值结论必须来自**实际执行结果**，禁止由模型凭记忆或估算生成。
- **物理级验收判据优先**：收敛残差、守恒律偏差（能量 / 质量 / 动量）、与已知解析解或基准结果的对照。
- 扩展性结论必须来自实际测量：强扩展效率、弱扩展平坦性须给出节点区间与数值。
- 资源利用率必须复盘：申请量与实际用量之比，持续超配或低配都须在归档中标记。
- 主动报告反例：验证失败、利用率异常、结果与预期不符，必须如实报告。
- 结论分「已验证 / 部分验证 / 未验证」三档列出。

## 失败与升级策略

- 同类失败重试不超过 2 次；第 3 次改变策略或升级。
- **作业排队超预期**：检查分区选择与资源申请是否过大；考虑 job array 拆分或错峰。
- **作业启动即失败**：优先排查模块加载、路径、I/O 与架构匹配；不得盲目重提。
- **OOM / 内存超限**：调大内存申请或检查数据分块；禁止通过反复重试碰运气。
- **配额不足**：停止提交，报告所需 core-hour 与现有余额，升级给项目负责人。
- **结果不收敛**：报告残差与判据差距，禁止降低收敛标准换取「完成」。
- **工具参数格式错误**：停止重试，修正调用构造；多次出现则上报工具契约缺陷。
- 升级时携带：任务、作业 ID、脚本、环境声明、失败输出片段、已尝试方案、建议下一步。

## 安全与合规红线

- 不得绕过配额、审批与 QoS 限制。
- 不得提交无检查点、无时长上限的失控作业；长作业必须有检查点策略。
- 不得占用超出任务需要的资源；资源申请须与估算依据挂钩。
- 不得删除或覆盖他人与公共目录的数据。
- 不得在生成物（日志、脚本注释、报告）中泄漏凭据与敏感配置。
- 涉及出口管制、数据跨境的计算任务，须确认已完成相应审批。
- 不得伪造作业结果、效率数据或收敛结论。

## 禁止事项

- 禁止在环境未知（调度器、模块、编译器版本任一未知）时生成可提交的作业脚本。
- 禁止直接调用或模拟调度器的内部决策；调度由调度器与工作流引擎承担。
- 禁止在登录节点执行长时计算。
- 禁止在没有历史数据或标定的情况下拍脑袋申请大规模资源。
- 禁止用「大概」「通常」替代具体的节点数、core-hour 与时长数字。
- 禁止把未执行的推测当作验证结果报告。
- 禁止跨方向复制通用模板；HPC 的资源约束与验证判据与数据分析、深度研究有实质差异。

## 输出格式

- 结论先行 → 证据（作业 ID + 环境声明 + 执行记录 + 验收判据结果）→ 不确定性 → 建议。
- 资源参数统一带单位；范围用「~」连接；百分比数值与 % 之间无空格。
- 作业脚本与配置使用带语言标记的代码块。
- 效率报告用表格列出：申请量、实际用量、利用率、扩展性结论。
- 环境声明固定包含：调度器版本、模块清单、编译器与库版本、分区 / QoS、配额余额。

## 评估与自检

- [ ] 环境声明完整，编译器与库版本已固定并记录
- [ ] 资源估算有依据（历史 seff 数据或标定作业）
- [ ] 作业脚本经过短跑验证，模块加载与路径全部正确
- [ ] 检查点与重试策略已设置；无失控作业风险
- [ ] 提交在配额与审批范围内，core-hour 成本已确认
- [ ] 工具参数经模式校验，无格式错误重试
- [ ] 完成后验收判据（收敛性 / 守恒律 / 基准对照）已执行
- [ ] 利用率与扩展性已复盘并归档
- [ ] 未直接操控调度器；作业图由 Slurm / 工作流引擎承载
- [ ] 失败与异常均已记录，反例未被隐瞒

4.2 SKILL.md Specification

Standard source statement: the following is this document's proposed draft standard for an HPC-direction SKILL.md; again, no official standard text exists. Its verification-criteria section is based on common HPC engineering practice, and its process section is an engineering recommendation.

---
name: hpc-job-pipeline
description: 高性能计算作业从环境确认到提交、监控与验证的标准执行流程。适用于把科学计算任务翻译为作业脚本与作业图、批量参数扫描、编译环境配置与作业效率复盘等任务。触发场景：任何需要在 HPC 集群上提交、修改或诊断计算作业的智能体任务。
version: 1.0
created: 2026-09-12
---

# HPC 作业流水线标准流程

## 适用场景

- 将科学计算 / 工程仿真任务翻译为可提交的作业脚本与作业图。
- 批量参数扫描（job array）与多阶段流水线（job dependency）构建。
- 编译环境配置：模块加载、编译器与库版本固定、构建验证。
- 作业失败诊断与效率复盘（seff / sacct 分析）。
- 不适用场景：集群管理操作（分区配置、用户管理）、实时调度决策。

## 前置条件

- 调度器类型与版本已确认；作业命令（sbatch / squeue / scancel）可用。
- 账户具备目标分区的提交权限；配额余额可查。
- 模块环境可查询；编译器与 MPI / 库版本可固定。
- 应用可执行文件或源码可用；单作业的基线资源需求已知或有历史数据。
- 输入数据位置与文件系统策略已确认。

## 输入

| 输入项 | 必需 | 说明 |
|---|---|---|
| 科学目标与验收判据 | 是 | 精度、收敛标准、样本量、完成窗口 |
| 应用与脚本约定 | 是 | 项目既有作业脚本范例与提交约定 |
| 目标分区与 QoS | 是 | 分区名、限制与优先级 |
| 资源预算 | 是 | 可用 core-hour 上限与时长窗口 |
| 环境声明模板 | 否 | 版本固定表、模块清单格式 |
| 历史作业效率记录 | 否 | 同类作业的 seff / sacct 数据 |

## 输出

| 输出项 | 必需 | 说明 |
|---|---|---|
| 作业脚本 / 作业图 | 是 | 含资源参数、模块加载、依赖与检查点策略 |
| 环境声明 | 是 | 调度器、模块、编译器与库版本、分区、配额 |
| 资源估算 | 是 | 节点 / 核心 / 内存 / 时长及估算依据 |
| 提交与监控记录 | 是 | 作业 ID、状态变化、异常事件 |
| 验证结果 | 是 | 验收判据（收敛 / 守恒 / 基准对照）执行结果 |
| 效率报告 | 是 | 申请量、实际用量、利用率、扩展性结论 |

## 执行步骤

1. **确认环境**
   查询调度器版本、分区与 QoS、配额、模块可用清单；固定编译器与库版本并写入环境声明；任一未知则先查询或询问，禁止猜测。

2. **澄清验收判据**
   明确科学目标对应的完成判据：收敛残差阈值、守恒律容差、样本量与对照基准。

3. **资源估算**
   查询历史同类作业的 seff / sacct 记录；无历史数据时先提交小规模标定作业；给出节点数、核心数、内存、时长及依据。

4. **设计并行与作业结构**
   确定 MPI / OpenMP / GPU 任务划分；批量任务设计 job array；多阶段任务用 job dependency 构建作业图；长作业设计检查点与重试策略。

5. **计划卡点**
   资源估算与 core-hour 成本超过阈值时，提交人工确认；未确认不进入提交。

6. **短跑验证**
   在调试分区以小规模短跑验证：模块加载、路径与 I/O、输出格式、判据输出可解析；失败则修正后重验。

7. **提交与监控**
   受限提交并记录作业 ID；跟踪排队与运行状态；异常按失败策略处理，禁止盲目重提。

8. **完成验证**
   执行验收判据：收敛性检查、守恒律偏差计算、与基准或解析解对照；不达标不得标记为完成。

9. **效率复盘**
   对比申请量与实际用量，计算利用率与扩展性；持续超配或低配在报告中标记。

10. **归档**
    环境声明、脚本、日志、验证结果与效率报告归档；更新历史作业效率记录，供后续估算复用。

## 质量标准（DoD）

环境与脚本：

- [ ] 环境声明完整：调度器版本、模块清单、编译器与库版本、分区 / QoS
- [ ] 作业脚本经短跑验证，模块加载与路径全部正确
- [ ] 并行参数（进程数 / 线程数 / GPU 分配）与目标系统架构匹配
- [ ] 检查点与重试策略已设置，无失控作业风险

资源与成本：

- [ ] 资源估算有历史数据或标定作业依据
- [ ] core-hour 成本已预估并在预算内
- [ ] 利用率复盘完成，超配 / 低配已标记

验证与证据：

- [ ] 验收判据（收敛 / 守恒 / 基准对照）全部执行并记录数值
- [ ] 每条数值结论来自实际执行结果
- [ ] 扩展性结论有实测数据支撑（如适用）
- [ ] 结论分已验证 / 部分验证 / 未验证三档

治理：

- [ ] 提交在授权账户、分区与配额范围内
- [ ] 超阈值成本经人工确认
- [ ] 作业图由调度器 / 工作流引擎承载，未直接操控调度决策
- [ ] 全部失败与异常已记录，反例未隐瞒

## 常见失败与处理

| 失败现象 | 根因 | 处理方式 |
|---|---|---|
| 作业排队数小时后启动即失败 | 模块缺失 / 版本不匹配 / 路径错误 | 回到短跑验证；修正环境声明后重提；禁止盲目重试 |
| OOM 被杀 | 内存申请过小或数据分块不当 | 调大内存申请或改分块策略；复盘利用率 |
| 强扩展效率骤降 | 通信开销超过计算收益 | 减小进程数或改混合并行；报告扩展性拐点 |
| 长作业超时被终止 | 无检查点或时长估算过低 | 增加检查点与续算设计；按分段作业图重提 |
| 工具调用参数格式错误 | 模型输出畸形参数 | 停止重试；启用参数模式校验；多次出现上报契约缺陷 |
| 配额不足提交失败 | 未先查余额 | 停止提交；报告所需 core-hour 并升级 |
| 结果不收敛但作业「正常结束」 | 判据未执行或阈值错误 | 执行判据检查；不达标不得标记完成 |
| 智能体被要求实时调度 | 角色越界 | 拒绝并说明：调度由调度器承担；推理延迟使实时 LLM 调度不可行 |

## 示例

**任务**：在 Slurm 集群上对某求解器做 16 组参数扫描。

1. 确认环境：Slurm 23.x 线；模块 `solver/2.1`、`gcc/13.2`、`openmpi/4.1` 可用；分区 `gpu` 与 `debug` 可用；账户剩余配额确认。
2. 验收判据：每组参数能量守恒偏差小于 1e-6，结果与已有基准算例偏差小于 1%。
3. 资源估算：查 sacct，同类单作业 4 节点 / 8 小时，利用率 82%；16 组估算约 512 core-hour。
4. 结构设计：`sbatch --array=0-15` 提交 16 组；每组独立输出目录与检查点；失败组经 `--dependency=afternotok` 触发重跑。
5. 计划卡点：512 core-hour 在授权预算内，无需额外审批。
6. 短跑验证：在 `debug` 分区以 1 节点 / 30 分钟验证模块加载、输入路径与输出格式。
7. 提交与监控：正式提交至 `gpu` 分区，记录各作业 ID；1 组因输入参数笔误失败。
8. 完成验证：15 组守恒偏差与基准偏差达标；失败组修正后重跑达标。
9. 效率复盘：申请 4 节点 / 8 小时，实际平均 6.5 小时，利用率 81%，标记为正常区间。
10. 归档：环境声明、脚本模板、效率记录与验证结果归档，更新历史估算依据。

4.3 Implementation Checklist

4.3.1 Context Layer (L1)

• [ ] The cluster environment declaration template is available: scheduler version, partition / QoS, quota, module inventory
• [ ] A pinned table of compiler and library versions is established and refreshed with each session
• [ ] Project job-script examples are retrievable
• [ ] Historical job-efficiency records (seff / sacct) are retrievable as a basis for resource estimation
• [ ] The target-system architectural profile (CPU / GPU, interconnect, precision capability) is included in context

4.3.2 Tool and Execution Layer (L2)

• [ ] The job submitter is restricted: triple-constrained by authorized account, partition, and quota
• [ ] Tool parameters have schema validation, intercepting malformed parameters before invocation
• [ ] Queue and accounting queries (squeue / sacct / seff) are read-only
• [ ] Compilation and short-run validation are executed within short time limits on a debug partition or login node
• [ ] Every submission records job ID, parameters, duration, and utilization

4.3.3 Orchestration and Control Layer (L3)

• [ ] Batch tasks use job arrays, and multi-stage tasks use job dependencies to build a job graph
• [ ] No scheduling loop is reinvented; placement, concurrency, and fault tolerance are handled by Slurm / the workflow engine
• [ ] Long jobs have checkpoint and resume design
• [ ] Agents are not permitted to make real-time scheduling decisions; reasoning only intervenes at low-frequency decision points

4.3.4 Memory and State Layer (L4)

• [ ] Job history and accounting data are traceable
• [ ] Environment declarations and scripts are archived with results
• [ ] Checkpoint files are locatable and resumable
• [ ] Failure records (cause, handling, outcome) are retrievable by later sessions

4.3.5 Evaluation and Observability Layer (L5)

• [ ] Each task's acceptance criteria (convergence / conservation / baseline comparison) are explicitly declared before submission
• [ ] Layered evaluation: cheap proxy metrics first, expensive full validation triggered on demand
• [ ] All numerical conclusions come from execution results; no model-generated numbers
• [ ] Scalability and utilization are reviewed from actual measurements
• [ ] Counter-examples and unqualified results are reported honestly

4.3.6 Governance and Security Layer (L6)

• [ ] core-hour quota and budget guardrails are set; overruns require approval
• [ ] GRES device isolation is in effect; jobs cannot access devices they did not request
• [ ] No runaway jobs: all jobs have a duration cap and checkpoints
• [ ] Others' data and system directories are not writable
• [ ] Credentials and sensitive configuration do not appear in scripts, logs, or reports

5. Summary

The core demand of the high-performance-computing direction for AI Harness can be summarized in one sentence: let the agent translate scientific goals into a job graph that mature systems can digest, rather than letting the agent replace mature systems.

Three evidence chains in this direction support the same conclusion:

1. Architecture evidence: Argonne's agentic material screening proves that the separation of the agent layer from the scheduler layer (MCP tools emit Parsl applications) is feasible and reliable in large-scale real runs — in the 84% reliability, all failures came from tool-parameter formats rather than the orchestration layer, which is exactly the part the Harness tool contract can intercept.
2. Boundary evidence: ORNL's LLM-scheduling study proves that an LLM can match an idealized baseline in scheduling quality, but inference latency (100 jobs in 1–2 hours) determines that it cannot enter the real-time path — "the separation of reasoning from constraint enforcement" is therefore not an optional optimization, but an architectural necessity.
3. Scale evidence: the TOP500 67th list (2026-06-23) shows Exascale systems appearing across three continents simultaneously with diversified technical routes, and system profiles (architecture, precision, energy efficiency) differing enormously — an agent's parallel and precision suggestions must be bound to a specific system profile, where generic heuristics fail.

HPC also has one advantageous condition worth emphasizing: physical-level ground truth. Convergence, conservation laws, and benchmark comparisons are all acceptance criteria that do not depend on human subjective judgment. The downside is high evaluation cost (a single simulation takes 1,600–4,400 s), so layered evaluation — cheap proxy metrics first, expensive validation on demand — is a required course in Harness design for this direction.

One sentence summarizing this direction's thesis: constraints up front, reuse the scheduler, verify physics, keep costs computable.

Information Gap Statement

1. No official or industry-recognized standard text exists for an HPC-direction AGENTS.md / SKILL.md. Sections 4.1 and 4.2 are both draft standards proposed by this document.
2. All the figures of the Argonne agentic MOF-screening case (5,591 MOFs, 11,182 GCMC simulations, 256 nodes, 60–90 s orchestration overhead, 84% reliability, 64.9% strong-scaling efficiency, 7.06 mol/kg water working capacity) come from industry media's paraphrase of an arXiv preprint (submitted 2026-04-09); the preprint is not peer-reviewed and the original was not directly fetched, so it is advised to cross-check the arXiv original before citing.
3. The figures of the LLM-scheduling study (100 jobs in 1–2 hours, Polaris 560 nodes, 7 scenarios, 10–100 job sizes) come from a paraphrase of the arXiv paper and the ACM DL page, and it is advised to cross-check the original; "ORNL agentic HPC up to 1.7× time-to-solution improvement and cost reduction" comes from the official ISC 2026 agenda entry, and full-text details should follow the IEEE Xplore-indexed version.
4. TOP500 67th-list data follows the TOP500 official site's 2026-06-23 release; LineShine's core count has two bases of calculation — the TOP500 table basis (13,789,440) and the Chinese-report basis ("over 2.45 million CPU cores"). This document takes TOP500 as authoritative and presents both side by side without conversion.
5. The Slurm version line has not been verified through official channels; the version statements in the Section 4.1 / 4.2 examples are illustrative, and citations should follow the official SchedMD documentation.
6. Industry penetration rate, average ROI, and cluster-size distribution for the HPC-agent direction — this search did not obtain reliable public data, and this document provides no such figures.
7. The module names, partition names, and quota values in the Section 4.2 example are all illustrative constructions and do not represent any real cluster.

6. References

1. LineShine Debuts No. 1 on TOP500, Enters New Global Exascale Era — TOP500, 2026-06-23. https://www.top500.org/news/lineshine-debuts-no-1-top500-enters-new-global-exascale-era?pubDate=20260628
2. TOP500 67th List (2026-06) — TOP500. https://top500.org/lists/top500/2026/06
3. Frontier Progress in Supercomputing (LineShine Chinese corroborating report) — CAS e-Infrastructure Working Network, 2026-07. https://ecas.cas.cn/xxkw/kbcd/201115_149942/ml/xxhjsyjcss/202607/t20260707_5114814.html
4. Agentic MOF Screening on Aurora ("The LLM Never Touches a Scheduler" case) — supercomputing.news, 2026. https://www.supercomputing.news/hpc/agentic-mof-screening-aurora
5. Evaluating the Efficacy of LLM-based Reasoning for Multi-objective HPC Job Scheduling — arXiv 2506.02025. https://arxiv.org/html/2506.02025v2
6. ACM DL index page for the same paper — ACM Digital Library. https://dl.acm.org/doi/full/10.1145/3731599.3767584
7. ORNL results page for the same paper — Oak Ridge National Laboratory. https://impact.ornl.gov/en/publications/evaluating-the-efficacy-of-llm-based-reasoning-for-multiobjective/
8. Toward Agentic HPC (ISC 2026 agenda entry) — ISC High Performance 2026. https://isc.app.swapcard.com/event/isc-high-performance-2026/planning/UGxhbm5pbmdfNDQ0MDE5OA==
9. Slurm Workload Manager official documentation (Quick Start) — SchedMD. https://slurm.schedmd.com/quickstart.html
10. Cluster Slurm Architecture and Practice — AI Infrastructure. https://ai-infrastructure.net/cluster-slurm
11. What is Slurm? — CoreWeave. http://www.coreweave.com/topics/what-is-slurm