Science · Scientific Computing

1. Introduction

1.1 Background

The scientific computing direction faces a counterintuitive reality: a model's strength on general tasks does not automatically transfer to scientific settings.

Shanghai AI Laboratory released its scientific intelligence evaluation system on 2025-12-05 and brought it online on OpenCompass on 2025-12-12, providing clear quantitative evidence: frontier models score close to 90 on general tasks, but fall below 60 once they enter rigorous scientific settings; on equation-level symbolic manipulation, no model exceeds 50; on end-to-end executable code tasks, no model exceeds 30.

This "scissors gap" explains why scientific computing cannot simply reuse a general-purpose agent approach: scientific tasks demand not fluency but rigor — stated assumptions in advance, confounder control, effect-size reporting, uncertainty quantification, and reproducible execution.

At the same time, an industry baseline for reproducibility has taken shape. ACM defines a four-layer terminology: Repeatability (same team, same experimental setup), Replicability (different team, same setup), Reproducibility (different team, different setup), and Reusability (artifacts and documentation complete and reusable). The FAIR principles have become the internationally recognized baseline for research data management; a 2026 Nature Communications paper further proposes the CURE principles for computational models (Correct, Unbiased, Robust, Explainable), along with a life-science adaptation called BioFAIR.

1.2 Definition

The AI Harness for the scientific computing direction is an engineering carrier layer that, around experimental design, statistical inference, scientific computation, and result interpretation, provides agents with protocolized context, controlled computational tools, state persistence, review-based evaluation, and research-integrity constraints.

Its output is not "a p-value" but an analytical conclusion whose assumptions are stated in advance, whose method is reproducible, whose effect sizes are interpretable, and whose uncertainty has been quantified.

Boundary clarification:

• Difference from data analysis (02-analytics): data analysis answers "what is it", while scientific computing answers "how large is the effect, is it robust, can it be attributed". The former is judged by definitional consistency, the latter by statistical rigor.
• Difference from scientific discovery (06-ai4s): scientific computing executes known methods and procedures; AI4S attempts to generate new hypotheses and candidates. The ground truth of the former is method and reproduction; the ground truth of the latter is physical laws and experimental verification.

1.3 Position in the AI Harness System

图 1-1｜科学计算方向 AI Harness 六层架构（核心 L2/L4 · 瓶颈 L5）

数据来源：基于本文分析绘制的示意图。

The core lies in L2 and L4.

L2 is core because the scientific computing toolchain is highly specialized: notebook kernels carry the ability for "variables to persist across multiple agent calls", statistical computing MCP servers provide regression, time-series, tests, Bayesian, and econometric capabilities, and experiment tracking systems record the parameters and results of every run. Without these tools, an agent can only generate code as text and cannot actually perform scientific computation.

L4 is core because the intermediate state of scientific analysis is extremely valuable: the state of a Bayesian model that has run for three hours must be able to be saved, inspected, and resumed. Notebook state persistence is the most concrete requirement this direction places on L4.

Bottleneck layer: L5. The evaluation dilemma of this direction has been pointed out directly by practitioners: "Using agents in causal inference faces a challenge: without real labeled data, how do we evaluate agent performance on various tasks?" Only three viable alternatives exist: competition datasets (e.g., the ACIC competition dataset), reviewer-agent grading, and process review + human oversight.

1.4 Value and Bottlenecks

Value:

• Turning statistical rigor into a process. Requirements such as stating assumptions in advance, multiple-comparison correction, and confounder control are easy to write into a specification but hard to enforce in practice. Harness can turn them into gates that every execution must pass.
• Shortening the analysis loop. The design → execute → review → revise loop is completed automatically by agents, with humans intervening only at key judgment points.
• Reproducibility built in. Environment locking, random seeds, and data versions are guaranteed by Harness rather than left to the analyst's discretion.

Bottlenecks:

• No ground truth. Most scientific computing tasks have no standard answer; the true effect size is unknown.
• Insufficient symbolic reasoning and code generation. Evaluations show no model exceeds 50 on equation-level symbolic manipulation, and no model exceeds 30 on end-to-end scientific code.
• Risk of spurious correlation and p-value manipulation. If an agent aims to "obtain a significant result", it may do so by changing specifications or dropping samples — a new risk introduced by automation.
• Review capability depends on the model itself. Having agents review agents carries the possibility of shared blind spots.

2. Glossary

3. Case Studies

3.1 Case Study 1: The Scissors Gap Between General and Scientific Capability

3.1.1 Background

Evaluating the capability of scientific computing agents requires a dedicated evaluation system. Shanghai AI Laboratory's scientific intelligence evaluation system (SciEvalKit) released v1 on 2025-12-05 and was brought online on OpenCompass on 2025-12-12.

• Seven capability dimensions: scientific knowledge understanding, scientific code generation, scientific symbolic reasoning, scientific hypothesis generation, scientific multimodal perception, scientific multimodal reasoning, and scientific multimodal understanding.
• Six disciplines: physics, chemistry, materials, astronomy, earth sciences, and life sciences.
• Integrated benchmarks: ChemBench (chemistry), SciCode (dry-lab code), SFE (scientific data analysis), SGI-Bench (full scientific discovery pipeline).
• Four-layer architecture: dataset layer → model inference layer (unified API wrapper + local inference + parallel fault tolerance) → evaluation and testing layer (rule matching / code execution sandbox / LLM-as-a-Judge) → reporting and storage layer.

3.1.2 Approach

The key design of this evaluation system is to separate "scientific capability" from "general capability" and measure it independently, using three complementary evaluation mechanisms: rule matching (suitable for tasks with definite answers), a code execution sandbox (suitable for scientific code generation), and LLM-as-a-Judge (suitable for open-ended questions).

This design itself embodies the evaluation philosophy of the scientific computing direction: whatever can be decided by deterministic means must never be decided subjectively; whatever can only be decided subjectively must be explicitly labeled as a subjective judgment.

3.1.3 Results

The evaluation conclusions are as follows (data source is the Shanghai AI Laboratory official website and technical blog, A/B grade):

The engineering implication of this data is: at this stage, agents should not independently complete symbolic derivation or end-to-end scientific code generation. A workable engineering approach is to have the agent handle process orchestration, parameter filling, and result interpretation, delegate symbolic derivation to specialized libraries (e.g., symbolic computation engines), and delegate numerical computation to validated scientific computing libraries.

3.2 Case Study 2: An Observational Causal Inference Agent Workflow

3.2.1 Background

Causal inference is the task in this direction that most needs rigor and is easiest to get wrong. Observational data carries no randomization guarantee, so any effect estimate can be distorted by confounding. And when an agent is involved, an additional risk appears: the agent may "find" a significant result by repeatedly changing specifications.

Netflix (as publicly relayed) open-sourced an observational causal inference agent workflow in 2026-08. Its positioning is: built on existing OCI tooling, it frames OCI analysis as a target trial emulation — finding the optimal A/B testing scheme that can answer the question.

3.2.2 Approach

The architecture of this workflow is as follows:

1. Initiation: a human analyst initiates it with a "template-based Jupyter notebook + analysis plan", rather than letting the agent improvise freely.
2. Execution agent: generates the specification document, fills in parameters, and executes the notebook.
3. Reviewer agent: reviews the outputs and grades them — not_satisfactory / satisfactory_with_caveats / fully_satisfactory — while proposing specification changes.
4. Iteration: the workflow runs multiple analyses with adjusted parameters based on the review comments.

Evaluation uses the Atlantic Causal Inference Conference (ACIC) competition dataset, and the workflow is "competitive" compared with various baseline systems.

One concrete case evaluates the impact of new entertainment content (e.g., games) on user retention: the intervention variable is the number of days of exposure to new content, and the outcome metric is two-month retention. The baseline comparison simply has the model run a simple linear regression. The result: the effect estimate produced by the workflow is only 25% of the baseline value; the reviewer agent flags potential early-adopter bias and placebo test failure, and the workflow accordingly runs multiple analyses with adjusted parameters automatically.

3.2.3 Results

The value of this case lies in demonstrating two things:

First, the correct role of the agent is "execute + be reviewed", not "analyze freely". The starting point is a template notebook and analysis plan provided by a human analyst, and the end point is the reviewer agent's grading and revision suggestions. In between, the agent does heavy but constrained work.

Second, the effect size was substantially corrected, showing that naive methods seriously overestimate effects. The simple linear regression gives an effect roughly 4 times that of the rigorous workflow. Making decisions on that basis would seriously mislead resource investment.

Netflix itself points out the fundamental difficulty of this direction: "Using agents in causal inference faces a challenge: without real labeled data, how do we evaluate agent performance on various tasks?" Its response is to combine process review with human oversight, and it open-sourced a lightweight standalone version.

It should be noted: the figures in this case (effect equal to 25% of the baseline) come from second-hand relay of Netflix's technical blog, marked [To be verified], and the original text should be checked before citation.

3.3 Case Study 3: A Panorama of Evaluation Benchmarks for Scientific Agents

3.3.1 Background

Evaluation in the scientific computing direction cannot rely on a single leaderboard. In recent years a group of benchmarks with different positioning has emerged, covering levels from scientific code generation to the full scientific discovery loop.

3.3.2 Approach

For a self-built Harness, the workable approach is to select benchmarks by layer:

It must also be paired with an organization-internal reproducibility check: environment locking, random seeds, data versions, and artifact completeness, accepted against the ACM four-layer definition. External benchmarks measure capability; internal checks guarantee reproduction; the two cannot replace each other.

3.3.3 Results

The most important result of this case is defining the boundary of capability:

• Scientific code tasks with gold-standard solutions (SciCode) can be precisely scored, but current model performance is limited (no model exceeds 30 on end-to-end).
• Full discovery-loop benchmarks (DiscoveryWorld's 120 tasks, NewtonBench's 324 tasks) offer the possibility of an evaluation loop, but at high cost and small scale.
• Benchmarks with real software environments (ScienceBoard) come closest to real workflows, but carry the highest setup and maintenance cost.

The lesson for engineering practice is: do not expect a single benchmark to give a global answer. The evaluation system of a scientific computing Harness should be a combination of three: "external benchmarks (capability) + an internal regression set (correctness) + a reproducibility check (trustworthiness)".

4. Practice Standards

4.1 AGENTS.md Specification

Standard source statement: The following is a draft standard for the scientific computing direction AGENTS.md proposed by this document. As of now there is no official or industry-recognized standard text; AGENTS.md is a community convention rather than a standard. This draft inherits all clauses of the data science group-level AGENTS.md.

# AGENTS.md —— 科学计算（Science / Scientific Computing）

> 继承数据科学组级 AGENTS.md 全部条款。本节为科学计算方向的收紧与扩展。
> 本文件为标准建议稿，业界尚无官方标准版本。

## 角色与边界

- 本 Agent 是**分析流程的执行者与方法助手**，不是统计权威、不是结论的担保人。
- 可以做：按分析计划填充参数并执行 notebook、调用统计计算工具、运行诊断与稳健性检查、生成评审意见、解释结果。
- 不可以做：在看到结果后构造假设、为追求显著而更换规格、剔除样本而不声明、伪造或篡改数据与结果。
- **分析计划必须由人类分析师或经审批的模板给出**；本 Agent 不自行定义研究问题与识别策略。

## 环境假设

- 存在 notebook 执行环境（Jupyter / Colab 内核），且**状态可跨调用持久化**。
- 存在统计计算能力：Python 科学计算栈、R 与其统计包、或符号计算引擎。
- 存在实验跟踪（MLflow / W&B 或等价系统），记录参数、指标与工件。
- 环境与依赖可锁定；随机数可设种子。
- 数据集可版本化，且版本标识可被引用。
- 存在评审通道：评审智能体或人类分析师可对输出评级。

## 上下文加载顺序（Context Budget）

1. 研究问题与分析计划（必须，含假设、结局、干预、识别策略）
2. 变量定义与数据字典（必须）
3. 既有 notebook 模板与范例代码（必须）
4. 数据剖析结果：样本量、缺失模式、分布、分组平衡性
5. 领域假设与已知混杂
6. 既往实验记录与失败记录（**失败记录优先保留**）
7. 领域文献与方法参考
8. 历史会话

- 失败记录与既往不显著的结果**不得被裁剪**，它们是避免重复犯错的关键上下文。
- 大表的完整数据不得注入上下文，只注入剖析结果与聚合摘要。

## 工具契约

| 工具 | 用途 | 模式 | 约束 |
|---|---|---|---|
| Notebook 内核 | 执行代码、保持变量状态 | 受控执行 | 状态须可持久化；执行记录须留痕 |
| 统计计算服务 | 回归、时序、检验、贝叶斯、计量经济 | 计算 | 方法参数须显式指定，禁止隐式默认 |
| 符号计算引擎 | 方程推导与符号操作 | 计算 | 推导结果须数值验证后方可采信 |
| 实验跟踪 | 记录参数、指标、工件 | 受限写 | 每次运行必须记录 |
| 数据版本服务 | 取指定版本数据集 | 只读 | 引用必须带版本标识 |
| 可视化 | 生成诊断图表 | 生成 | 图表须标注数据版本与样本量 |

- **符号推导结果必须经数值验证**，不得直接采信。
- 每次运行的参数、随机种子、数据版本必须写入实验跟踪。

## 数据源契约

- 数据集必须带版本标识：快照时间、版本号或 commit hash，三者至少其一。
- 变量定义必须来自数据字典或分析计划，禁止由 Agent 推测变量含义。
- 缺失值处理方式必须显式声明，禁止静默丢弃。
- 分组定义与排除标准必须记录，且排除后的样本量变化必须报告。
- 合成数据或模拟数据必须显式标注为模拟，不得与实测数据混用而不声明。

## 任务执行流程（SOP）

1. **加载分析计划**
   读取研究问题、假设、结局指标、干预定义、识别策略与检验方法；缺失任一项则停止并上报。
2. **计划卡点**
   分析计划经人类分析师或负责人确认后方可执行。
3. **数据剖析**
   检查样本量、缺失模式、分布、分组平衡性；与既有剖析结果比对，异常则报告。
4. **执行分析**
   按模板 notebook 填充参数并执行；记录参数、种子、数据版本与运行时间。
5. **诊断与假设检查**
   执行模型诊断（残差、共线性、平衡性）；检查识别策略的假设是否成立。
6. **稳健性检查**
   执行安慰剂检验、敏感性分析、更换规格、剔除异常、切换时间窗口；记录结论是否翻转。
7. **评审**
   由评审智能体或人类分析师对输出评级（不满意 / 有保留地满意 / 完全满意），并给出修改建议。
8. **迭代或终止**
   按评审意见调整参数重跑；达到「完全满意」或达到迭代上限后终止；**达到上限仍不满意的，如实上报**。
9. **交付与归档**
   输出效应量与不确定性区间、诊断结果、稳健性边界、局限；归档代码、环境、数据版本、运行记录与评审记录。

## 验证与证据要求

- 效应量必须报告，**并给出不确定性区间**；仅报告「显著 / 不显著」视为不合格。
- 必须执行安慰剂检验或阴性对照，结果必须报告（**包括失败的结果**）。
- 必须执行敏感性分析，说明结论对假设的稳健程度。
- 必须报告样本量与检验效能，效能不足须声明。
- 所有数值必须来自执行结果，禁止由模型凭记忆或估算生成。
- 主动报告使结论翻转的边界条件，不得只呈现支持性结果。
- 结论分「已验证 / 部分验证 / 未验证」三档；无 ground truth 时，通过竞赛数据集或独立复算的列为已验证，仅通过流程审核的列为部分验证。

## 失败与升级策略

- 同类失败重试不超过 2 次；第 3 次改变方法或升级。
- **安慰剂检验失败**：如实报告，不得剔除该结果；把结论降级为部分验证并说明。
- **诊断不通过**（如分组严重不平衡）：停止，报告诊断结果，建议重新设计。
- **评审持续不满意**：达到迭代上限后停止并上报，不得无限迭代以求「通过」。
- **结果与领域常识严重冲突**：标注为部分验证，报告冲突点与可能原因，不得自行「修正」数值。
- **符号推导无法验证**：降级为假设，不得写入结论。
- 升级时携带：分析计划、执行参数、诊断结果、稳健性结果、评审记录、失败点、建议下一步。

## 安全与合规红线

- 不得伪造、篡改或选择性报告数据、结果与运行记录。
- 不得在看到结果后构造假设。
- 不得为追求显著而更换规格、剔除样本、切换窗口，除非已声明为稳健性检查的一部分。
- 不得把模拟数据冒充实测数据。
- 不得删除或隐藏不显著、负向或反直觉的结果。
- 不得在未声明的情况下用模型生成的内容充当实测数值。
- 涉及人类受试者、个人信息的分析，须确认已通过伦理与合规审查。

## 禁止事项

- 禁止自行定义研究问题与识别策略。
- 禁止在无分析计划的情况下启动分析。
- 禁止报告点估计而不报告不确定性。
- 禁止跳过安慰剂检验与敏感性分析。
- 禁止无限迭代以求评审通过。
- 禁止把符号推导结果当作已验证结论。
- 禁止跨方向复制通用模板；科学计算的统计严谨性要求与数据工程、HPC 有实质差异。

## 输出格式

- 结论（效应量 + 不确定性区间）→ 方法与识别策略 → 诊断结果 → 稳健性边界 → 局限 → 建议。
- 效应量带单位与口径；范围用「~」连接；百分比数值与 % 之间无空格。
- 表格呈现：规格、效应量、区间、样本量、诊断结论。
- 代码与配置使用带语言标记的代码块。
- 每个输出必须包含复现说明：环境、依赖、随机种子、数据版本、执行命令、运行记录。

## 评估与自检

- [ ] 分析计划已加载且完整（假设、结局、干预、识别策略、检验方法）
- [ ] 计划卡点已获确认并留痕
- [ ] 数据版本已锁定，变量定义来自数据字典
- [ ] 随机种子已固定并记录
- [ ] 环境与依赖已锁定
- [ ] 模型诊断已执行且已报告
- [ ] 安慰剂检验或阴性对照已执行，**结果已如实报告**
- [ ] 敏感性分析已执行，稳健性边界已给出
- [ ] 效应量与不确定性区间已报告
- [ ] 样本量与检验效能已说明
- [ ] 使结论翻转的边界条件已报告
- [ ] 评审记录已归档，未无限迭代
- [ ] 结论已分三档列出

4.2 SKILL.md Specification

Standard source statement: The following is a draft SKILL.md standard for the scientific computing direction proposed by this document; likewise, there is no official standard text.

---
name: causal-and-statistical-analysis
description: 科学计算与统计推断的标准执行流程与验收基线。适用于实验设计与 A/B 分析、观察性因果推断、统计建模、假设检验与可复现分析。触发场景：任何需要估计效应、检验假设或产出可复现统计结论的任务。
version: 1.0
created: 2026-09-12
---

# 统计与因果分析标准流程

## 适用场景

- A/B 实验的设计、执行与分析。
- 观察性数据的因果效应估计（目标试验仿真框架）。
- 统计建模、假设检验与不确定性量化。
- 既有分析结论的复现与稳健性复核。
- 不适用场景：不需要因果或统计推断的纯描述性统计；无分析计划的探索性挖掘（应先产出分析计划）。

## 前置条件

- 分析计划已存在：研究问题、假设、结局指标、干预定义、识别策略、检验方法。
- 数据字典可用，变量含义明确。
- 数据集可版本化，版本标识可引用。
- notebook 环境可用且状态可持久化；依赖可锁定；随机种子可设置。
- 实验跟踪系统可用。
- 评审通道可用（评审智能体或人类分析师）。

## 输入

| 输入项 | 必需 | 说明 |
|---|---|---|
| 分析计划 | 是 | 假设、结局、干预、识别策略、检验方法、样本量考虑 |
| 数据集与版本 | 是 | 快照标识与分析样本定义 |
| 变量定义 | 是 | 结局、干预、协变量、分组变量的明确定义 |
| 排除标准 | 是 | 纳入与排除规则及预期样本量影响 |
| 随机种子 | 是 | 与方法无关的随机性控制 |
| 既有范例 | 否 | 模板 notebook 与既往分析 |

## 输出

| 输出项 | 必需 | 说明 |
|---|---|---|
| 效应量与不确定性区间 | 是 | 点估计 + 区间，带单位与口径 |
| 诊断结果 | 是 | 平衡性、残差、共线性等假设检查 |
| 稳健性结果 | 是 | 安慰剂检验、敏感性分析、规格更换结果 |
| 样本量与效能 | 是 | 实际样本量与检验效能说明 |
| 局限与边界 | 是 | 已知假设、使结论翻转的条件 |
| 复现说明 | 是 | 环境、依赖、种子、数据版本、执行命令 |
| 评审记录 | 是 | 评级与修改建议的完整链条 |

## 执行步骤

1. **加载与校验分析计划**
   确认假设、结局、干预、识别策略与检验方法齐全；缺失则停止并上报。

2. **计划卡点确认**
   分析计划经人类分析师或负责人确认。

3. **数据剖析与样本构建**
   按排除标准构建分析样本；检查样本量、缺失模式、分布与分组平衡；报告排除造成的样本量变化。

4. **基线执行**
   按模板 notebook 填充参数，执行主规格；记录参数、种子、数据版本与运行时间。

5. **模型诊断**
   执行假设检查：残差、共线性、平衡性、比例风险假设等（按方法而定）；诊断不通过则停止并上报。

6. **安慰剂与阴性对照**
   用虚构干预时点或虚构结局检验方法是否会「发现」不存在的效应；报告全部结果，包括失败。

7. **敏感性分析**
   检验结论对未观测混杂与关键假设的稳健程度；给出结论翻转的边界。

8. **稳健性检查**
   更换规格、剔除异常、切换时间窗口；记录效应量变化；报告使结论不稳定的条件。

9. **评审与迭代**
   提交评审；按评审意见调整参数重跑；达到「完全满意」或迭代上限后终止；上限仍不满意则如实上报。

10. **交付与归档**
    输出效应量、区间、诊断、稳健性边界、局限与建议；归档代码、环境、数据版本、运行记录与评审链条。

## 质量标准（DoD）

严谨性：

- [ ] 假设在分析执行前已声明，非事后构造
- [ ] 识别策略已明确（随机实验 / 自然实验 / 目标试验仿真 / 显式不可识别）
- [ ] 效应量与不确定性区间已同时报告
- [ ] 多重比较已校正，或已声明未校正的理由
- [ ] 样本量与检验效能已说明
- [ ] 安慰剂检验或阴性对照已执行，结果已如实报告（含失败）
- [ ] 敏感性分析已执行，稳健性边界已给出

正确性：

- [ ] 数据版本已锁定，变量定义来自数据字典
- [ ] 随机种子已固定并记录
- [ ] 环境与依赖已锁定
- [ ] 模型诊断已执行并报告
- [ ] 全部数值来自执行结果，非模型生成
- [ ] 符号推导结果已数值验证

可复现性（按 ACM 四层）：

- [ ] 目标等级已声明（Repeatability / Replicability / Reproducibility / Reusability）
- [ ] 代码进入版本控制
- [ ] 最终结论由一次全新执行产生
- [ ] 工件完备：数据、代码、环境、运行记录齐全

治理：

- [ ] 分析计划经人确认（计划卡点）
- [ ] 未删除或隐藏不显著、负向或反直觉结果
- [ ] 未为求显著而更换规格（除已声明的稳健性检查）
- [ ] 涉及人类受试者的已确认通过伦理审查
- [ ] 评审记录已归档，未无限迭代

## 常见失败与处理

| 失败现象 | 根因 | 处理方式 |
|---|---|---|
| 效应量看起来很大但不可信 | 未控制混杂（如早期采用者偏差） | 回到识别策略；加入混杂控制；重跑并比较效应量变化 |
| 安慰剂检验「发现」了效应 | 方法或时点设定有问题 | 如实报告；检查干预定义与窗口；结论降级为部分验证 |
| 更换规格后结论翻转 | 结论不稳健 | 报告翻转边界；不得择一上报；结论降级 |
| 分组严重不平衡 | 随机化失败或观察性偏差 | 停止；报告诊断结果；改用匹配或加权并声明 |
| 样本量不足导致不显著 | 检验效能不足 | 报告效能分析；不得据此断言「无效应」，只能说「未检出」 |
| 符号推导结果代入数值后不成立 | 推导错误 | 降级为假设；改由数值方法求解；不写入结论 |
| 评审多轮仍不满意 | 方法或数据有结构性问题 | 达到上限后停止并上报，不无限迭代 |
| 结果与领域常识严重冲突 | 口径、单位或变量定义错误 | 标注部分验证；核对变量定义与单位；报告冲突点 |
| 环境与依赖变了结果就变 | 未锁定环境 | 锁定依赖与镜像；固定种子；重跑确认 |

## 示例

**任务**：评估新内容模块对用户两个月留存的影响（观察性数据）。

1. 加载分析计划：结局 = 两个月留存率；干预 = 接触新内容的天数；识别策略 = 目标试验仿真 + 倾向得分匹配；协变量 = 历史活跃度、注册时长、设备类型；检验方法 = 匹配后均值差 + 自助法区间。
2. 计划卡点：经数据分析负责人确认。
3. 数据剖析：构建分析样本；检查匹配前的组间平衡（历史活跃度差异显著，需匹配）。
4. 基线执行：执行主规格，得到效应量点估计与区间；记录参数、种子与数据版本。
5. 模型诊断：匹配后标准化均值差全部降至阈值以下；检查共同支撑区间。
6. 安慰剂检验：用干预开始前 60 天的虚构窗口重跑；**结果检出虚假效应 → 如实报告**。
7. 敏感性分析：计算使结论翻转所需的未观测混杂强度；给出边界值。
8. 稳健性检查：早期采用者（注册后 7 天内即接触）单独分析，发现效应显著变小 → 报告早期采用者偏差。
9. 评审与迭代：评审智能体给出 `satisfactory_with_caveats`，建议加入早期采用者分层；调整后重跑，评级提升；最终效应量约为朴素线性回归基准的 25%。
10. 交付归档：效应量 + 区间 + 稳健性边界 + 局限（安慰剂检验部分未通过、存在早期采用者偏差）+ 复现说明 + 评审链条。

4.3 Landing Checklist

4.3.1 Context Layer (L1)

• [ ] The analysis plan can be loaded and its completeness validated (assumptions, outcome, intervention, identification strategy, testing method)
• [ ] Variable definitions come from the data dictionary, not from model speculation
• [ ] Template notebooks and prior examples can be retrieved
• [ ] Data profiling results can be generated and injected automatically
• [ ] Failure records and prior non-significant results are not trimmed

4.3.2 Tool & Execution Layer (L2)

• [ ] Notebook kernels support state persistence across calls
• [ ] Statistical computing capability covers regression, time series, tests, Bayesian, and econometrics
• [ ] A symbolic computation engine is available, and derived results must be numerically validated
• [ ] The experiment tracking system records parameters, metrics, and artifacts
• [ ] A data versioning service can lock and reference dataset versions

4.3.3 Orchestration & Control Layer (L3)

• [ ] The design → execute → diagnose → robustness → review → iterate loop is implemented
• [ ] An iteration cap is set, and work stops and reports once it is reached
• [ ] The plan gate takes effect before execution
• [ ] Failed diagnostics can interrupt the process

4.3.4 Memory & State Layer (L4)

• [ ] Notebook state can be persisted and restored
• [ ] Experiment records include parameters, seeds, data versions, run time, and results
• [ ] Failure records are saved in a structured way and can be retrieved
• [ ] Artifact completeness is accepted against the ACM four-layer definition

4.3.5 Evaluation & Observation Layer (L5)

• [ ] External benchmarks are selected on demand (SciCode / SFE / DSBench / MLE-Bench / DiscoveryWorld, etc.)
• [ ] An internal regression set is established and covers key analyses
• [ ] Placebo tests and sensitivity analysis are mandatory steps, and results must be reported
• [ ] Conclusions are classified into the three tiers of "verified / partially verified / unverified"
• [ ] Both reviewer-agent and human review channels are available
• [ ] It is explicitly acknowledged that most tasks lack ground truth, without claiming "N% accuracy"

4.3.6 Governance & Safety Layer (L6)

• [ ] Research-integrity red lines are internalized as process gates (no fabrication, no selective reporting)
• [ ] Stating assumptions in advance and forbidding post-hoc hypothesis construction are auditable
• [ ] Analyses involving human subjects must confirm ethical review approval
• [ ] Handling of PII and sensitive data complies with classification requirements
• [ ] A human review gate exists before results are released

5. Summary

The core contradiction of the scientific computing direction is: it most needs rigor, yet most lacks ground truth.

The data analysis direction can intercept errors with result-set comparison, and the AI4S direction can constrain conclusions with physical laws and experimental verification, but the scientific computing direction — especially observational causal inference — often never knows the true effect. This means Harness cannot accept work by asking "is the answer right", and can only fall back to "is the process right": whether assumptions were stated in advance, whether the identification strategy was declared, whether diagnostics were run, whether the placebo test passed, whether sensitivity boundaries were given, and whether conclusions were classified into tiers.

This is precisely where the value of a Harness design for this direction lies: turning conclusions that cannot be verified into processes that can be verified.

Three main threads run through this direction:

1. Plan first, agent executes. The starting point is an analysis plan and template notebook provided by a human analyst, and the agent does heavy but constrained execution work. This is not distrust of the agent, but a direct response to the risk of "post-hoc hypothesis construction".
2. Review is a necessity, not a bonus. Public practice shows that a rigorous workflow's effect estimate can be only 25% of a naive method's — the difference comes from confounder control and bias identification, which precisely require an independent review perspective to be discovered.
3. Capability boundaries must be respected. Evaluations show no model exceeds 50 on equation-level symbolic manipulation and no model exceeds 30 on end-to-end scientific code. Until these capabilities mature, the correct approach is to hand symbolic derivation to specialized engines with numerical validation, and numerical computation to validated scientific computing libraries.

The last point is especially important: strength in general capability does not equal strength in scientific capability. Frontier models score close to 90 on general tasks and drop below 60 in rigorous scientific settings. This scissors gap reminds us that in the scientific computing direction, Harness's role is not to amplify model capability but to constrain model capability — keeping it within the process, within the method, and within reproducibility.

Information Gap Statement

1. There is no official or industry-recognized standard text for AGENTS.md / SKILL.md in the scientific computing direction. Sections 4.1 and 4.2 are both standard drafts proposed by this document.
2. The claim that Netflix oci-agent's "effect value is only 25% of the baseline" comes from second-hand relay of Netflix's technical blog, marked [To be verified], and the original text should be checked before citation. Likewise, details such as the "actor-critic loop", "reviewer agent's three-level grading", and "ACIC dataset evaluation" are also relayed.
3. The claim in the SciEval evaluation conclusions that "GPT 5.1 declined rather than improved" is a leaderboard-specific, time-point conclusion; citation must note the leaderboard time point and source; this document has noted it when citing.
4. A Chinese technical article claiming that after Meituan unified its A/B experiment platform, "the share of ineffective experiments fell by more than 70% and iteration cycles shortened by more than 50%" is second-hand relay with low credibility, and this document has not adopted the specific figures.
5. Figures from unsigned marketing articles such as "an e-commerce SaaS experiment's effectiveness rate rose from 41% to 89%" have been proactively excluded, and this document has not adopted any of them.
6. Anthropic's "Marlin" project (A/B testing code generation models in 2026) has no first-hand source, and this document has not adopted it.
7. Industry penetration rate, average ROI, and average implementation cycle of scientific computing agents — this search did not obtain reliable public data, and this document provides no such figures.
8. The CURE principles and BioFAIR come from the 2026 Nature Communications paper, a public source; this document has not cited its specific experimental data.
9. The variable names and parameters in the Section 4.2 example are illustrative constructions and do not represent any real system; plot points such as "placebo test partially failed" and "early-adopter bias" are used to illustrate the process shape.
10. Some entries in the benchmark table (e.g., ResearchClawBench's star count, Terminal-Bench Science's specific composition) come from survey-style lists, and individual details may change with version updates.

6. References

1. Shanghai AI Laboratory scientific intelligence evaluation system (SciEvalKit) release notes — Shanghai AI Laboratory. https://www.shlab.org.cn/news/5444233
2. SciEvalKit official repository — InternScience. https://github.com/InternScience/SciEvalKit
3. Awesome AI for Science (panorama list of scientific agent benchmarks) — GitHub. https://github.com/ai-boost/awesome-ai-for-science
4. Survey paper on scientific agents (arXiv 2503.16416) — arXiv. https://ar5iv.labs.arxiv.org/html/2503.16416
5. Paper on scientific agent evaluation (arXiv 2503.24047) — arXiv. https://arxiv.org/pdf/2503.24047.pdf
6. Netflix open-source causal inference agent workflow (oci-agent), related relay (to be verified) — Tencent News relaying the Netflix technical blog. https://new.qq.com/rain/a/20260828A05U8C00
7. Agents for R&D Science (autonomous scientific discovery and reviewer agent practice) — Orchestra Bio. https://orchestra.bio/blog/agents-for-r-d-science
8. ACM Artifact Evaluation terminology definitions — arXiv 1908.05986. https://arxiv.org/pdf/1908.05986.pdf
9. FAIR + CURE principles (Nature Communications 2026) — Nature. https://www.nature.com/articles/s41467-026-72667-3
10. Chinese Academy of Sciences scientific data management specification chain description — National Basic Disciplines Public Science Data Center. https://cms.casdc.cn/article/312
11. FAIR decade survey data and respondent quotes — Springer Nature. https://springernature.com/gp/librarians/the-link/open-science-blogpost/open-data-progress-challenges-institutions/52378334
12. MCP statistical computing and symbolic computing tool ecosystem (R / Wolfram / MATLAB) — technical guide. https://chatforest.com/guides/mcp-scientific-research-laboratory/
13. MCP for Data Scientists: Notebooks (Jupyter / Colab MCP) — MCPFind. https://mcpfind.org/blog/mcp-for-data-scientists-notebooks