Data · Data Engineering

1. Introduction

1.1 Background

Data engineering is one of the earliest directions in which AI Harness is deployed and one of the easiest to expose weaknesses. On one hand, AI-assisted pipeline-code authoring has become common practice — an industry report states that 84.2% of Airflow users write Airflow code with AI assistance (the figure comes from a secondhand retelling, marked [To be verified]); on the other hand, the same batch of reports presents a jarring set of contrasting numbers: only 9% of data engineers are satisfied with DAGs generated by general-purpose AI tools, with the top complaints being "hallucination rate" and "outdated syntax".

These numbers reveal not a model-capability problem but a context-and-verification problem: general-purpose models lack the project's DAGs, lack schema, lack execution-environment information, lack existing conventions, and also lack an evaluation layer that can stop errors before merge. This is precisely where AI Harness finds its use in the data engineering direction.

The governance baseline on the industry side is rising in parallel. The DAMA Guide to the Data Management Body of Knowledge (DAMA-DMBOK) defines 11 data-management knowledge areas, of which those most directly relevant to data engineering are data architecture, data modeling and design, data integration and interoperability, metadata management, data quality management, and reference data and master data. The "Data Management Capability Maturity Assessment Model" (GB/T 36073-2018) defines 8 capability domains and 5 maturity levels; DCMM 2.0 (GB/T 36073-2025), published in 2025 and effective on 2026-07-01, expands the capability domains to 9, adds the "Data Asset" capability domain, and increases capability items from 28 to 33 (some details pending verification, see the Information Gap Statement).

1.2 Definition

An AI Harness for the data engineering direction is the engineering carrier layer that, around stages such as data ingestion, cleaning, modeling, lineage, and quality gates, provides agents with context assembly, tool contracts, orchestration reuse, artifact memory, executable verification, and governance constraints.

It does not change the methodology of data engineering itself (dimensional modeling, ELT, declarative transformations, test-driven data quality) but instead enables agents to work within these existing methodologies rather than bypassing them.

Two points need clarification at the boundary:

• A data engineering Harness is not a "robot that automatically generates pipelines." Its outputs are reviewable changes (model code, test cases, documentation, lineage updates), and the final merge still goes through PR + CI.
• A data engineering Harness does not replace engines such as Airflow, dbt, and Spark. It takes on the layer of "letting agents correctly understand and use these engines."

1.3 Position within the AI Harness System

图 1-1｜数据工程六层能力模型：核心在 L1 上下文与 L6 治理

数据来源：基于本文分析绘制的示意图。

The data engineering direction's emphasis within the six-layer capability model is as follows (this emphasis analysis is based on public case facts, not a standard):

The core lies in L1 and L6.

L1 is core because the root cause of the "only 9% satisfied" data point is precisely missing context: the model does not know which models exist in the project, how they depend on each other, what the execution environment is, or what the existing naming and layering conventions are. Lineage and data quality provide trustworthy corpus for the context — without lineage, the agent cannot judge whom a change would affect; without quality gates, the agent cannot judge whether its own change is safe.

L6 is core because data engineering is the only direction where "a wrong output directly contaminates all downstream consumers." A miscalculated pipeline quietly feeds wrong numbers to reports, models, and financial systems without raising any error. Data classification and masking must therefore be hard constraints enforced by the Harness layer's permission system, not by the model's self-restraint.

Bottleneck layer: L1. The vast majority of current failures occur during context assembly — the model cannot access schema, cannot access lineage, cannot access execution-environment assumptions, so it generates code that is syntactically valid but semantically wrong. By comparison, L5's tools (Great Expectations, dbt test) are already fairly mature; what is missing is wiring them into a closed loop with the agent workflow.

1.4 Value and Bottlenecks

The value is reflected in three places:

• Lowering the cost of getting started: pressing the cost of "writing the first version" close to zero, so engineers can focus their effort on design and review.
• Turning verbal conventions into executable assertions: things previously kept in people's memory — lineage, grain, uniqueness, freshness — become gates the agent must check every time.
• Shortening fault-diagnosis time: when a downstream failure occurs, the agent can automatically trace back along lineage to the root-cause model, instead of manual layer-by-layer triage.

The bottlenecks likewise number three:

• Silent errors: see the case in Section 3.3; routine checks (row counts, freshness, non-null) cannot catch grain and semantics errors.
• Context scale: the number of tables and columns in an enterprise warehouse far exceeds the context budget, so retrieval-based schema linking is mandatory.
• API drift: data engineering tools differ greatly across versions (e.g., Airflow from classic operators to the TaskFlow @task decorator), and model training data often lags the current version, giving rise to "outdated syntax"-type complaints.

2. Glossary

3. Case Studies

3.1 Case 1: The Gap Between Expectations and Reality When an Agent Writes Airflow DAGs

3.1.1 Background

Data engineering is one of the areas where AI-assisted coding has penetrated fastest. According to retold industry reports: 84.2% of Airflow users already write Airflow code with AI assistance; 32% of organizations already have GenAI/MLOps use cases in production, up 5 percentage points year over year; 72% of data teams list AI-assisted coding as a priority; data trust rose from 66% to 83% (all of these figures come from secondhand retellings sourced from the annual reports of dbt Labs and Astronomer, all marked [To be verified]).

But the same batch of reports offers one strongly counterintuitive figure: only 9% of data engineers are satisfied with DAGs generated by general-purpose AI tools, with hallucination rate and outdated syntax being the top complaints.

3.1.2 Approach

A root-cause analysis of this gap points to three intervenable points, which correspond exactly to three of the Harness layers:

• L1 missing context: the model does not understand the project's existing models, dependencies, and naming conventions. The countermeasure is to assemble the project DAG, schema, and existing examples into the context rather than relying on the model's general knowledge.
• L2 unknown execution environment: the model does not know whether the target environment uses KubernetesPodOperator or DockerOperator, whether the XCom backend is custom, or which API style the Airflow version supports. The countermeasure is to make environmental assumptions explicit and expose them through tool contracts.
• API drift: the Airflow operator API changes significantly across versions (classic operators → the TaskFlow @task decorator), and general-purpose models have insufficient weight in their training data for this. The countermeasure is to use versioned API documentation and in-project examples as retrieval corpus, and to run version-consistency checks on the generated results.

Meanwhile, the product evolution on the dbt side provides a comparison sample: dbt Copilot generates SQL models, tests, and documentation based on the project's DAG, schema definitions, and existing model conventions; the dbt Fusion engine, rewritten in Rust and natively understanding SQL, parses roughly 30× faster, in part so that models can fetch rich project metadata in real time (this speedup is self-reported by the vendor, marked [To be verified]).

3.1.3 Results

These numbers themselves are the evidence of the outcome: the high penetration of AI-assisted coding (84.2%) coexists with low satisfaction (9%), showing that the value lies not in "whether the model can write" but in "whether the Harness provides the model with the right context and verification."

Trust indicators on the industry side point the way to improvement: data trust rose from 66% to 83% (to be verified), which aligns in timing with the practice of "wiring lineage, quality checks, and change governance into a closed loop." The inferable conclusion is: the value curve of a data engineering agent is determined by context completeness, not by model capability alone.

3.2 Case 2: A Lineage Closed-Loop Triage Driven by the dbt MCP Server

3.2.1 Background

The typical way data-platform incidents are handled is manual: an anomaly appears on a downstream dashboard → someone asks in the group chat → a data engineer logs into the scheduler → traces upward layer by layer → locates the failed model → runs tests → determines the blast radius. This process is time-consuming and highly dependent on personal experience.

In parallel, MCP is becoming the standard binding layer for data tools. The dbt MCP Server exposes a dbt project as a queryable API: listing models, reading lineage, triggering dbt runs by selector, and fetching test results.

3.2.2 Approach

Wire the MCP tool contracts together with lineage and quality checks into an automatic closed loop:

1. Trigger: a downstream DAG failure or abnormal dashboard-data event enters the triage queue.
2. Trace back: the agent reads lineage through the dbt MCP Server and traces up the dependency chain to locate the failed or anomalous model.
3. Targeted verification: run a targeted dbt test on the located model rather than rerunning the whole project, to control cost.
4. Impact analysis: derive downstream, along lineage, the affected data products and consumers.
5. Triage output: generate a comment in the collaboration tool containing the root-cause model, the failed tests, the blast radius, and the suggested fix direction.
6. Human in the loop: changes still require PR + CI; the agent only produces the triage conclusion and fix suggestions.

Parallel productized capabilities also include: the dbt Semantic Layer centrally defines metrics / dimensions / grain in YAML; the dbt Developer Agent (2026-05-06 preview) supports describing changes in natural language and builds, refactors, tests, and documents models based on lineage, the semantic layer, and governance, with every change auditable.

3.2.3 Results

This closed loop turns incident triage from "experience-dependent manual investigation" into "deterministic trace-back along lineage + targeted verification." Its value lies not in a one-time speedup, but in three points:

• Blast radius is computable: only with lineage can one answer "whom does this change affect," which is a prerequisite for change approval.
• Verification cost is controllable: targeted tests replace a full rerun, directly reducing contamination and cost.
• Auditable: every change by the dbt Developer Agent is auditable, satisfying the governance side's question of "what did the AI change."

It should be noted: specific benefit figures for this loop (fault-diagnosis time, cost savings) did not yield reliable public data in this search; only the mechanism is described here, with no quantified conclusions.

3.3 Case 3: Silent Failure — the Pipeline Does Not Crash, It Just Lies Fluently

3.3.1 Background

There is a widely cited description of a failure mode for data engineering agents: a crashing pipeline tells you it is broken; a miscalculated pipeline tells you nothing.

A publicly reported concrete scenario: while an agent refactored a dbt staging model, it did a left join on the promotions table. That table has multiple rows per order (one row per promo code), so the join caused a grain fan-out and sum(order_total) was double-counted. Throughout, all routine checks passed — tests passed, freshness was normal, row counts were normal. It was not discovered by Finance until 21 days later.

This case comes from a public narrative blog, serving as a failure-mode example; it has no statistical representativeness and must not be converted into an incidence rate. But it depicts a class of real and costly engineering risk.

3.3.2 Approach

For this kind of silent error, the defense that should be added on the Harness side is not "more row-count checks" but semantic-layer assertions:

At the same time, data quality engineering itself should establish four types of metrics: coverage, freshness of lineage edges, MTTD (mean time to detect), and MTTR (mean time to repair).

3.3.3 Results

The direct outcome of this case is that it exposes a kind of structural blind spot: routine data quality checks (non-null, uniqueness, row counts, freshness) are completely ineffective against semantics errors.

Two executable conclusions follow:

1. Grain and uniqueness assertions must be placed before the join, rather than discovering after aggregation that the numbers are wrong.
2. Key metrics must have an independent recomputation path. Self-validation within the same path cannot detect systematic bias; only a recomputation that does not share upstream can expose the problem.

This conclusion also explains why the data engineering direction depends most heavily on L1 + L6: lineage (L1 corpus) answers "whom does the change affect," while classification and approval (L6) answer "who is authorized to change it."

4. Practice Standards

4.1 AGENTS.md Specification

Standard-source statement: the following is this article's proposed draft standard for an AGENTS.md in the data engineering direction. As of now, no original AGENTS.md specification for the data science / data engineering direction has been published by an official body, industry association, or standards organization; AGENTS.md is a community convention rather than a standard. This draft inherits all clauses of the data-science group-level AGENTS.md and tightens and extends them for the data engineering direction.

# AGENTS.md —— 数据工程（Data Engineering）

> 继承数据科学组级 AGENTS.md 全部条款。本节为数据工程方向的收紧与扩展。
> 本文件为标准建议稿，业界尚无官方标准版本。

## 角色与边界

- 本 Agent 是**数据管道的协作者**：可以读元数据、写代码、跑测试、做分诊分析。
- 不可以直接执行的操作：生产环境 DDL、直接写入生产表、绕过 CI 合并、修改数据分级、删除历史分区。
- 所有变更的合法出口只有一条：**分支 → PR → CI → 人工评审 → 合并**。
- 指标口径的最终裁定权归业务所有权人，不归本 Agent。

## 环境假设

- 编排器：Apache Airflow（须明确版本与 API 风格：经典 operator 还是 TaskFlow）
- 转换层：dbt（须明确版本、adapter 与目标数仓方言）
- 计算引擎：Spark / 数仓原生 SQL（须明确方言：BigQuery / Snowflake / PostgreSQL / DuckDB 等）
- 质量框架：Great Expectations 或 Soda（须明确断言存放位置与执行入口）
- 元数据 / 血缘：OpenLineage 兼容后端（Marquez / DataHub / OpenMetadata / Atlas）
- **以上任一项未知时，先问，不要猜**。版本未知时禁止生成依赖版本特性的代码。

## 上下文加载顺序（Context Budget）

1. 任务与验收口径（必须）
2. 目标模型的 schema、grain 声明与既有测试（必须）
3. 血缘：上游依赖 + 下游消费者（**写操作前必须**）
4. 项目 DAG 与分层约定（staging / intermediate / marts 等）
5. 语义层中的指标与维度定义（涉及指标时必须）
6. 同类既有模型代码（命名、物化策略、增量策略的范例）
7. 数据剖析结果：行数、空值率、基数、主键唯一性
8. 执行环境声明：operator 类型、XCom 后端、资源与配额
9. 数据分级与 PII 字段清单（必须，决定脱敏与可见性）

- 表数量多时走**检索式 schema linking**，禁止把整个 warehouse 的 schema 一次性注入。

## 工具契约

| 工具 | 用途 | 模式 | 约束 |
|---|---|---|---|
| dbt MCP Server | 列模型、读血缘、按 selector 触发 run、取测试结果 | 读 + 受限写 | 写操作限定在指定 selector 与 dev target |
| 元数据服务 | 查询表/列血缘、所有权、分级标记 | 只读 | 不得修改分级 |
| 数仓连接 | 执行查询、剖析数据、干跑 EXPLAIN | 只读 | 默认只读角色；成本预估超阈值须中止 |
| 质量框架 | 执行断言集 | 读 + 执行 | 不得静默禁用失败断言 |
| 编排器 | 查看 DAG 与运行历史、触发重跑 | 读 + 受限写 | 不得修改生产 DAG 定义 |
| 版本控制 | 建分支、提 PR、读 CI 结果 | 受限写 | 禁止直接推送到主分支 |

- 工具参数必须做模式校验；参数格式错误在调用前拦截，**不得靠重试碰运气**。
- 每个工具调用记录时间、参数摘要、返回摘要、耗时与成本。

## 数据源契约

- 每个输入表必须有：唯一标识、负责人、刷新频率、grain 声明、分级标记。
- 数据引用必须带版本：快照时间、分区、commit hash 或数据集版本号，四者至少其一。
- **join 前必须校验关联键唯一性**；不唯一时先聚合或改用其他关联策略，禁止直接 join。
- 缺失值与异常值处理方式必须显式声明，禁止静默丢弃。
- 跨源关联前确认主键与粒度；粒度不匹配禁止 join。
- 临时数据源必须标注为临时，并在产出物中声明。

## 任务执行流程（SOP）

1. 澄清：复述任务，确认目标模型、grain 与验收口径。
2. 盘点：列出将读写的表、依赖的上游、受影响的下游、所需权限。
3. 影响分析：通过血缘计算 blast radius；影响关键数据产品时先升级。
4. 设计：给出模型结构、物化与增量策略、测试清单、回填方案。
5. 计划卡点：方案经人确认后进入执行。
6. 探查：先剖析数据（行数、空值、基数、唯一性），再动手写模型。
7. 实现：写代码 + 测试 + 文档，遵循项目既有命名与分层约定。
8. 验证：在 dev target 执行 dbt build；执行质量断言；执行语义回归集。
9. 提交：建分支、提 PR，附影响分析与验证结果。
10. 交付：合并后观察首轮运行结果与下游新鲜度；归档变更记录。

## 验证与证据要求

- 每一条数值结论必须来自**执行结果**，禁止由模型凭记忆生成。
- 必须执行的四类断言：
  - **grain 断言**：声明并断言主键唯一，捕获扇出与重复计数
  - **关联断言**：join 键在右表唯一的检查前置到 join 之前
  - **新鲜度与行数断言**：监控但不作为正确性的唯一依据
  - **语义回归集**：关键指标的已验证查询，每次变更回归
- **关键指标必须有独立复算路径**：用不共享上游的方式重算，比对差值并说明差异原因。
- 主动报告反例：若验证中发现与预期不符的数据，必须报告，不得只呈现支持性结果。
- 结论分「已验证 / 部分验证 / 未验证」三档列出。

## 失败与升级策略

- 同类失败重试不超过 2 次；第 3 次改变策略或升级。
- 工具参数格式错误：停止重试，检查模式定义并修正调用构造。
- 血缘缺失导致无法评估影响范围：**停止写操作**，先补血缘或走人工评估。
- 回填规模超预期：暂停，报告行数、成本与预计时长，获批后继续。
- 质量断言失败：不得禁用断言或放宽阈值以求得通过；报告失败项与根因。
- 出现 PII 或分级不明确的字段：停止处理，升级给数据负责人。
- 升级时携带：任务、已完成步骤、失败点、已尝试方案、blast radius、建议下一步。

## 安全与合规红线

- 不得绕过 CI、审批、权限与配额。
- 不得修改或降低数据分级；不得将高分级数据写入低分级目标。
- 不得在无人评审的情况下执行生产写操作或 DDL。
- 不得在生成物（日志、缓存、PR 描述、图表）中泄漏敏感字段值。
- 不得为求通过而禁用、删除或放宽质量断言。
- 不得伪造数据来源、血缘关系或测试结果。
- 涉及个人信息、跨境传输的管道变更，须确认已完成相应审批。

## 禁止事项

- 禁止在未知 Airflow 版本与 API 风格时生成 DAG 代码。
- 禁止在未确认 grain 的情况下做 join 或聚合。
- 禁止把整个 warehouse schema 一次性注入上下文。
- 禁止生成不可回滚的批量变更；回填必须可分批、可中止。
- 禁止用「大概」「通常」替代具体的行数、成本与时间数字。
- 禁止把未执行的推测当作验证结果报告。
- 禁止跨方向复制通用分析模板；数据工程的断言体系与科学计算、HPC 有实质差异。

## 输出格式

- 结论先行 → 证据（数据源 + 版本 + 代码 + 执行记录）→ 不确定性 → 建议。
- 变更产出统一包含：模型代码、测试、文档、影响分析、回填方案、验证结果。
- 数值带单位；范围用「~」连接；百分比数值与 % 之间无空格。
- 代码与配置使用带语言标记的代码块。
- 血缘与影响范围用表格列出：下游对象、负责人、影响类型。

## 评估与自检

- [ ] 目标模型的 grain 已声明并断言唯一
- [ ] 所有 join 的关联键唯一性已前置校验
- [ ] 上游依赖与下游影响已通过血缘列出
- [ ] 数据分级已确认，PII 字段已按策略处理
- [ ] Airflow 版本与 API 风格已确认，生成代码与之匹配
- [ ] dbt build 在 dev target 通过，测试无跳过
- [ ] 语义回归集已执行，关键指标与基准一致
- [ ] 关键指标已有独立复算，差异已说明
- [ ] 回填方案可分批、可中止，成本已预估
- [ ] 变更已走 PR + CI，人工评审记录已留痕

4.2 SKILL.md Specification

Standard-source statement: the following is this article's proposed draft standard for a SKILL.md in the data engineering direction, for which no official standard original likewise exists. Its data-quality section is based on the ISO 8000 series; its process section is engineering advice.

---
name: data-pipeline-change
description: 数据管道变更的标准执行流程与验收基线。适用于新建或重构 dbt 模型、编写或修改 Airflow DAG、执行回填、处理管道故障分诊等任务。触发场景：任何会改变数据管道产出内容的智能体任务。
version: 1.0
created: 2026-09-12
---

# 数据管道变更标准流程

## 适用场景

- 新建、重构或删除 dbt 模型；调整物化与增量策略。
- 编写或修改 Airflow DAG；调整调度、依赖与重试策略。
- 执行历史数据回填。
- 管道故障的分诊与根因定位。
- 不适用场景：一次性即席查询、不涉及管道变更的只读分析。

## 前置条件

- 目标环境已确认：Airflow 版本与 API 风格、dbt 版本与 adapter、数仓方言。
- 目标模型的 grain 已声明；上游表的 grain 已知。
- 血缘可查；若血缘缺失，先补血缘或走人工影响评估。
- 数据分级与 PII 字段清单可用。
- 质量框架可用且断言集可执行。
- 回填成本与时长可预估；存在预算上限。

## 输入

| 输入项 | 必需 | 说明 |
|---|---|---|
| 变更需求与验收口径 | 是 | 业务目标、目标模型、期望口径 |
| 目标模型 grain 声明 | 是 | 一行代表什么业务实体 |
| 上游表清单与 grain | 是 | 含关联键与唯一性判断 |
| 影响范围约束 | 是 | 可影响的下游、禁止影响的关键资产 |
| 成本与时间预算 | 是 | 回填规模上限、允许的执行窗口 |
| 既有范例 | 否 | 项目内同类模型代码与测试 |

## 输出

| 输出项 | 必需 | 说明 |
|---|---|---|
| 变更代码 | 是 | 模型 / DAG 代码，遵循项目约定 |
| 测试与断言 | 是 | grain、关联、唯一性、新鲜度、语义回归 |
| 文档 | 是 | 模型说明、字段口径、依赖关系 |
| 影响分析 | 是 | 上游依赖 + 下游消费者 + blast radius |
| 验证结果 | 是 | dbt build 结果、断言结果、独立复算差值 |
| 回填方案 | 否 | 分批策略、中止条件、成本预估 |

## 执行步骤

1. **确认环境与版本**
   读取 Airflow 版本与 API 风格、dbt 版本与 adapter、数仓方言；任一未知则先询问，禁止猜测。

2. **声明 grain 与关系**
   声明目标模型 grain；列出上游表及其 grain；标注所有 join 键并在探查中验证其唯一性。

3. **血缘与影响分析**
   通过元数据服务读取上下游血缘；列出下游消费者与负责人；影响关键数据产品时先升级并等待确认。

4. **数据探查**
   在只读角色下执行剖析：行数、空值率、基数、主键唯一性、分区分布；对 join 键执行唯一性检查并记录结果。

5. **设计变更方案**
   给出模型结构、物化策略、增量策略、测试清单、回填方案与回滚方案。

6. **计划卡点确认**
   方案与影响分析提交人工确认；未确认不进入实现。

7. **实现与本地验证**
   在 dev target 编写模型、测试与文档；执行 `dbt build`；修复失败项，**不得通过禁用断言求通过**。

8. **语义回归与独立复算**
   执行关键指标的语义回归集；用不共享上游的路径独立复算关键指标，比对并说明差值。

9. **提交与评审**
   建分支、提 PR，附影响分析、验证结果、回填方案；等待 CI 与人工评审。

10. **上线观察与归档**
    合并后观察首轮运行与下游新鲜度；记录变更、成本与遗留问题；更新血缘与文档。

## 质量标准（DoD）

数据质量（依据 ISO 8000-8:2015 三维度与 8000-120/130/140）：

- [ ] 句法质量：字段类型、编码与结构符合声明规范，已自动校验
- [ ] 语义质量：字段含义与语义层 / 数据字典一致，已对照校验
- [ ] 语用质量：产出可被下游按预期使用，已与至少一名下游消费者确认
- [ ] 溯源：每个字段可追溯到上游来源与加工过程
- [ ] 准确性：关键指标经独立复算，差值已说明
- [ ] 完整性：覆盖范围与缺失情况已声明，无静默丢弃

管道与变更：

- [ ] grain 已声明且主键唯一断言通过
- [ ] 所有 join 键唯一性已前置校验
- [ ] dbt build 在 dev target 通过，无跳过的测试
- [ ] 语义回归集全通过
- [ ] 回填可分批、可中止，成本已预估且在预算内
- [ ] 回滚方案可执行
- [ ] 文档与血缘已同步更新

治理：

- [ ] 数据分级已确认，PII 按策略脱敏
- [ ] 变更走 PR + CI，人工评审已留痕
- [ ] 影响分析已列出全部下游消费者与负责人
- [ ] 未在生成物中泄漏敏感字段值

## 常见失败与处理

| 失败现象 | 根因 | 处理方式 |
|---|---|---|
| 生成的 DAG 报「没有这个模块 / 参数」 | 版本与 API 风格未知 | 先确认 Airflow 版本；按项目既有范例改写；禁止猜测 API |
| 聚合值偏大 | join 键不唯一导致扇出 | 回到探查步骤校验唯一性；先聚合右表再 join；加 grain 断言 |
| 增量模型漏数 | 增量键与迟到数据策略不当 | 检查增量键与 lookback 窗口；改用可重放的分区覆盖策略 |
| 回填跑了很久还没完 | 未分批、未预估 | 中止；改为按分区分批，设置每批上限与检查点 |
| 断言失败但业务说没问题 | 断言定义过时或口径已变 | 走口径变更流程，由所有权人确认后更新断言，**不得直接禁用** |
| 血缘查不到受影响的下游 | 血缘采集不全 | 停止写操作；改用人工影响评估或先补血缘 |
| 看到不该看的字段 | 权限未生效或未脱敏 | 立即停止并记录；升级给数据负责人，不得继续处理 |
| 成本超预算 | 未做干跑与成本预估 | 执行 EXPLAIN 干跑；设置扫描量上限；超限即中止 |

## 示例

**任务**：为「订单履约时效」新增一个 mart 层指标模型。

1. 确认环境：Airflow 2.x + TaskFlow 风格，dbt 1.x + Snowflake adapter，数仓方言 Snowflake。
2. 声明 grain：目标模型为「每订单一行」（`order_id` 唯一）；上游 `orders` 为每订单一行，`promotions` 为每订单每促销码一行（**不唯一**）。
3. 血缘分析：上游为 `stg_orders`、`stg_promotions`；下游为履约看板与财务日报（负责人已列）。
4. 数据探查：`orders` 主键唯一性通过；`promotions.order_id` 基数大于行数，确认不唯一。
5. 方案设计：`promotions` 先按 `order_id` 聚合为一行再关联；目标模型物化为 table；增量策略按 `order_date` 分区覆盖；测试包含 `unique(order_id)`、`not_null(order_id)`、时效值域断言。
6. 计划卡点：影响分析显示财务日报使用本模型，提交财务侧确认口径。
7. 实现与验证：dev target 执行 `dbt build`，全部测试通过。
8. 语义回归：执行履约时效的已验证查询集；用不经本模型的路径独立复算 P50 时效，差值 0。
9. 提交：PR 附影响分析、验证结果、回填方案（按 30 天分批，可中止）。
10. 上线观察：合并后观察首轮运行与财务日报新鲜度；归档变更记录。

4.3 Deployable Checklist

4.3.1 Context Layer (L1)

• [ ] The project DAG and layering conventions can be retrieved by the agent
• [ ] Schema supports retrieval-based linking, without relying on full injection
• [ ] Every model has a grain declaration, and the declaration enters the context
• [ ] Semantics definitions in the semantic layer / metrics registry can be retrieved
• [ ] Existing model code can be retrieved as examples
• [ ] Data profiling results (row counts, null rates, cardinality, uniqueness) can be generated and injected automatically

4.3.2 Tools and Execution Layer (L2)

• [ ] The dbt MCP Server is connected, and write operations are limited to the dev target and specified selectors
• [ ] The warehouse connection defaults to a read-only role, and write operations require explicit authorization
• [ ] Tool parameters have schema validation, and format errors are blocked before the call
• [ ] All tool calls record latency, cost, and a return summary
• [ ] Orchestrator write permissions are restricted, prohibiting modification of production DAG definitions

4.3.3 Orchestration and Control Layer (L3)

• [ ] Native Airflow / dbt DAGs are reused, without rebuilding the orchestration layer
• [ ] Backfill jobs can run in batches, can be aborted, and carry checkpoints
• [ ] The change process is fixed as branch → PR → CI → review → merge
• [ ] The retry policy is explicit, with no more than 2 retries for the same type of failure

4.3.4 Memory and State Layer (L4)

• [ ] Data snapshots and model versions are traceable
• [ ] Run history and backfill records are queryable
• [ ] Change records are archived, including impact analysis and verification results
• [ ] Lineage and documentation are updated in sync with changes

4.3.5 Evaluation and Observability Layer (L5)

• [ ] Grain and uniqueness assertions are placed before the join
• [ ] A semantic regression set is established and regressed on every change
• [ ] Key metrics have an independent recomputation path
• [ ] Failed quality assertions must not be disabled or relaxed to force a pass
• [ ] The four metrics — lineage coverage, edge freshness, MTTD, MTTR — are live

4.3.6 Governance and Security Layer (L6)

• [ ] Pipeline-level PII tagging and data classification are established
• [ ] RLS / CLS are enforced at the data access layer
• [ ] Sensitive values do not appear in logs, caches, or PR descriptions
• [ ] Cost guardrails are set (scan-volume limits, core-hour quotas)
• [ ] Metric-semantics changes require owner confirmation and leave an audit trail

5. Summary

The core demand of the data engineering direction on AI Harness can be summarized in one sentence: let agents work within the existing methodology rather than bypassing it.

The greatest temptation in this direction is "automatically generating pipelines" — and the greatest risk lies precisely there. A pipeline generated by an agent that lacks grain assertions, lacks join-key uniqueness checks, lacks lineage impact analysis, and lacks independent recomputation will run in a seemingly perfectly healthy state, then quietly feed wrong numbers to all downstream consumers. The publicly reported case of "discovered by Finance 21 days later" was not about the model being unintelligent, but about a missing verification layer.

That is why this direction's focus falls on L1 and L6:

• L1 provides trustworthy corpus so the agent knows what exists in the project, how the parts depend on one another, and what the existing conventions are. The "only 9% satisfied" gap is, in essence, a context gap.
• L6 provides hard constraints so that classification, masking, approval, and cost guardrails do not depend on the model's self-restraint.

Data engineering also has an advantage rarely found in other directions: it has the richest executable ground truth. SQL can be compared, tests can be run, lineage can be traced, and metrics can be regressed. This means in this direction every bit of the Harness's investment can be directly verified — as long as one is willing to make verification a gate rather than a document.

Information Gap Statement

1. No official or industry-recognized standard original of AGENTS.md / SKILL.md exists for the data engineering direction. Sections 4.1 and 4.2 are both proposed draft standards from this article.
2. The following figures come from secondhand retellings and were not verified in the primary official reports; they are marked [To be verified]: 84.2% of Airflow users write code with AI assistance; 32% of organizations have GenAI/MLOps use cases in production; only 9% of data engineers are satisfied with general-purpose AI-generated DAGs; 72% of data teams prioritize AI-assisted coding; data trust 66% → 83%; dbt Fusion parsing roughly 30× faster. Verify against the official reports — Astronomer's State of Airflow 2026 and dbt Labs' 2026 State of Analytics Engineering — before citing.
3. Databricks Genie Code's internal benchmark text-to-SQL accuracy rising from 32% to 90%+ is vendor-internal self-reported data and is not adopted in this document; Snowflake Cortex Code "used by more than half of customers" is unverified and not adopted in this document.
4. dCMM 2.0's "security weight 15% → 25%" and "AI governance mandatory at level 4+" come from secondhand retellings and are not adopted in this document; "capability domains 8 → 9, addition of data assets, effective 2026-07-01" is a retelling consistent across multiple sources and is adopted.
5. The grain fan-out case in Section 3.3 is a failure-mode example from a public narrative blog and has no statistical representativeness; it must not be converted into an incidence rate; the "21 days" therein is a description of an individual case.
6. Industry penetration, average ROI, and average implementation cycle of data engineering agents — this search did not yield reliable public data, so this document gives no such figures.
7. The specific clause text of national standards related to scientific-data submission and data quality was not obtained; this document cites only the numbers and the publicly disclosed process frameworks.
8. The table names, field names, and parameters in the Section 4.2 example are all illustrative constructions and do not represent any real system.

6. References

1. AI for Data Engineering 2026 (including retold data from Astronomer's State of Airflow 2026 and dbt Labs' 2026 State of Analytics Engineering, to be verified) — technical blog. https://baeseokjae.github.io/posts/ai-for-data-engineering-2026
2. The Agent-Written Data Pipeline (including the grain fan-out failure-mode case, narrative source) — technical blog. https://theaivibe.org/blog/the-agent-written-data-pipeline
3. DAMA Guide to the Data Management Body of Knowledge (DAMA-DMBOK) knowledge-area framework — DAMA International. https://www.longshidata.com/blog/c/c2026062301.html
4. "Data Management Capability Maturity Assessment Model" (GB/T 36073-2018) and DCMM 2.0 (GB/T 36073-2025) explanations — China National Standards. https://ima.qq.com/wiki/?shareId=0258e6e62f0b37168d979fc9920268b9e4a313954973dbcf7e0469f132358841
5. Summary of the ISO 8000 data quality standard series — ISO/TC 184/SC 4. https://quality.arc42.org/standards/iso-8000
6. OpenLineage official specification documentation — Linux Foundation. https://openlineage.io/
7. Engineering methods and metrics for data lineage deployment (coverage, edge freshness, MTTD, MTTR) — Customer Science. https://customerscience.com.au/customer-experience-2/implementing-lineage-tracking-step-by-step/
8. Great Expectations / Soda and OpenLineage integration (quality scores entering the lineage graph) — Conduktor. http://conduktor.io/glossary/data-lineage-tracking-data-from-source-to-consumption
9. MCP data science workflow tool ecosystem (including the dbt MCP Server) — technical guide. https://chatforest.com/guides/mcp-data-science-workflows
10. Data Agent architecture survey: from architecture to practice — technical analysis blog. https://ibillxia.github.io/blog/2026/08/20/data-agent-servey-from-arch-to-practice
11. W3C PROV concept model overview — W3C. https://www.w3.org/TR/prov-overview/
12. 2025 survey of data-governance maturity of Chinese enterprises (average score below 3 on every dimension, to be verified) — industry survey retelling. https://baeseokjae.github.io/posts/ai-for-data-engineering-2026