Analytics · 数据分析

1. Introduction

1.1 Background

"Asking data in natural language" is the capability that agents are the easiest to demonstrate — and the easiest to get wrong. There is a widely cited observation in the industry: when a retrieval system fails, it returns nothing and you notice immediately; when a Q&A system fails, it returns a number — correctly formatted, accompanied by a chart.

This sentence captures the fundamental risk of the data analytics direction: failure is graceful and silent.

From the supply side, around 2026 the three major data platforms all independently made the "semantic layer" a prerequisite for trustworthy AI Q&A:

Three different technical routes converging on the same conclusion is itself the best annotation of this direction's core proposition: the accuracy problem of Q&A is, in essence, a context-engineering problem.

1.2 Definition

The AI Harness for the data analytics direction is the engineering carrier layer that, around metric definitions, self-service analytics, and business-intelligence Q&A, provides agents with semantic-context assembly, controlled query execution, metric-drift detection, access-leakage protection, and cost guardrails.

Its output is not "a runnable piece of SQL" but an answer that is consistent with the organization's metric definitions, permission-compliant, cost-controlled, and verifiable by regression.

Boundary clarification:

• The data analytics Harness is not the same as "connecting a database to a large model." A raw connection introduces the triple risk of metric definitions, permissions, and cost all at once.
• The data analytics Harness and data engineering (01-data) have a clear division of labor: data engineering is responsible for turning data into trustworthy assets, and data analytics is responsible for ensuring trustworthy assets are correctly consumed. The semantic layer is the interface between the two.

1.3 Position in the AI Harness Architecture

图 1-1｜数据分析 AI Harness 六层定位：核心在 L1 与 L5

数据来源：基于本文分析绘制的示意图。

The core lies in L1 and L5.

L1 is core because this direction's input is not documentation but an extremely large schema. Enterprise databases often contain more than 1,000 columns, making full injection infeasible; the correct approach is to treat schema linking as a retrieval problem — first retrieve the relevant tables and columns, then assemble the context. The value of the semantic layer is precisely that it turns knowledge that was scattered — "which tables and columns are authoritative" and "how this metric should be computed" — into context that can be retrieved, governed, and sustainably updated.

L5 is core because this direction happens to have the most ideal evaluation conditions: executable ground truth. SQL result sets can be compared directly, verified queries can form a regression set, and execution plans can be evaluated for efficiency. This makes the direction one of the highest in Harness in terms of "return on evaluation investment".

Bottleneck layer: L1. In production, the majority of Q&A failures are not because the model cannot write SQL, but because the model got the wrong tables, the wrong metric definitions, or simply has no idea whether the organization's "revenue" means recognized revenue or order amount.

1.4 Value and Bottlenecks

Value:

• Turn metric definitions from verbal agreements into executable assets: once the semantic layer is established, metric definitions no longer depend on word of mouth.
• Lower the barrier to self-service analytics: business users neither need to master SQL nor wait for the data team's schedule.
• Evaluation can be closed-loop: executable ground truth makes regression testing possible — a condition other directions envy.

Bottlenecks:

• The enterprise cliff: high accuracy on leaderboards cannot transfer to production environments (see 3.2).
• Metric drift: the same metric yields different numbers in different reports and at different times, and usually no one notices.
• Access leakage: natural language bypasses the permission encapsulation already present in the dashboard layer and can expose rows and columns a user should not see.
• Cost explosion: a single inefficient query can full-scan away an entire month's budget.

2. Glossary

3. Case Studies

3.1 Case 1: Three Platforms, One Destination — The Semantic Layer Is the Prerequisite for Trustworthy Q&A

3.1.1 Background

Early "AI Q&A" generally followed the raw-connection route: stuffing the schema into the prompt and letting the model write SQL directly. This approach performed well in demos but failed frequently in production. Its failure modes are grouped into five categories: schema hallucination (fabricating columns or tables), join error (joining at the wrong granularity), metric drift (computing business numbers under a non-approved definition), access leakage (seeing data one should not see), and cost explosion (a full table scan from an inefficient query).

The product evolution of the three platforms around 2026 can be read as a collective response to these five failure modes.

3.1.2 Solution

The commonality across the three is worth emphasizing: all treat "govern the context first, then handle Q&A" as a prerequisite. Microsoft Fabric's approach is especially typical — it secures permissions and organization-level policies first, and only then opens the natural-language entry point.

The new capabilities in 2026 also continue this direction: Databricks's Genie One can step beyond a single Genie Space, connect to enterprise knowledge sources, unify structured and unstructured data, and automatically learn and maintain a business map from dashboards, notebooks, and existing pipelines; Snowflake folds Cortex Analyst into Snowflake Intelligence, extending federated Q&A to unstructured data; Microsoft reports that Data Agent workloads' end-to-end response time fell by up to 50% in 2026 (self-reported vendor data, marked [To be verified]).

3.1.3 Results

The effect of this case lies not in some accuracy number, but in the formation of industry consensus:

• Snowflake has publicly stated that Cortex Analyst is one of the most difficult products it has designed and released; the difficulty is not in writing SELECT, but in whether the answer respects business semantics (revenue-recognition rules, data-sharing policies, customer hierarchies).
• Microsoft Fabric, for its part, calls data fragmentation the "poison" of AI performance.

Both statements point to the same conclusion: the correctness boundary of a Q&A system is determined by the level of context governance, not by model capability. This is fully consistent with the core position of L1 in the Harness six-layer model.

It should be noted: the above statements all come from secondhand accounts and are marked [To be verified]; this case does not cite any undisclosed accuracy or customer-count data.

3.2 Case 2: The Enterprise Cliff — The Accuracy Drop-off from Leaderboards to Production Databases

3.2.1 Background

Text-to-SQL is one of the few directions with mature public benchmarks. The positioning of the three mainstream benchmarks differs greatly:

3.2.2 Solution

Placing the different systems' results on these three benchmarks side by side, the "enterprise cliff" becomes immediately visible (the following is the leaderboard status as of mid-2026; leaderboard data changes frequently):

Broken down by query complexity, the shape of the cliff becomes clearer:

3.2.3 Results

This data yields three core conclusions:

1. The same system can differ by dozens of percentage points between academic benchmarks and real enterprise databases. GPT-4o scores 86.6% on Spider 1.0 but only 10.1% on Spider 2.0; a specialized model scores 71.83% on BIRD but only 16.3% on Spider 2.0-DK. The gap comes from schema scale (more than 1,000 columns), dialect differences, and multi-step workflow requirements.
2. The industry's acknowledged "single biggest lever" is schema linking — treating table and column retrieval as a RAG problem rather than stuffing the entire schema into the prompt at once.
3. Production-grade deployment needs four gates: schema linking as a RAG problem; predefined metrics in the semantic layer (some sources claim this can eliminate roughly 60% of hallucinations, but the figure's source credibility is low, marked [To be verified]); three-stage query validation (syntax parsing → EXPLAIN dry run against the target dialect → database role-level read-only policy); and defense-in-depth guardrails.

In addition, evaluation metrics should not look at EX alone. One must also look at VES (valid efficiency score) — as the industry puts it: a query that returns the correct rows through a full table scan is a production incident waiting to happen.

3.3 Case 3: Ground Truth Itself Can Be Wrong — Benchmark Annotation Errors and Wording Loss

3.3.1 Background

This direction has executable ground truth, which is viewed as an advantage. But a study published at CIDR 2026 (University of Illinois, authors including Jin, Choi, Zhu, and Kang) calls the benchmarks themselves into question:

• Of the questions in BIRD Mini-Dev, 29.7%, and in Spider 2.0-Snow, 25%, were judged ambiguous (for example, two tables share a column with the same name, and the question does not indicate which diagnostic field is meant).
• Among questions with a public gold query, 52.8% in BIRD Mini-Dev and 66.1% in Spider 2.0-Snow contain annotation errors — in the researchers' words, "there are more wrong ground truths than right ones."
• Rerunning the five leading agents with the corrected answers: CHESS rose from 62% to 81%, jumping from 4th to 1st place, while the original 1st place fell to 3rd.
• The BIRD maintainers released a cleaned development set in 2025-11.

The above data comes from a secondhand account of the CIDR 2026 paper, is marked [To be verified], and citation is advised only after checking the original paper.

Another independent finding comes from the BIRD team's LiveSQLBench: when the same batch of questions was posed in both "clear phrasing" and "how real people phrase it," the 2025-09 results showed Gemini-2.5-Pro scoring 35.67 with normal phrasing and 28.67 with colloquial phrasing on the base-full set — with the same schema and same data, wording alone cost 7 percentage points.

3.3.2 Solution

These two sets of findings yield three direct implications for Harness design:

1. The evaluation set itself must be governed. Ground truth is not naturally correct; it needs the same version management, human review, and periodic cleaning as production data. The BIRD maintainers' practice of releasing a cleaned development set should be replicated by every builder of internal evaluation sets.
2. Ambiguity must be handled explicitly. When a question admits multiple reasonable interpretations, the correct behavior is to ask a clarifying question, not to pick one arbitrarily. This is an explicit branch the L3 orchestration layer must support.
3. Colloquial input needs a normalization layer. Users will not phrase questions the way benchmarks do. Harness should provide term-mapping and question-rewriting capabilities in L1 to align colloquial input with the canonical terms in the semantic layer.

3.3.3 Results

The most important effect of these findings is breaking the superstition about benchmark scores:

• An 80% leaderboard score may rest on ground truth with more than 50% annotation errors;
• A system that performs well in production may fare poorly on a leaderboard because it used a non-approved metric definition;
• Real user phrasings will score about 7 percentage points lower than benchmark phrasings.

Therefore the viable evaluation strategy for this direction is: use the internal verified queries set as the primary evaluation basis, and treat public benchmarks as reference rather than target. This is consistent with the Section 4.2 DoD setting in which "semantic regression set fully passing" takes precedence over "leaderboard score".

4. Practice Standards

4.1 AGENTS.md Specification

Standard source statement: the following is this document's proposed draft of an AGENTS.md standard for the data analytics direction. As of now there is no official or industry-recognized standard text; AGENTS.md is a community convention rather than a standard. This draft inherits all provisions of the data science group-level AGENTS.md.

# AGENTS.md —— 数据分析（Analytics / BI）

> 继承数据科学组级 AGENTS.md 全部条款。本节为数据分析方向的收紧与扩展。
> 本文件为标准建议稿，业界尚无官方标准版本。

## 角色与边界

- 本 Agent 是**口径的执行者，不是口径的定义者**。
- 可以做：检索语义层与指标注册表、检索 schema、生成并执行只读查询、解释结果、指出歧义并反问。
- 不可以做：定义或修改指标口径、绕过 RLS/CLS、执行写操作、用非批准口径回答业务问题。
- 遇到语义层中不存在或存在冲突的指标定义：**停止并上报口径缺口**，禁止自行发明口径。

## 环境假设

- 存在语义层或指标注册表（semantic views / 度量值 / 指标字典），且可检索。
- 存在可解析的元数据：表、列、关系、数据分级、所有权人。
- 查询在只读角色下执行；RLS/CLS 在数据访问层强制生效。
- 存在成本可见性：可获取查询计划与扫描量预估。
- 存在已验证查询集 / 黄金数据集用于回归。
- 目标方言已知（Snowflake / BigQuery / PostgreSQL / DAX / KQL 等）；未知时禁止生成查询。

## 上下文加载顺序（Context Budget）

1. 问题本身与用户身份（决定权限与可见范围）
2. **语义层中的指标定义与同义词**（涉及指标时必须，优先级高于 schema）
3. 经检索选中的表与列的 schema（**检索式 schema linking，禁止全量注入**）
4. 表间关系与 join 路径
5. 已验证查询集中与该问题最接近的范例
6. 数据剖析结果：行数、空值率、取值分布、基数
7. 历史查询与用户偏好
8. 领域知识与业务术语表

- 大 schema 必须走检索式 schema linking；一次装配的表数量应有上限。
- 语义层定义与模型记忆冲突时，**以语义层为准**。

## 工具契约

| 工具 | 用途 | 模式 | 约束 |
|---|---|---|---|
| 语义层 / 指标注册表 | 查询指标、维度、关系、同义词 | 只读 | 不得写入或修改定义 |
| 元数据服务 | 检索表与列、读取分级与所有权 | 只读 | 不得修改元数据 |
| 查询执行器 | 执行只读查询、获取结果 | 只读 | 只读角色；返回前按 RLS/CLS 过滤 |
| 查询计划器 | EXPLAIN 干跑、成本预估 | 只读 | 预估超阈值须中止并上报 |
| 已验证查询集 | 检索相似范例、触发回归 | 只读 | 不得修改标准答案 |
| 可视化 | 生成图表 | 读 + 生成 | 图表继承源数据分级 |

- 每个工具调用记录：时间、参数摘要、返回摘要、耗时、扫描量与成本。
- 查询结果在返回用户前必须经过权限过滤；**权限由系统强制，不由 Agent 判断**。

## 数据源契约

- 只允许使用语义层中已登记并授权的数据源。
- 每个指标引用必须标注：指标名、权威定义来源、版本或最近更新时间。
- 数据引用必须带版本：快照时间、分区或数据集版本号，四者至少其一。
- 涉及多个口径候选时，必须列出全部候选并说明差异，**不得静默择一**。
- 样本不等于总体：基于抽样的结论必须写明抽样方式与样本量。

## 任务执行流程（SOP）

1. **理解与对齐**：复述问题，识别所涉指标与维度，与语义层对齐术语。
2. **歧义检查**：检查问题是否存在多种合理解读（同名列、未指明的口径、未指明的粒度）；存在则**反问澄清**，不自行假设。
3. **口径确认**：确认指标使用的是批准口径；语义层缺失则上报口径缺口。
4. **schema linking**：检索相关表与列，而非全量注入。
5. **查询构造**：按目标方言生成查询；优先复用已验证查询集中的范例。
6. **三阶段校验**：
   - 语法解析：确认语句合法
   - `EXPLAIN` 干跑：确认可执行并预估扫描量与成本
   - 权限与只读策略：确认在只读角色下且未触碰越权对象
7. **执行**：在只读角色下执行；记录耗时、扫描量与成本。
8. **结果校验**：与已验证查询集或独立复算路径比对；检查量级合理性。
9. **解释与限定**：给出答案、口径说明、覆盖范围、不确定性与已知局限。
10. **归档**：有价值的问答对经验证后进入已验证查询集。

## 验证与证据要求

- 每条数值必须来自**执行结果**，禁止由模型凭记忆或估算生成。
- 必须执行三项校验：**语法解析 → EXPLAIN 干跑 → 权限与只读策略检查**。
- 关键指标必须与**已验证查询集**或**独立复算路径**比对，差值须说明。
- 除执行准确率外，必须评估**有效效率得分（VES）**：通过全表扫描返回正确结果的查询，视为不合格。
- 主动报告反例与不利数据，不得只呈现支持性结果。
- 结论分「已验证 / 部分验证 / 未验证」三档列出。

## 失败与升级策略

- 同类失败重试不超过 2 次；第 3 次改变策略或升级。
- **歧义问题**：反问澄清，提供候选解读供用户选择，禁止任选其一。
- **语义层缺失该指标**：停止回答，上报口径缺口并说明需要谁补定义。
- **权限不足**：明确告知无权限，禁止用代理字段或近似口径绕过。
- **成本超阈值**：中止执行，报告预估扫描量与成本，申请放量或缩小范围。
- **结果为零或为空**：先区分「真的没有数据」与「查询条件写错」，两者不得混报。
- **结果与直觉严重不符**：标注为部分验证，报告可能的口径差异，不得自行「修正」数字。
- 升级时携带：原始问题、候选口径、生成的查询、执行计划、失败点、建议下一步。

## 安全与合规红线

- 不得绕过、推测或代为申请 RLS/CLS 权限。
- 不得用近似字段、代理指标或非批准口径回答受限问题。
- 不得在未声明的情况下使用未登记的口径。
- 不得把受限数据写入低分级位置（缓存、日志、图表、导出文件）。
- 不得执行任何写操作。
- 不得隐瞒查询的成本、覆盖范围与已知局限。
- 涉及个人信息与敏感字段的查询，须确认用户具备相应权限后才可执行。

## 禁止事项

- 禁止把整个 warehouse schema 一次性注入上下文。
- 禁止在语义层缺失时自行发明指标口径。
- 禁止用「大概」「通常来说」替代具体数值与口径来源。
- 禁止跳过 `EXPLAIN` 干跑直接执行高成本查询。
- 禁止在未确认方言的情况下生成查询。
- 禁止把执行成功当作结果正确——执行成功只说明能跑，不说明算对了。
- 禁止跨方向复制通用模板；数据分析的口径治理要求与深度研究、HPC 有实质差异。

## 输出格式

- 答案先行 → 口径说明 → 证据（数据源 + 版本 + 查询 + 执行记录）→ 不确定性 → 建议。
- 数值带单位与口径说明；范围用「~」连接；百分比数值与 % 之间无空格。
- 涉及多口径候选时用表格列出：口径、定义、数值、差异原因。
- 每个答案必须包含覆盖范围（时间范围、数据范围、过滤条件）。
- 查询与配置使用带语言标记的代码块。

## 评估与自检

- [ ] 问题中的术语已与语义层对齐
- [ ] 歧义已检查；存在歧义时已反问澄清
- [ ] 使用的是批准口径，且已标注定义来源与版本
- [ ] schema 经检索式 linking 装配，未全量注入
- [ ] 三阶段校验（语法 / EXPLAIN 干跑 / 权限只读）已全部执行
- [ ] 执行结果已与已验证查询集或独立复算比对
- [ ] 已评估有效效率得分，非全表扫描换来的正确结果
- [ ] 权限过滤已生效，未发生越权可见
- [ ] 成本在阈值内，或已上报
- [ ] 覆盖范围、局限与不确定性已声明

4.2 SKILL.md Specification

Standard source statement: the following is this document's proposed draft of a SKILL.md standard for the data analytics direction; likewise, no official standard text exists.

---
name: metric-qa-and-analysis
description: 指标问答与自助分析的标准执行流程与验收基线。适用于自然语言问数、指标口径核对、多步下钻分析、口径漂移检测与已验证查询集建设。触发场景：任何需要基于企业数据用自然语言回答业务数字的任务。
version: 1.0
created: 2026-09-12
---

# 指标问答与自助分析标准流程

## 适用场景

- 业务用户用自然语言询问指标数值、趋势与构成。
- 指标口径核对与差异解释（两个报表数字不一致时定位原因）。
- 多步分析：澄清 → 取数 → 校验 → 解释 → 下钻。
- 口径漂移检测与已验证查询集建设。
- 不适用场景：需要写操作的数据处理、无语义层且无指标字典的裸库问答（应先补语义层）。

## 前置条件

- 语义层或指标注册表可用且可检索。
- 元数据可用：表、列、关系、分级、所有权人。
- 只读数据库角色已配置，RLS/CLS 已生效。
- 目标查询方言已知。
- 成本可见：可获取 EXPLAIN 与扫描量预估。
- 已验证查询集存在（可为空集，但需有建设路径）。

## 输入

| 输入项 | 必需 | 说明 |
|---|---|---|
| 自然语言问题 | 是 | 用户的原始提问，含其惯用措辞 |
| 用户身份与权限上下文 | 是 | 决定可见行与可见列 |
| 语义层 / 指标定义 | 是 | 指标的权威定义、维度、粒度 |
| 目标方言 | 是 | Snowflake / BigQuery / PostgreSQL / DAX / KQL 等 |
| 成本阈值 | 是 | 单次查询可接受的最大扫描量或费用 |
| 时间范围默认值 | 否 | 用户未指定时使用的默认窗口 |

## 输出

| 输出项 | 必需 | 说明 |
|---|---|---|
| 答案 | 是 | 数值 + 单位 + 口径 + 覆盖范围 |
| 口径说明 | 是 | 使用的指标定义、来源与版本 |
| 查询 | 是 | 实际执行的语句，可复现 |
| 验证结果 | 是 | 与已验证查询集或独立复算的比对结果 |
| 效率评估 | 是 | 扫描量与有效效率评估结论 |
| 不确定性 | 是 | 歧义、局限、已知口径差异 |

## 执行步骤

1. **术语对齐**
   把用户措辞映射到语义层中的规范术语与同义词；口语化表述须显式归一化；无法映射的术语进入待澄清清单。

2. **歧义检测**
   检查同名列、未指明的口径、未指明的粒度与时间范围；存在多解时生成候选解读并向用户确认。

3. **口径确认**
   在语义层中定位指标的权威定义；缺失则停止并上报口径缺口，禁止自行定义。

4. **schema linking**
   基于已对齐的术语检索相关表与列；限制装配数量；优先使用语义层声明的关系路径。

5. **范例检索**
   在已验证查询集中检索最相近的范例；命中则优先复用并按需微调。

6. **查询构造与三阶段校验**
   按方言生成查询；依次执行语法解析、`EXPLAIN` 干跑（获取扫描量与成本）、权限与只读策略检查；任一阶段不通过则停止。

7. **执行与取数**
   在只读角色下执行；记录耗时、扫描量与成本；结果按权限过滤后返回。

8. **结果校验**
   与已验证查询集或独立复算路径比对；检查量级是否合理；检查是否命中已知口径差异。

9. **效率评估**
   评估有效效率得分；若正确答案依赖全表扫描，标记为待优化，不得视为合格交付。

10. **解释与归档**
    输出答案、口径、覆盖范围、不确定性与建议；有价值的问答对经验证后进入已验证查询集；记录本次成本。

## 质量标准（DoD）

口径与语义：

- [ ] 使用的指标定义来自语义层，且已标注来源与版本
- [ ] 用户措辞已映射到规范术语，映射关系已记录
- [ ] 全部歧义已澄清或已列为不确定性
- [ ] 多口径候选已用表格列出，未静默择一

正确性：

- [ ] 三阶段校验全部通过
- [ ] 结果与已验证查询集或独立复算一致，差值已说明
- [ ] 量级合理性已人工或规则检查
- [ ] 结论分为已验证 / 部分验证 / 未验证三档

效率与成本：

- [ ] 已评估有效效率得分，非全表扫描换来的正确结果
- [ ] 扫描量与成本在阈值内，或已上报获批

治理：

- [ ] RLS/CLS 已生效，无越权可见
- [ ] 查询在只读角色下执行
- [ ] 输出物继承源数据分级，未泄漏受限内容
- [ ] 覆盖范围与局限性已声明

## 常见失败与处理

| 失败现象 | 根因 | 处理方式 |
|---|---|---|
| 模型引用了不存在的字段 | 全量注入 schema 或缺少 schema linking | 改为检索式 schema linking；装配前校验字段存在性 |
| 数字比预期大很多 | 在错误粒度上 join 导致扇出 | 校验关联键唯一性；改用语义层声明的关系路径 |
| 两个报表口径不一致 | 使用了非批准口径 | 列出两个口径的定义与差异，交所有权人裁定 |
| 用户看到了不该看的行 | RLS 未生效或被绕过 | 立即停止；权限由数据访问层强制，不得靠提示词约束 |
| 查询跑了几分钟还没完 | 未做 EXPLAIN 干跑与成本预估 | 中止；加分区过滤与聚合下推；设置扫描量上限 |
| 同一问题两次答案不同 | 数据未定版或时间窗口未固定 | 固定时间范围与数据版本；使用快照 |
| 回答了但用户说「不是这个口径」 | 术语映射错误 | 回到术语对齐步骤；把正确映射补充进同义词表 |
| 结果为空 | 条件过严或数据缺失 | 先区分「真的没有」与「条件写错」，分别报告 |
| 榜单分数高但生产表现差 | 基准与生产分布不一致 | 以内部已验证查询集为第一评估依据，公开基准仅作参考 |

## 示例

**任务**：业务用户问「上个月华东区的新客收入是多少」。

1. 术语对齐：「上个月」→ 上一个自然月（非滚动 30 天，需确认）；「华东区」→ 语义层中的 `region = 'East China'`；「新客」→ 语义层中 `is_new_customer` 口径（定义为首次成交在当前自然月内的客户）；「收入」→ 语义层中 `net_revenue`（**已确认收入，非下单金额**）。
2. 歧义检测：「上个月」存在自然月与滚动 30 天两种解读；「新客」存在首次注册与首次成交两种口径 → 生成候选解读向用户确认。
3. 口径确认：语义层中 `net_revenue` 定义明确，来源为指标注册表 v2026-08。
4. schema linking：检索到 `fct_orders`、`dim_customer`、`dim_region` 三张表及所需列，未全量注入。
5. 范例检索：已验证查询集中存在「分大区月度净收入」范例，复用其关系路径。
6. 三阶段校验：语法通过；`EXPLAIN` 预估扫描 12 GB，低于阈值；只读角色确认。
7. 执行：返回净收入数值，记录耗时与扫描量。
8. 结果校验：与「分大区月度净收入」范例中华东区上月数值比对，一致。
9. 效率评估：命中分区裁剪，有效效率得分合格。
10. 输出：答案 + 口径说明（已确认收入 / 首次成交口径 / 自然月）+ 覆盖范围 + 不确定性（若用户本意是滚动 30 天，数值会不同）+ 建议把该问答对加入已验证查询集。

4.3 Implementation Checklist

4.3.1 Context Layer (L1)

• [ ] Semantic layer / metric registry established, with complete metric definitions, dimensions, granularity, and synonyms
• [ ] Retrieval-based schema linking supported, limiting the number of tables and columns assembled per query
• [ ] Mapping table from business terms and colloquial phrasing to canonical terms available
• [ ] Verified queries set searchable and reusable as exemplars
• [ ] Data profiling results (cardinality, value distribution, null rate) can be injected into context

4.3.2 Tool and Execution Layer (L2)

• [ ] Queries executed under a read-only role
• [ ] Correct dispatch to NL2SQL / NL2DAX / NL2KQL by source type
• [ ] EXPLAIN dry-run and scan-volume estimation capability provided
• [ ] Schema validation for tool parameters
• [ ] Duration, scan volume, and cost recorded for every query

4.3.3 Orchestration and Control Layer (L3)

• [ ] Ambiguity branch supported: ask a clarifying question when multiple interpretations exist, rather than assuming
• [ ] Multi-step analysis supported: clarify → fetch data → validate → explain → drill down
• [ ] Clear escalation path for metric-definition gaps
• [ ] Retry policy explicit, no more than 2 retries for the same type of failure

4.3.4 Memory and State Layer (L4)

• [ ] Query history and user preferences queryable
• [ ] Valuable Q&A pairs can be accumulated into the verified queries set
• [ ] Term-mapping corrections can flow back into the synonym table
• [ ] Data snapshots and versions traceable

4.3.5 Evaluation and Observability Layer (L5)

• [ ] Verified queries set / golden dataset established, covering key metrics
• [ ] Every semantic-layer or model change triggers regression execution
• [ ] Both execution accuracy (EX) and valid efficiency score (VES) evaluated
• [ ] Metric-drift detection in place (multi-path comparison of the same metric)
• [ ] Evaluation set itself has version management and periodic cleaning
• [ ] Internal evaluation prioritized over public benchmarks; public benchmarks for reference only

4.3.6 Governance and Security Layer (L6)

• [ ] RLS / CLS enforced at the data access layer, not dependent on prompts
• [ ] Access-violation queries intercepted with audit records
• [ ] Per-query cost threshold set and abortable
• [ ] Outputs (cache, logs, charts, exports) inherit source data classification
• [ ] Metric-definition changes require owner confirmation and leave an audit trail

5. Summary

The data analytics direction enjoys the most ideal conditions in AI Harness, and faces its most graceful failures.

The most ideal conditions are its executable ground truth: SQL result sets can be compared, verified queries can be regressed, and execution plans can be evaluated for efficiency. This makes the direction one of the highest in "evaluation return on investment" in Harness — every investment in the evaluation layer translates directly into accuracy gains and fewer incidents.

The most graceful failure is that when it errs, it returns a well-formatted number accompanied by a chart. A retrieval system returns nothing when it fails, and you notice immediately; a Q&A system hands you a number when it fails.

Three main threads run through this direction:

1. Context governance sets the ceiling on correctness. The three platforms all converged on placing the semantic layer before Q&A, which is essentially an admission that Q&A questions are first and foremost context-engineering problems. Enterprise databases often contain more than 1,000 columns, so full injection is infeasible and schema linking must be treated as a retrieval problem.
2. Benchmark scores do not equal production readiness. The gap between 86.6% on Spider 1.0 and 10.1% on Spider 2.0 is the very shape of the enterprise cliff. More worrying still, the ground truth in benchmarks themselves can be riddled with annotation errors — ground truth must be governed, just as data must be governed.
3. Correct execution is not the same as a correct answer. Beyond execution accuracy, one must also look at the valid efficiency score; beyond a correct result, one must confirm the metric definition is the approved one, the permissions are effective, and the cost stays within threshold.

In one sentence, this direction's Harness proposition: first govern metric definitions into searchable context, then let the model answer questions; first validate against the verified queries set, then let the answer be seen.

Information Gap Statement

1. No official or industry-recognized standard text exists for the data analytics direction's AGENTS.md / SKILL.md. Sections 4.1 and 4.2 are both proposed standard drafts in this document.
2. Accuracy leaderboard data changes frequently; all leaderboard figures in this document are marked "leaderboard status as of mid-2026", and the latest leaderboard should be checked when citing.
3. The following figures come from secondhand accounts and are marked [To be verified]: from the CIDR 2026 paper, BIRD Mini-Dev 29.7% / Spider 2.0-Snow 25% ambiguity rates, 52.8% / 66.1% annotation-error rates, and CHESS 62% → 81% ranking change; refer to the original paper before citing.
4. The LiveSQLBench 35.67 / 28.67 comparison comes from a secondhand account and is marked [To be verified]; "7-point loss" is the direct difference of the data point, not an independent statistical conclusion.
5. "The semantic layer can eliminate roughly 60% of hallucinations" comes from a low-credibility source, is marked [To be verified], and is not used as a conclusion in this document.
6. Microsoft Fabric Data Agent's end-to-end response time reduced by up to 50% is vendor-reported data, marked [To be verified].
7. Platform pricing information (Databricks DBU allowances, Snowflake free inclusions, Fabric F2 tier pricing) is channel-sourced and changes frequently; this document does not cite specific prices.
8. Actual customer counts, ARR, and vendor-reported internal benchmark accuracy for various platforms (e.g., a vendor-reported text-to-SQL improvement from 32% to 90%+) did not have a reliable primary source this time and are not used here.
9. Statements by relevant Snowflake and Microsoft executives all come from secondhand accounts and are marked [To be verified].
10. The metric names, table names, and parameters in the Section 4.2 example are illustrative constructions and do not represent any real system.

6. References

1. Text-to-SQL Accuracy in 2026: The Performance Cliff Nobody Mentions (positioning and accuracy data for the three benchmarks) — Onvo AI. https://www.onvo.ai/blog/text-to-sql-accuracy-in-2026-the-performance-cliff-nobody-mentions
2. Natural Language to SQL Accuracy in 2026 (accuracy broken down by complexity) — AI for Database. https://www.aifordatabase.com/blog/natural-language-sql-accuracy-2026
3. Plausible Wrong Numbers (CIDR 2026 benchmark annotation-error research, LiveSQLBench wording loss) — Mantissa AI. https://www.mantissaai.com/blog/nl-to-sql-plausible-wrong-numbers
4. Snowflake Arctic-Text2SQL-R1-32B benchmark performance — Awesome Agents. https://awesomeagents.ai/models/snowflake-arctic-text2sql-r1-32b
5. Text-to-SQL LLM in Production: Schema Linking and Guardrails (the four gates and VES) — AI Workflow Lab. http://aiworkflowlab.dev/article/text-to-sql-llm-production-schema-linking-guardrails-2026
6. Data Agent Architecture Survey: From Architecture to Practice (comparison of the three platforms' semantic layers) — technical analysis blog. https://ibillxia.github.io/blog/2026/08/20/data-agent-servey-from-arch-to-practice
7. Databricks Genie vs Snowflake Cortex Analyst vs Microsoft Fabric Data Agents — Agilytic. https://www.agilytic.com/blog/databricks-genie-cortex-analyst-fabric-data-agents
8. The Five Failure Modes of Bad AI-generated SQL (source of the analysis framework) — Perplexity AI Magazine. https://perplexityaimagazine.com/?p=3450/
9. DAMA Guide to the Data Management Body of Knowledge (data warehousing and business intelligence knowledge areas) — DAMA International. https://www.longshidata.com/blog/c/c2026062301.html
10. MCP Data Science Workflow Tool Ecosystem (database-type MCP Servers) — technical guide. https://chatforest.com/guides/mcp-data-science-workflows
11. OpenLineage official specification documentation — Linux Foundation. https://openlineage.io/
12. ISO 8000 Data Quality Standards Series Summary — ISO/TC 184/SC 4. https://quality.arc42.org/standards/iso-8000