RAG · 索引、切分、召回、重排与上下文装配


1. 介绍

1.1. 背景

检索增强生成(Retrieval-Augmented Generation,RAG)的出现,是为了解决大模型在企业落地中最致命的一个问题:模型的知识是参数化的、静态的、不可追溯的

RAG 的基本思想很朴素:先在外部语料中检索出与问题相关的片段,再把这些片段作为上下文交给模型生成答案。但在工程实践中,RAG 的失败率极高,且失败模式隐蔽——答案看起来流畅专业,实际上召回的片段与问题无关,或者相关片段被埋在上下文中部而未被利用。

RAG 的失败来源按影响程度排序,通常为:

  1. 切分(Chunking):语义单元被切断,导致片段本身不可理解;
  2. 召回(Retrieval):嵌入模型域外、纯向量召回丢失关键词命中;
  3. 装配(Assembly):召回正确但放置位置错误,或上下文超预算;
  4. 生成(Generation):模型脱离上下文自由发挥。

前三项都属于 Harness 工程范畴,只有第四项与模型能力直接相关。这正是本方向的核心论点:RAG 质量的绝大部分改进空间在工程层,而非模型层

1.2. 定义

RAG(检索增强生成):在生成答案之前,先从外部语料中检索相关片段并装配进上下文,使生成过程建立在可追溯证据之上的技术体系。

一个完整的 RAG 管线包含五个阶段:

阶段输入 → 输出关键决策
切分原始文档 → 片段集合块大小、重叠比例、切分策略(递归/语义/层次)
索引片段 → 向量索引 + 关键词索引嵌入模型、维度、量化、元数据字段
召回查询 → 候选集稠密/稀疏/混合策略、融合算法、Top-N
重排候选集 → 有序候选集重排模型、候选数、阈值
装配有序候选集 → 上下文Top-K、预算控制、放置顺序、引用标注

1.3. 在 AI Harness 体系中的定位

图 1-1|RAG 六层能力模型:主层 L1 上下文工程、次层 L5 评估观测

RAG 在 Harness 六层能力模型中的定位 该映射为本文分析 · 示意:基于本文分析绘制 L1 上下文工程层 主层 · 瓶颈所在 切分、索引、召回、重排、装配全部环节 · 上下文预算、关键信息首尾放置 L2 工具与执行层 语料连接器 · 脚本化切分与跑分 · MCP 连接器、scripts/ L3 编排与控制层 多阶段检索:改写查询 → 多路召回 → 融合 → 重排 · 流水线编排 L4 记忆与状态层 索引版本 · 查询缓存 · 会话历史 · 索引快照、KV Cache 复用 L5 评估与观测层 次层 检索评测与生成评测 · RAGAS、BEIR、MTEB/MMTEB、Trace L6 治理与安全层 权限感知检索 · 数据分级 · 审计 · 召回阶段权限过滤 结构解读:瓶颈全部在 L1 上下文工程 —— 相关片段一旦未被召回,生成层无法补救,质量改进空间绝大多数在工程层。

数据来源:基于本文分析绘制的示意图。

RAG 在 Harness 六层能力模型中的定位为:主层 L1 上下文工程层,次层 L5 评估与观测层(该映射为本文分析)。

在 RAG 中的体现关键机制
L1 上下文工程(主)切分、索引、召回、重排、装配的全部环节上下文预算、预算裁剪优先级、关键信息首尾放置
L2 工具与执行语料连接器、脚本化切分与跑分MCP 连接器、scripts/
L3 编排与控制多阶段检索(改写查询→多路召回→融合→重排)检索流水线编排
L4 记忆与状态索引版本、查询缓存、会话历史索引快照、KV Cache 复用
L5 评估与观测(次)检索评测与生成评测RAGAS、BEIR、MTEB/MMTEB、Trace
L6 治理与安全权限感知检索、数据分级、审计召回阶段权限过滤

瓶颈所在层:RAG 的瓶颈全部在 L1。理由如下:

  1. RAG 的本质是"决定模型看到什么",这正是 L1 的职责定义;
  2. RAG 的所有可调参数(切分、嵌入、融合、重排、Top-K、预算)都是上下文工程参数;
  3. RAG 的失败绝大多数不可在生成层修复——一旦相关片段未被召回,再强的模型也无能为力。

为什么长上下文不能替代 RAG:这是本方向最常被误解的问题。三个权威证据链给出了否定答案:

  • Lost in the Middle(Liu et al., TACL 2024):多文档问答中固定文档数、移动含答案文档的位置,得到 U 形曲线——答案在首部或尾部时准确率最高,中部显著下降;某些配置下把相关信息埋在中部,表现甚至低于该模型的闭卷(closed-book)表现,即召回的上下文是净负收益
  • RULER(Hsieh et al., NVIDIA,COLM 2024):原文批评 NIAH "is indicative of only a superficial form of long-context understanding";在评测的 17 个长上下文模型中,只有约一半能在 32K 长度维持令人满意的表现
  • NoLiMa(Modarressi et al., ICML 2025):改写 needle 以最小化字面重叠后,在 13 个宣称支持 ≥128K 的模型中,11 个在 32K 时就跌破其自身短上下文准确率的一半;GPT-4o 从 99.3% 降到 69.7%。

反过来的边界(何时不用 RAG):Anthropic 官方给出的一条极具工程价值的规则——"If your knowledge base is smaller than 200,000 tokens (about 500 pages of material), you can just include the entire knowledge base in the prompt... with no need for RAG or similar methods."

价值:RAG 是 L1 上下文工程层最成熟、最可度量、最可优化的实现。它把"模型知道什么"这个不可控问题,转化成了"我们给模型看了什么"这个可控问题。

1.4. 技术全景与关键参数

来源可靠性提示:1.4.1 与 1.4.2 中的量化数值绝大多数来自 SEO/聚合内容站,未回溯到 Chroma Research、Weaviate、NVIDIA FinanceBench、OpenAI 官方定价页、MTEB 官方榜单等原始来源,因此全部标注 ,不得作为设计依据直接引用,只能作为调优方向的线索。1.4.3 节(Anthropic Contextual Retrieval)为一手官方数据(R1),可直接引用。

1.4.1. 切分(Chunking)
参数/结论数值来源级别
生产默认基线递归字符切分,400~512 tokens + 10%~20% 重叠
重叠比例经验值块大小的 10%~20% 为通行默认;NVIDIA FinanceBench 在 1024-token 块下发现 15% 最优超过 30% 收益迅速递减而索引线性增长
按查询类型的块大小事实型查询 256~512 tokens;分析/多跳查询 512~1024 tokens;混合负载起点 400~512 tokens;500-token 块建议 50~100 tokens 重叠
递归 vs 语义(召回口径)Chroma 研究:递归字符切分在 400 tokens 下达 85%~90% 召回;语义切分达 91%~92% 召回(代价是切分阶段需对每个句子做一次嵌入调用)
递归 vs 语义(端到端口径)2026 年 2 月一份跨 50 篇学术论文的基准:递归 512-token 切分端到端准确率 69%,语义切分 54%(差 15 个百分点),原始出处未找到
切分方法差距上限Weaviate 2025 基准:同一语料、同一嵌入模型与检索器下,最优与最差切分方法召回差距最高 9 个百分点
层次化/父子切分小"子块"用于检索、大"父块"送入 LLM;典型配置 子块 256 / 父块 2048;Amazon Bedrock Knowledge Bases 支持两级层次切分(注意:多个子块命中同一父块时返回数会少于请求数)
装配上下文上限每次调用装配上下文建议低于 8K tokens;有分析指出 2,500 tokens 附近存在"上下文悬崖(context cliff)"
主流库默认值LlamaIndex SentenceSplitter 默认 chunk_size=1024, chunk_overlap=200;LangChain 推荐 RecursiveCharacterTextSplitter 为通用默认;OpenAI Assistants 默认 800 tokens + 400 tokens 重叠(在 Chroma 评测中表现不佳)

可写入正文的分析结论:切分是 RAG 失败的第一来源,但其收益高度依赖语料——必须以自有黄金集实测 Recall@K 定参,不得照搬默认值。注意上表中"语义切分召回更优"与"语义切分端到端准确率更低"两组数据方向相反,这正说明不同口径下结论可能反转,进一步印证了"必须实测"的必要性。

1.4.2. 检索与重排
参数/结论数值来源级别
稠密 vs BM25 vs 混合关键词密集查询:纯稠密 0.58 NDCG、纯 BM25 0.88 NDCG、混合 RRF 0.89 NDCG;复杂混合查询:混合 RRF 0.85,叠加 cross-encoder 重排后 0.93
混合检索实现方式Dense(向量)+ Sparse(BM25)+ RRF(Reciprocal Rank Fusion) 融合;RRF 无参数,先把原始分转为排名再合并
嵌入模型(闭源 API)text-embedding-3-large:MTEB 约 64.6%、3072 维、8191 上下文、$0.13/MTok;text-embedding-3-small:62.3%、1536 维、$0.02/MTok;ada-002:61.0%,不建议新项目使用(价格与维度可到 OpenAI 官网核实)
开源/自托管BGE-M3(568M 参数、1024 维、8192 token、dense+sparse+multi-vector 三路输出);bge-large-zh-v1.5(326M、1024 维、512 token);Qwen3-Embedding(0.6B/4B/8B,Apache 2.0)(规格建议以官方模型卡为准)
检索子榜(C-MTEB 中文)bge-m3 综合 71.4 / Retrieval ndcg@10 73.9;bge-large-zh-v1.5 68.6 / 71.5;bge-base-zh-v1.5 67.5 / 70.2;m3e-base 61.5 / 64.3(随榜单更新会变化)
量化代价FP16 召回损失 <0.1%、吞吐 1.5~2 倍;INT8 损失 0.5%~1.5%、2~3 倍;INT4 损失 2%~5%、3~4 倍
生产上下文长度团队通常把生产上下文压到 16K~50K tokens/请求,文档分析偶尔 80K;RAG 管线通常只给模型 3K~10K token 的高信号切片
Top-K 经验送入 20 个块优于 10 个和 5 个,但存在上限
1.4.3. Anthropic Contextual Retrieval(一手官方数据,R1)

来源:Anthropic 官方博客 https://www.anthropic.com/news/contextual-retrieval

方案Top-20 检索失败率相对基线的降低
基线(仅 Embedding)5.7%
标准混合(Embedding + 原生 BM25)5.0%小幅优化(该数值来自中文转述,官方博客主表未列 → )
Contextual Embeddings3.7%35%
Contextual Embeddings + Contextual BM252.9%49%
再叠加 Reranking1.9%67%

官方原文表述:"This method can reduce the number of failed retrievals by 49% and, when combined with reranking, by 67%."

指标定义:失败率 = 1 - recall@20,即"前 20 个检索结果中找不到相关文档的失败率";测试域覆盖代码库、小说、ArXiv 论文、科学文献。

两阶段工程流程(官方原文)

  • 离线预处理(一次性):切分为数百 token 的块 → 对每个块向 Claude 传入完整文档 + 该块,生成 50~100 token 的上下文描述 → 前置拼接到块上 → 对拼接文本生成语义嵌入 → 对拼接文本构建 BM25 索引。
  • 在线推理:BM25 与向量双路召回 → Rank Fusion 融合去重 →(可选)对 Top 150 候选做重排 → 取 Top 20 送入 LLM。

成本控制(官方原文):利用 Prompt Caching,完整文档只加载一次,各块共享缓存;预处理一次性成本约 $1.02 / 百万文档 token

这条数据是本方向最有价值的一手证据:在模型不变的前提下,纯工程手段把检索失败率降低了 67%

1.4.4. 评测方法学

RAGAS(R1/R2,NVIDIA NeMo 微服务文档、Red Hat OpenShift AI 文档):

类别指标
检索侧context_recall(参考信息被召回的比例)、context_precision(召回块与问题的相关度)、context_relevancecontext_entity_recall
生成侧faithfulness(回答与召回上下文的事实一致性)、response_relevancy(需 embeddings_model)、response_groundedness
综合侧answer_correctness(语义相似度 + 事实相似度)、answer_similaritynoise_sensitivity(对噪声/无关上下文的鲁棒性)
  • 分值区间:context_recall 为 0~1,越高越好。
  • 依赖:需要一个 Judge LLM;部分指标额外需要 Judge Embeddings。
  • 支持离线(预生成回答)与在线(自动生成回答)两种评测模式。
  • 落地用法:自动化质量门(每次 commit 或模型更新后触发,防止回归)、评估驱动开发(对比不同切分策略/嵌入模型/检索算法)、事实一致性阈值(对 faithfulness 设阈值)、生产规模化(remote provider 模式下分布式跑数千条样本)。

BEIR(R2,原始论文 arXiv:2104.08663,Thakur et al., NeurIPS 2021 Datasets and Benchmarks):

  • 18 个数据集、9 类检索任务:fact-checking、citation prediction、duplicate-question retrieval、argument retrieval、news retrieval、QA、tweet retrieval、bio-medical IR、entity retrieval。
  • 主指标 nDCG@10,同时计算 MAP、Recall@k、Precision@k、MRR(k 从 1 到 1000)。
  • 核心方法论贡献:零样本(zero-shot)评估——稠密检索器在域内常优于 BM25,跨域则常落败,BEIR 暴露了这一泛化鸿沟。
  • 实用提醒:部分数据集需授权或手动下载(TREC-NEWS、Robust04、Signal-1M、BioASQ),公开可跑子集约 15~17 个不同子集算出的"BEIR 均值"不可直接比较

MTEB / MMTEB(R2,原始论文 arXiv:2210.07316,Muennighoff et al.):

  • 原始 MTEB:8 类任务、58 个数据集、112 种语言;指标分别为 retrieval(nDCG@10)、classification(accuracy)、clustering(v-measure)、pair classification(AP)、reranking(MAP)、STS(Spearman)、summarization(Spearman)、bitext mining(F1)。
  • MMTEB(Enevoldsen et al., ICLR 2025,arXiv:2502.13595):扩展至 500+ 评测任务、250+ 语言;榜单按命名切分(English v2 / multilingual)并改用 Borda 计数排名。
  • 对 RAG 选模型的关键建议(R1 级方法论):"Quote the retrieval sub-score, not the overall MTEB rank."——RAG 场景应看 retrieval 子分,而非总排名。

2. 名词解释

术语英文/缩写释义
检索增强生成RAGRetrieval-Augmented Generation,先检索后生成的技术体系
切分Chunking把文档切为可检索片段的过程
递归字符切分Recursive Character Splitting按字符层级(段落→句子→词)递归切分的通用策略
语义切分Semantic Chunking按语义相似度边界切分的策略,需对句子做嵌入
父子块Parent-Child Chunk小子块用于检索、大父块送入生成的层次化结构
重叠Overlap / Chunk Overlap相邻块之间重复的内容,用于缓解边界语义丢失
稠密检索Dense Retrieval基于向量相似度的检索
稀疏检索Sparse Retrieval基于词频统计(如 BM25)的检索
混合检索Hybrid Search稠密与稀疏两路召回的融合
倒数排名融合RRFReciprocal Rank Fusion,无参数的多路排名融合算法
重排Reranking用 cross-encoder 等模型对候选集精排序
交叉编码器Cross-encoder同时编码查询与文档的相关性打分模型
上下文装配Context Assembly把重排结果组织为最终送入模型的上下文
上下文悬崖Context Cliff上下文超过某一长度后质量急剧下降的现象
召回率Recall@K前 K 个结果中包含相关文档的比例
归一化折损累积增益nDCG@10前 10 个结果的排序质量指标
忠实度Faithfulness生成内容与所提供上下文的事实一致性
上下文召回Context Recall参考信息被成功召回的比例(RAGAS 指标)
噪声敏感度Noise Sensitivity对噪声/无关上下文的鲁棒性(RAGAS 指标)
提示词缓存Prompt Caching复用已缓存前缀以降低重复输入的成本
上下文检索Contextual Retrieval为每个块生成上下文描述后再嵌入与索引的方法
干草堆找针NIAHNeedle in a Haystack,长上下文检索能力评测
中间迷失Lost in the Middle长上下文中部信息利用率显著下降的现象
权限感知检索Permission-aware Retrieval在召回阶段按权限过滤结果的检索方式
黄金集Golden Dataset组织内部标注的高质量评测集

3. 案例

3.1. LinkedIn:知识图谱增强的客服 RAG

3.1.1. 背景

LinkedIn 的客服团队需要处理大量技术性问题,历史解决方案散落在 Jira 工单中。纯文本 RAG 的难点在于:工单之间的关系(克隆自、关联、导致)承载了大量语义信息,而这些信息在文本切分后会丢失。

3.1.2. 方案

依据 ZenML LLMOps 数据库与 Evidently AI 的汇总(R2,原始来源为工程分享):

  • 将历史 Jira 工单构建为树状知识图谱:节点 = Summary / Description / Priority / Steps to Reproduce / Fix Solution。
  • 显式关系来自 Jira 元数据(cloned from / related to / caused by)。
  • 隐式关系用 E5 嵌入余弦相似度 + 阈值建立。
  • GPT-4 负责解析与生成;Qdrant 存储向量。
  • 已部署至多条产品线,生产运行约 6 个月
3.1.3. 效果

依据上述来源(R2):

指标数值
MRR提升 77.6%
BLEU提升 0.32
每问题中位解决时间降低 28.6%

该案例的工程启示是:结构化关系 + 向量检索的组合,比纯向量 RAG 更能捕捉工单类语料的语义结构。

3.2. DoorDash:护栏与 LLM Judge 构成的评测闭环

3.2.1. 背景

DoorDash 的客服聊天机器人直接面向终端用户,一次错误回答会造成实际业务损失。因此其技术重点不只是"答得准",还包括"能发现答得不准"。

3.2.2. 方案

依据 Evidently AI 汇总(R2,原始来源为 DoorDash 工程博客):

  • 系统架构 = RAG 系统 + LLM Guardrail:在线监控每一条回答的准确性与合规性。
  • LLM Judge 从五个维度评测:检索正确性、回答准确性、语法语言准确性、上下文连贯性、与用户请求的相关性。
3.2.3. 效果

该来源未披露具体的指标提升数值(R2)。其价值在于给出了一套可复制的线上评测结构:把检索正确性与回答准确性分开评测,使得"检索错了"与"检索对了但生成错了"可以被区分定位——这正是 RAGAS 等评测框架的设计思想在生产中的对应实现。

3.3. Bell:模块化文档嵌入管线与增量索引

3.3.1. 背景

Bell 是加拿大电信运营商,其文档体系庞大且持续更新。一次性全量重建索引在成本与时延上均不可接受。

3.3.2. 方案

依据 Evidently AI 汇总(R2,原始来源为会议演讲视频):

  • 构建模块化文档嵌入管线,支持批处理与增量更新。
  • 文档增删时自动更新索引,避免全量重建。
  • 每个组件作为独立服务,按 DevOps 原则建设与维护。
3.3.3. 效果

该来源未披露量化指标(R2)。其工程价值在于明确了 RAG 的运维侧要求:索引不是一次性产物,而是需要持续演进的服务;增量更新与组件解耦是 RAG 上生产的必要条件。

其他可参考案例(R2)

案例要点
Harvard Business School · ChatLTV语料约 200 份文档、1,500 万词(案例、教学笔记、书籍、博客、课程 Slack 历史问答);集成在课程 Slack 频道,支持私聊与公开两种模式
Vimeo视频转写 → 处理 → 片段检索 → 视频问答;需处理说话人识别与长上下文检索问题
GrabReport Summarizer 调用 Data-Arks API 生成表格数据再由 LLM 总结;每份报告节省约 3~4 小时
Pinterest表描述向量索引 → 语义检索候选表 → LLM 精选子集 → 生成 SQL
Glean(企业 AI 平台)厂商宣称:110 小时/用户/年节省、内部支持工单减少 20%、2 年内企业采用率 93%、ROI 周期 <6 个月、相比现成 MCP 工具 token 用量降低 30%

Glean 数据为厂商宣称,引用时须标注。


4. 实践标准

4.1. AGENTS.md 规范

以下为 RAG 方向的行业标准 AGENTS.md 完整可复制原文,体现向量库、切分器、嵌入模型、重排模型、评测框架等专有工具链。

# AGENTS.md —— RAG(检索增强生成)

## 角色与边界
- 你是检索增强智能体,负责切分、索引、召回、重排、装配与评测。
- 你可以:读写索引、执行检索与重排、调整检索参数并提交评测、生成评测报告、调用脚本做跑分。
- 你不可以:在未见评测结果的情况下把参数变更推到生产、绕过权限过滤、
  把评测集混入索引语料、删除历史索引快照、编造评测数值。
- 判定原则:任何影响检索结果的变更,必须先评测后上线。

## 环境假设
- 运行环境提供:向量库、BM25 索引、嵌入服务、重排服务、切分器、
  评测框架(RAGAS / 自有黄金集)、对象存储(索引快照)、Trace 与审计日志。
- 语料具备版本/快照标识;索引具备版本号,可与语料版本对应。
- 检索服务在召回阶段即完成权限过滤(permission-aware retrieval)。
- 具备上下文预算读数(已装配 Token 数 / 上限)。

## 上下文加载顺序(Context Budget)
1. 查询与任务契约(常驻,不压缩)
2. 检索配置:切分参数、嵌入模型、融合方式、重排模型、Top-K(常驻)
3. 候选集(重排后的 Top-K,附 doc_id + 版本 + 片段定位)
4. 会话历史(多轮时,压缩为要点)
5. 参考示例(按需)
- 装配上限:默认低于 8K tokens;关键信息置于首部或尾部,不得埋在中部。
- 关键片段优先,低相关片段先被裁剪。

## 工具契约
- 切分器:递归 / 语义 / 层次化;参数必须由自有黄金集实测确定,不得照搬默认值。
- 向量库:写入前校验幂等;索引变更生成新版本快照;旧快照保留可回溯。
- 检索:默认向量 + BM25 混合召回 + RRF 融合;纯向量检索不得作为生产默认。
- 重排:候选集进入上下文前做 cross-encoder 重排;重排阈值写入配置。
- 评测:RAGAS(context_recall / context_precision / faithfulness / answer_correctness /
  noise_sensitivity)+ 自有黄金集;需要 Judge LLM 与 Judge Embeddings。
- 脚本:切分、跑分、批量评测、格式转换必须调用 scripts/,不得用生成方式替代。

## 任务执行流程(SOP)
1. 定界:语料白名单与版本、密级、查询类型(事实型 / 分析型 / 多跳)。
2. 判定模式:语料 < 200,000 tokens 走全量装配,不做切分召回。
3. 检索:向量 + BM25 → 权限过滤 → RRF 融合 → 重排 → Top-K。
4. 装配:按预算组织上下文;关键信息首尾放置;标注来源标识。
5. 生成:逐条陈述挂来源标识;无召回支撑不输出事实性结论。
6. 评测:任何参数变更先在黄金集上跑 Recall@K 与忠实度,与基线对比。
7. 上线:指标不低于基线方可上线;记录基线变化与变更日志。
8. 回流:把失败查询与人工修正加入评测集与负样本。

## 验证与证据要求
- 证据包:来源清单(doc_id + 版本 + 片段定位)、检索记录(查询、召回数、
  重排 Top-K、权限过滤命中数)、评测报告(指标 + 基线对比)、变更日志、索引版本快照。
- 引用外部基准时须注明所跑子集与指标口径(不同 BEIR 子集的均值不可直接比较)。
- 嵌入模型选型须引用 retrieval 子分,而非 MTEB/MMTEB 总排名。

## 失败与升级策略
| 失败 | 处置 |
|---|---|
| 检索零命中 | 输出"未检索到可靠依据";检查索引版本与查询改写;禁止自由生成 |
| 召回正确但答案错误 | 检查装配位置与上下文预算;提高忠实度阈值 |
| 参数变更导致指标回退 | 回退上一配置;重新评测;记录归因 |
| 权限过滤命中异常 | 停止检索,记录审计事件,升级至权限管理员 |
| 索引与语料版本不匹配 | 暂停服务,重新索引并校验 |
| 评测不可比 | 固定子集、模型版本与随机种子;注明口径 |
- 每个循环必须有步数上限与 Token 预算上限;超限即停并升级。

## 安全与合规红线
- 不得绕过权限过滤;不得使用高权限账号做检索。
- 不得把评测集混入索引语料造成数据污染。
- 不得删除历史索引快照;不得编造评测数值。
- 未经评测通过的检索配置不得推到生产。

## 禁止事项
- 禁止照搬默认切分参数(400~512 tokens + 10%~20% 重叠仅为线索,须实测)。
- 禁止把 `[待核实]` 的聚合站数值当作设计依据。
- 禁止用纯向量检索作为生产默认。
- 禁止在无召回支撑时输出事实性结论。
- 禁止编造 doc_id、版本号、评测指标与 URL。
- 禁止使用 XX / XXX / ___ 等非标准占位符(统一用 [待填写] / [待核实])。
- 禁止使用 emoji 与署名。

## 输出格式
- 检索结果:片段内容 + 来源标识 + 相似度/重排分 + 用于哪一论点。
- 评测报告:指标 / 基线 / 当前值 / 差值 / 是否通过门控 / 口径说明。
- 数值带单位;范围用 ~ 连接;中文全角标点;中英文之间加空格。

## 评估与自检
- 九项自检:来源可回溯 / 无无源断言 / 数值一致 / 无非标占位符 / 数量与表格一致 /
  编号可核实 / 密级正确 / 评测已跑且可比 / 信息缺口已声明。
- 每次索引或模型变更后重跑评测集;指标回退视为缺陷。
- 失败查询与人工修正必须回流进评测集。

4.2. SKILL.md 规范

以下为 RAG 方向的行业标准 SKILL.md 完整可复制原文。

---
name: rag-eval
description: 构建 RAG 评测集并在切分/嵌入/融合/重排/装配变更后跑分,产出与基线的对比报告与上线门控结论。当用户要求"评测这套检索""对比两种切分策略""换嵌入模型后效果如何""生成 RAG 回归报告""为什么这条查询召回不到"时触发。
version: 1.0
created: 2026-09-12
---

# RAG 评测与调优(RAG Evaluation & Tuning)

## 适用场景
- 构建与维护黄金集(Golden Dataset)与负样本集。
- 切分参数、嵌入模型、融合方式、重排模型、Top-K 的对比评测。
- 索引或语料大版本更新后的回归验证。
- 单条查询召回失败的归因分析。

## 前置条件
- 已有基线指标(Recall@K、context_recall、faithfulness 等)与对应的配置快照。
- 评测集版本固定,且未混入索引语料。
- 运行环境提供 Judge LLM 与 Judge Embeddings。
- 已确定指标口径与门控阈值。

## 输入
- 待评测配置:切分参数 / 嵌入模型 / 融合方式 / 重排模型 / Top-K / 装配预算
- 评测集版本与门控阈值
- 可选:失败查询清单(用于归因分析)

## 输出
- 评测报告:指标 / 基线 / 当前值 / 差值 / 是否通过门控 / 口径说明
- 归因分析:失败查询归类(切分问题 / 召回问题 / 重排问题 / 装配问题)
- 上线建议:通过 / 回退 / 需人工裁决
- 评测集更新建议:新增负样本与边界用例

## 执行步骤
1. 固定变量:一次只变更一个因素,其余保持不变。
2. 跑检索侧指标:Recall@K、context_recall、context_precision、context_entity_recall。
3. 跑生成侧指标:faithfulness、response_relevancy、response_groundedness。
4. 跑综合与鲁棒性:answer_correctness、noise_sensitivity。
5. 与外部基准对齐(可选):BEIR 子集(注明子集与 nDCG@10 口径)、
   MTEB/MMTEB retrieval 子分(**引用子分,不引用总排名**)。
6. 对比基线,输出是否通过门控;未通过则回退并归因。
7. 把失败查询归类:切分 / 召回 / 重排 / 装配,并加入评测集。
8. 产出报告与变更日志;记录索引版本快照。

## 质量标准(DoD)
- 一次只变更一个变量,配置快照完整可复现。
- 指标口径明确;引用外部基准时注明子集。
- 与基线的差值、统计窗口、样本量均已披露。
- 失败查询已归类并进入评测集。
- 通过门控才可给出上线建议。

## 常见失败与处理
| 失败 | 根因 | 处置 |
|---|---|---|
| 指标不可比 | 子集/口径/模型版本变化 | 固定变量重跑;注明口径 |
| 检索好但生成差 | 装配位置或预算问题 | 关键信息首尾放置;压缩低相关片段 |
| 生成好但检索差 | 评测集过易或样本偏差 | 补充难样本与负样本 |
| 评测结果不稳定 | Judge 模型/随机性 | 固定 Judge 模型与种子;多次取均值 |
| 线上与离线不一致 | 索引版本不匹配 | 校验线上索引版本与评测时一致 |
| 换嵌入模型无提升 | 引用了总排名而非 retrieval 子分 | 改看 retrieval 子分与域内召回 |

## 示例
用户请求:把切分从 512 tokens/15% 重叠改为 1024 tokens/20% 重叠,评估是否可上线。
执行:
1. 固定嵌入模型、融合方式、重排模型、Top-K=20、装配预算 8K tokens。
2. 在黄金集(v1.3,300 条)上跑 Recall@20、context_recall、context_precision。
3. 跑生成侧 faithfulness 与 answer_correctness(Judge 模型固定)。
4. 对比基线:Recall@20 由 0.81 变为 0.79(-0.02),faithfulness 由 0.92 变为 0.91。
5. 结论:未通过门控(Recall 回退),建议回退 512/15% 配置。
6. 归因:长块稀释了事实型查询的信号;把 12 条失败查询加入评测集。
7. 输出报告:指标表 + 口径说明 + 回退建议 + 变更日志。
约束:禁止在未跑评测的情况下给出"建议上线"结论;禁止引用聚合站数值作为依据。

4.3. 落地检查清单

#检查项判定标准频次
1小语料豁免已判定语料 < 200,000 tokens 时走全量装配,不建索引项目启动
2切分参数已实测以自有黄金集 Recall@K 定参,未照搬默认值每次定参
3索引版本可追溯索引有版本号,与语料版本一一对应每次索引
4混合召回为默认向量 + BM25 + RRF 融合;纯向量不作为生产默认每次上线
5重排已启用候选集进入上下文前做 cross-encoder 重排每次上线
6权限过滤在召回阶段结果已过权限过滤,非事后脱敏每次检索
7装配预算可控装配上下文低于 8K tokens;关键信息首尾放置每次调用
8来源标识完整每条片段附 doc_id + 版本 + 片段定位每次交付
9评测已跑且可比固定变量、固定子集、固定口径每次变更
10门控已设置faithfulness 与 Recall@K 有阈值,未达标不上线每次变更
11引用子分而非总排名嵌入模型选型看 MTEB/MMTEB retrieval 子分每次选型
12评测集未污染评测集未混入索引语料每次索引
13增量索引可用支持文档增删的增量更新,无需全量重建持续
14失败已回流失败查询归类并进入评测集与负样本每周
15确定性操作已脚本化切分、跑分、批量评测走 scripts/每次执行

5. 总结

RAG 是知识协同组中技术密度最高、评测体系最完备、也最容易被低估工程复杂度的方向。

本方向最重要的五条结论:

  1. 工程手段的收益可以量化且巨大。Anthropic 官方数据显示,Contextual Retrieval 将 Top-20 检索失败率从 5.7% 降至 2.9%(-49%),叠加 Reranking 后降至 1.9%(-67%),而预处理一次性成本仅约 $1.02 / 百万文档 token(R1)。这是在模型不变前提下取得的。
  2. 长上下文不是 RAG 的替代品。Lost in the Middle 的 U 形效应、RULER 对 17 个长上下文模型的评测(仅约一半在 32K 维持满意表现)、NoLiMa 对 13 个 ≥128K 模型的评测(11 个在 32K 跌破自身短上下文准确率的一半),共同说明堆上下文可能带来净负收益。
  3. 小语料应直接全量入提示词。Anthropic 官方明确:< 200,000 tokens(约 500 页)无需 RAG。这条规则能显著降低小团队复杂度。
  4. 评测是上线门控,不是事后报告。RAGAS 把检索侧与生成侧分开度量,使得"检索错了"与"生成错了"可被区分定位;BEIR 强调零样本跨域评估,MTEB/MMTEB 强调引用 retrieval 子分而非总排名。
  5. 切分是第一失败来源,但无通用最优解。公开数据中"语义切分召回更优"与"语义切分端到端准确率更低"方向相反,说明必须以自有黄金集实测。

必须坦率指出:本方向在切分与检索的参数区,绝大多数公开数值来自 SEO/聚合内容站,未回溯到 Chroma Research、Weaviate、NVIDIA FinanceBench、OpenAI 官方定价页与 MTEB 官方榜单。这些数值已在 1.4.1 与 1.4.2 中全部标注 ,不应作为设计依据,只能作为调优方向线索。真正可信的量化锚点只有 Anthropic Contextual Retrieval 一组(R1)。

信息缺口声明

#缺口状态
1RAG 的 ISO/IEEE 正式标准暂无权威标准/规范
2Chroma Research 官方切分评测报告(chunking evaluation)未获取到原始页面,相关数值
3Weaviate 2025 官方切分基准未获取到原始页面,相关数值
4OpenAI 官方嵌入模型定价与维度未回溯官方定价页,相关数值
5MTEB / C-MTEB 官方榜单当前值未回溯官方榜单,相关数值
6NVIDIA FinanceBench 切分实验原始出处未获取到,
7"2026 年 2 月跨 50 篇论文基准:递归 69% vs 语义 54%" 的原始出处无结果
8生产上下文长度 16K~50K tokens 的原始统计来源
92,500 tokens "上下文悬崖" 的原始研究
10Top-K 经验值(20 > 10 > 5)的原始研究

6. 参考资料

  1. Introducing Contextual Retrieval — Anthropic,2024。https://www.anthropic.com/news/contextual-retrieval
  2. RAGAS Metrics — NVIDIA NeMo Microservices 文档。https://docs.nvidia.com/nemo/microservices/latest/evaluator/metrics/rag.html
  3. Evaluating AI Systems(RAGAS 落地用法)— Red Hat OpenShift AI 文档。https://docs.redhat.com/en/documentation/red_hat_openshift_ai_self-managed/3.3/html-single/evaluating_ai_systems/
  4. BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models — Thakur et al., NeurIPS 2021(arXiv:2104.08663)。https://arxiv.org/abs/2104.08663
  5. MTEB: Massive Text Embedding Benchmark — Muennighoff et al.(arXiv:2210.07316)。https://arxiv.org/abs/2210.07316
  6. MMTEB: Massive Multilingual Text Embedding Benchmark — Enevoldsen et al., ICLR 2025(arXiv:2502.13595)。https://arxiv.org/abs/2502.13595
  7. Lost in the Middle: How Language Models Use Long Contexts — Liu et al., TACL 2024。https://arxiv.org/abs/2307.03172
  8. RULER: What's the Real Context Size of Your Long-Context Models? — Hsieh et al., COLM 2024。https://arxiv.org/abs/2404.06654
  9. NoLiMa: Long-Context Evaluation Beyond Literal Matching — Modarressi et al., ICML 2025。https://arxiv.org/abs/2502.05167
  10. Needle in a Haystack — Pressure Testing LLMs — gkamradt/LLMTest_NeedleInAHaystack。https://github.com/gkamradt/LLMTest_NeedleInAHaystack
  11. Knowledge Graph-Enhanced RAG for Customer Service QA(LinkedIn 案例)— ZenML LLMOps Database。https://zenml.io/llmops-database/knowledge-graph-enhanced-rag-for-customer-service-question-answering
  12. RAG Examples(DoorDash / Bell / HBS / Vimeo / Grab / Pinterest 汇总)— Evidently AI。https://evidently.ai/blog/rag-examples
  13. Glean — 企业 AI 平台官网(厂商宣称数据)。https://www.glean.com
  14. Qdrant — 向量数据库(LinkedIn 案例使用)。https://qdrant.tech/

RAG · Indexing, Chunking, Retrieval, Reranking & Context Assembly

1. Introduction

1.1. Background

Retrieval-Augmented Generation (RAG) emerged to solve the most critical problem of deploying large models in enterprises: the model's knowledge is parameterized, static, and untraceable.

The basic idea of RAG is simple: first retrieve passages relevant to the question from external corpora, then feed those passages to the model as context to generate an answer. But in engineering practice, RAG has an extremely high failure rate, and the failure modes are subtle — answers look fluent and professional, yet the retrieved passages are actually unrelated to the question, or the relevant passages are buried in the middle of the context and never used.

The sources of RAG failure, ranked by impact, are usually:

  1. Chunking: semantic units get severed, making the passages themselves unintelligible;
  2. Retrieval: out-of-domain embedding models, or pure vector retrieval that misses keyword hits;
  3. Assembly: retrieval is correct but placement is wrong, or the context exceeds budget;
  4. Generation: the model free-wheels beyond the context.

The first three items all fall within the scope of Harness engineering; only the fourth is directly related to model capability. This is precisely the core thesis of this direction: most of the room for improving RAG quality lies in the engineering layer, not the model layer.

1.2. Definition

RAG (Retrieval-Augmented Generation): a technical system that, before generating an answer, first retrieves relevant passages from external corpora and assembles them into the context, grounding the generation process on traceable evidence.

A complete RAG pipeline consists of five stages:

StageInput → OutputKey Decisions
ChunkingRaw document → passage setChunk size, overlap ratio, chunking strategy (recursive/semantic/hierarchical)
IndexingPassage → vector index + keyword indexEmbedding model, dimensions, quantization, metadata fields
RetrievalQuery → candidate setDense/sparse/hybrid strategy, fusion algorithm, Top-N
RerankingCandidate set → ordered candidate setReranker model, candidate count, threshold
AssemblyOrdered candidate set → contextTop-K, budget control, placement order, citation annotation

1.3. Positioning in the AI Harness System

图 1-1|RAG 六层能力模型:主层 L1 上下文工程、次层 L5 评估观测

RAG 在 Harness 六层能力模型中的定位 该映射为本文分析 · 示意:基于本文分析绘制 L1 上下文工程层 主层 · 瓶颈所在 切分、索引、召回、重排、装配全部环节 · 上下文预算、关键信息首尾放置 L2 工具与执行层 语料连接器 · 脚本化切分与跑分 · MCP 连接器、scripts/ L3 编排与控制层 多阶段检索:改写查询 → 多路召回 → 融合 → 重排 · 流水线编排 L4 记忆与状态层 索引版本 · 查询缓存 · 会话历史 · 索引快照、KV Cache 复用 L5 评估与观测层 次层 检索评测与生成评测 · RAGAS、BEIR、MTEB/MMTEB、Trace L6 治理与安全层 权限感知检索 · 数据分级 · 审计 · 召回阶段权限过滤 结构解读:瓶颈全部在 L1 上下文工程 —— 相关片段一旦未被召回,生成层无法补救,质量改进空间绝大多数在工程层。

数据来源:基于本文分析绘制的示意图。

In the Harness six-layer capability model, RAG is positioned as: primary layer L1 (Context Engineering), secondary layer L5 (Evaluation & Observability) (this mapping is the analysis of this document).

LayerHow It Manifests in RAGKey Mechanisms
L1 Context Engineering (primary)All stages of chunking, indexing, retrieval, reranking, and assemblyContext budget, budget-trimming priority, placing key information at the head/tail
L2 Tools & ExecutionCorpus connectors, scripted chunking and benchmarkingMCP connectors, scripts/
L3 Orchestration & ControlMulti-stage retrieval (query rewriting → multi-recall → fusion → reranking)Retrieval pipeline orchestration
L4 Memory & StateIndex versions, query cache, session historyIndex snapshots, KV Cache reuse
L5 Evaluation & Observability (secondary)Retrieval evaluation and generation evaluationRAGAS, BEIR, MTEB/MMTEB, Trace
L6 Governance & SecurityPermission-aware retrieval, data classification, auditingPermission filtering at the retrieval stage

Bottleneck layer: RAG's bottlenecks are entirely in L1. The reasons are:

  1. The essence of RAG is "deciding what the model sees," which is precisely the definition of L1's responsibility;
  2. All of RAG's tunable parameters (chunking, embedding, fusion, reranking, Top-K, budget) are context-engineering parameters;
  3. The vast majority of RAG failures cannot be fixed at the generation layer — once a relevant passage is not retrieved, even the strongest model can do nothing about it.

Why long context cannot replace RAG: this is the most misunderstood question in this direction. Three authoritative evidence chains give a negative answer:

  • Lost in the Middle (Liu et al., TACL 2024): in multi-document QA, fixing the document count and moving the position of the answer-bearing document yields a U-shaped curve — accuracy is highest when the answer is at the head or tail and drops significantly in the middle; in some configurations, burying relevant information in the middle performs even worse than the model's closed-book performance, i.e., the retrieved context is a net negative.
  • RULER (Hsieh et al., NVIDIA, COLM 2024): the original paper criticizes NIAH as being "indicative of only a superficial form of long-context understanding"; among the 17 long-context models evaluated, only about half maintain satisfactory performance at 32K length.
  • NoLiMa (Modarressi et al., ICML 2025): after rewriting the needle to minimize literal overlap, among 13 models claiming support for ≥128K, 11 dropped below half of their own short-context accuracy at 32K; GPT-4o fell from 99.3% to 69.7%.

The boundary in reverse (when not to use RAG): Anthropic officially gives a rule of great engineering value — "If your knowledge base is smaller than 200,000 tokens (about 500 pages of material), you can just include the entire knowledge base in the prompt... with no need for RAG or similar methods."

Value: RAG is the most mature, most measurable, and most optimizable implementation in the L1 Context Engineering layer. It converts the uncontrollable question of "what the model knows" into the controllable question of "what we let the model see."

1.4. Technical Landscape & Key Parameters

Source reliability note: the quantitative values in 1.4.1 and 1.4.2 mostly come from SEO/aggregation content sites and have not been traced back to original sources such as Chroma Research, Weaviate, NVIDIA FinanceBench, OpenAI's official pricing pages, or the MTEB official leaderboard. They are therefore all marked [To be verified] and must not be directly cited as design basis; they can only serve as clues for tuning direction. Section 1.4.3 (Anthropic Contextual Retrieval) is first-party official data (R1) and can be cited directly.

1.4.1. Chunking
Parameter / ConclusionValueSource Level
Production default baselineRecursive character splitting, 400~512 tokens + 10%~20% overlap
Overlap ratio heuristics10%~20% of chunk size is the common default; NVIDIA FinanceBench found 15% optimal with 1024-token chunks; beyond 30%, returns diminish quickly while indexing grows linearly
Chunk size by query typeFactual queries 256~512 tokens; analytical/multi-hop queries 512~1024 tokens; mixed-load starting point 400~512 tokens; 500-token chunks suggest 50~100 tokens overlap
Recursive vs semantic (recall measure)Chroma research: recursive character splitting reaches 85%~90% recall at 400 tokens; semantic chunking reaches 91%~92% recall (at the cost of one embedding call per sentence during chunking)
Recursive vs semantic (end-to-end measure)A February 2026 benchmark across 50 academic papers: recursive 512-token chunking achieves 69% end-to-end accuracy, semantic chunking 54% (a 15-percentage-point gap), original source not found
Upper bound of chunking-method gapWeaviate 2025 benchmark: with the same corpus, embedding model, and retriever, the recall gap between the best and worst chunking methods is at most 9 percentage points
Hierarchical / parent-child chunkingSmall "child chunks" for retrieval, large "parent chunks" fed to the LLM; typical config child 256 / parent 2048; Amazon Bedrock Knowledge Bases supports two-level hierarchical chunking (note: when multiple child chunks hit the same parent chunk, the returned count can be lower than requested)
Assembled context upper boundKeep assembled context per call below 8K tokens; some analyses point to a "context cliff" near 2,500 tokens
Mainstream library defaultsLlamaIndex SentenceSplitter defaults to chunk_size=1024, chunk_overlap=200; LangChain recommends RecursiveCharacterTextSplitter as the general default; OpenAI Assistants defaults to 800 tokens + 400 tokens overlap (performs poorly in Chroma evaluations)

Analysis conclusion suitable for the main text: chunking is the first source of RAG failure, but its returns depend heavily on the corpus — parameters must be determined by measuring Recall@K on your own golden dataset, not by copying defaults. Note that "semantic chunking has better recall" and "semantic chunking has lower end-to-end accuracy" in the table point in opposite directions, which precisely shows that conclusions can reverse under different measures, further confirming the necessity of "must measure."

1.4.2. Retrieval & Reranking
Parameter / ConclusionValueSource Level
Dense vs BM25 vs hybridKeyword-dense queries: pure dense 0.58 NDCG, pure BM25 0.88 NDCG, hybrid RRF 0.89 NDCG; complex mixed queries: hybrid RRF 0.85, rising to 0.93 after adding cross-encoder reranking
How to implement hybrid retrievalDense (vector) + Sparse (BM25) + RRF (Reciprocal Rank Fusion) fusion; RRF is parameter-free, converting raw scores to ranks before merging
Embedding models (closed-source API)text-embedding-3-large: MTEB ~64.6%, 3072 dims, 8191 context, $0.13/MTok; text-embedding-3-small: 62.3%, 1536 dims, $0.02/MTok; ada-002: 61.0%, not recommended for new projects (price and dims can be verified on the OpenAI website)
Open-source / self-hostedBGE-M3 (568M params, 1024 dims, 8192 token, triple outputs of dense+sparse+multi-vector); bge-large-zh-v1.5 (326M, 1024 dims, 512 token); Qwen3-Embedding (0.6B/4B/8B, Apache 2.0) (specs should follow the official model card)
Retrieval sub-leaderboard (C-MTEB Chinese)bge-m3 composite 71.4 / Retrieval ndcg@10 73.9; bge-large-zh-v1.5 68.6 / 71.5; bge-base-zh-v1.5 67.5 / 70.2; m3e-base 61.5 / 64.3 (changes as the leaderboard updates)
Quantization costFP16 retrieval loss <0.1%, throughput 1.5~2×; INT8 loss 0.5%~1.5%, 2~3×; INT4 loss 2%~5%, 3~4×
Production context lengthTeams typically compress production context to 16K~50K tokens/request, occasionally 80K for document analysis; RAG pipelines usually give the model only 3K~10K token high-signal slices
Top-K heuristicsFeeding 20 chunks beats 10 and 5, but there is an upper limit
1.4.3. Anthropic Contextual Retrieval (first-party official data, R1)

Source: Anthropic official blog https://www.anthropic.com/news/contextual-retrieval

ApproachTop-20 retrieval failure rateReduction vs baseline
Baseline (Embedding only)5.7%
Standard hybrid (Embedding + native BM25)5.0%Small improvement (this figure comes from a Chinese retelling; the official blog's main table does not list it →)
Contextual Embeddings3.7%35%
Contextual Embeddings + Contextual BM252.9%49%
Plus Reranking1.9%67%

The official text states: "This method can reduce the number of failed retrievals by 49% and, when combined with reranking, by 67%."

Metric definition: failure rate = 1 - recall@20, i.e., the "failure rate of not finding a relevant document in the top 20 retrieval results"; the test domains cover codebases, novels, ArXiv papers, and scientific literature.

Two-stage engineering workflow (official text):

  • Offline preprocessing (one-time): chunk into blocks of a few hundred tokens → for each block, pass the full document + that block to Claude to generate a 50~100 token context description → prepend it to the block → generate a semantic embedding for the concatenated text → build a BM25 index on the concatenated text.
  • Online inference: dual-recall with BM25 and vectors → Rank Fusion to merge and deduplicate → (optional) rerank the Top 150 candidates → take the Top 20 into the LLM.

Cost control (official text): using Prompt Caching, the full document is loaded only once and all blocks share the cache; the one-time preprocessing cost is about $1.02 per million document tokens.

This data point is the most valuable first-hand evidence in this direction: with the model unchanged, purely engineering means reduced the retrieval failure rate by 67%.

1.4.4. Evaluation Methodology

RAGAS (R1/R2, NVIDIA NeMo microservices documentation, Red Hat OpenShift AI documentation):

CategoryMetrics
Retrieval sidecontext_recall (proportion of reference information that is recalled), context_precision (relevance of the recalled blocks to the question), context_relevance, context_entity_recall
Generation sidefaithfulness (factual consistency between the answer and the recalled context), response_relevancy (requires embeddings_model), response_groundedness
Composite sideanswer_correctness (semantic similarity + factual similarity), answer_similarity, noise_sensitivity (robustness to noisy/irrelevant context)
  • Score range: context_recall ranges from 0 to 1, the higher the better.
  • Dependencies: needs a Judge LLM; some metrics additionally require Judge Embeddings.
  • Supports both offline (pre-generated answers) and online (auto-generated answers) evaluation modes.
  • Practical usage: automated quality gates (triggered after every commit or model update to prevent regression), evaluation-driven development (comparing different chunking strategies/embedding models/retrieval algorithms), factual-consistency thresholds (setting a threshold on faithfulness), production scale-out (running thousands of samples in a distributed manner under the remote provider mode).

BEIR (R2, original paper arXiv:2104.08663, Thakur et al., NeurIPS 2021 Datasets and Benchmarks):

  • 18 datasets, 9 categories of retrieval tasks: fact-checking, citation prediction, duplicate-question retrieval, argument retrieval, news retrieval, QA, tweet retrieval, bio-medical IR, entity retrieval.
  • Primary metric nDCG@10, also computing MAP, Recall@k, Precision@k, MRR (k from 1 to 1000).
  • Core methodological contribution: zero-shot evaluation — dense retrievers often beat BM25 in-domain but often lose cross-domain; BEIR exposes this generalization gap.
  • Practical reminder: some datasets require authorization or manual download (TREC-NEWS, Robust04, Signal-1M, BioASQ); the publicly runnable subset is about 15~17, and "BEIR averages" computed over different subsets are not directly comparable.

MTEB / MMTEB (R2, original paper arXiv:2210.07316, Muennighoff et al.):

  • Original MTEB: 8 task categories, 58 datasets, 112 languages; the metrics are respectively retrieval (nDCG@10), classification (accuracy), clustering (v-measure), pair classification (AP), reranking (MAP), STS (Spearman), summarization (Spearman), bitext mining (F1).
  • MMTEB (Enevoldsen et al., ICLR 2025, arXiv:2502.13595): expanded to 500+ evaluation tasks, 250+ languages; the leaderboard is split by naming (English v2 / multilingual) and now uses Borda count ranking.
  • Key advice for choosing models for RAG (R1-level methodology): "Quote the retrieval sub-score, not the overall MTEB rank." — for RAG scenarios you should look at the retrieval sub-score, not the overall rank.

2. Glossary

TermEnglish / AbbreviationDefinition
Retrieval-Augmented GenerationRAGRetrieval-Augmented Generation, a technical system that retrieves first and generates after
ChunkingChunkingThe process of splitting a document into searchable passages
Recursive character splittingRecursive Character SplittingA general strategy of splitting recursively by character level (paragraph → sentence → word)
Semantic chunkingSemantic ChunkingA strategy that splits at semantically similar boundaries, requiring embedding sentences
Parent-child chunkParent-Child ChunkA hierarchical structure where small child chunks are used for retrieval and large parent chunks are fed into generation
OverlapOverlap / Chunk OverlapContent repeated between adjacent chunks, used to mitigate semantic loss at boundaries
Dense retrievalDense RetrievalRetrieval based on vector similarity
Sparse retrievalSparse RetrievalRetrieval based on term-frequency statistics (e.g., BM25)
Hybrid searchHybrid SearchFusion of dense and sparse dual-path recall
Reciprocal rank fusionRRFReciprocal Rank Fusion, a parameter-free multi-path ranking fusion algorithm
RerankingRerankingFine-ranking the candidate set using models such as cross-encoders
Cross-encoderCross-encoderA relevance-scoring model that encodes query and document jointly
Context assemblyContext AssemblyOrganizing the reranked results into the context finally sent to the model
Context cliffContext CliffThe phenomenon where quality drops sharply once the context exceeds a certain length
RecallRecall@KThe proportion of the top K results that contain the relevant document
Normalized discounted cumulative gainnDCG@10The ranking-quality metric of the top 10 results
FaithfulnessFaithfulnessFactual consistency between generated content and the provided context
Context recallContext RecallThe proportion of reference information successfully recalled (a RAGAS metric)
Noise sensitivityNoise SensitivityRobustness to noisy/irrelevant context (a RAGAS metric)
Prompt cachingPrompt CachingReusing cached prefixes to reduce the cost of repeated input
Contextual retrievalContextual RetrievalA method that generates a context description for each block before embedding and indexing
Needle in a haystackNIAHNeedle in a Haystack, evaluating long-context retrieval capability
Lost in the middleLost in the MiddleThe phenomenon where information utilization in the middle of long context drops significantly
Permission-aware retrievalPermission-aware RetrievalAn approach that filters results by permissions at the retrieval stage
Golden datasetGolden DatasetA high-quality internally annotated evaluation set

3. Case Studies

3.1. LinkedIn: Knowledge-Graph-Enhanced Customer Service RAG

3.1.1. Background

LinkedIn's customer service team needs to handle a large number of technical questions, and historical solutions are scattered across Jira tickets. The difficulty of plain-text RAG is that the relationships between tickets (cloned from, related to, caused by) carry a great deal of semantic information, which is lost after text chunking.

3.1.2. Approach

Based on the ZenML LLMOps database and Evidently AI's summary (R2, original source is an engineering talk):

  • Build historical Jira tickets into a tree-shaped knowledge graph: nodes = Summary / Description / Priority / Steps to Reproduce / Fix Solution.
  • Explicit relationships come from Jira metadata (cloned from / related to / caused by).
  • Implicit relationships are built using E5 embedding cosine similarity + a threshold.
  • GPT-4 handles parsing and generation; Qdrant stores the vectors.
  • Deployed across multiple product lines, running in production for about 6 months.
3.1.3. Results

Based on the above source (R2):

MetricValue
MRRUp 77.6%
BLEUUp 0.32
Median resolution time per questionDown 28.6%

The engineering takeaway of this case is: the combination of structured relationships + vector retrieval captures the semantic structure of ticket-like corpora better than pure vector RAG.

3.2. DoorDash: An Evaluation Closed Loop Built on Guardrails and an LLM Judge

3.2.1. Background

DoorDash's customer service chatbot faces end users directly, so a single wrong answer causes real business loss. Its technical focus is therefore not only on "answering accurately" but also on "being able to detect when it answered incorrectly."

3.2.2. Approach

Based on Evidently AI's summary (R2, original source is the DoorDash engineering blog):

  • System architecture = RAG system + LLM Guardrail: monitoring the accuracy and compliance of every answer online.
  • The LLM Judge evaluates along five dimensions: retrieval correctness, answer accuracy, grammatical/language accuracy, context coherence, and relevance to the user's request.
3.2.3. Results

This source does not disclose specific metric-improvement figures (R2). Its value lies in providing a replicable online evaluation structure: evaluating retrieval correctness and answer accuracy separately, so that "retrieval went wrong" and "retrieval was right but generation went wrong" can be distinguished and located — this is precisely the production counterpart of the design ideas behind evaluation frameworks such as RAGAS.

3.3. Bell: A Modular Document-Embedding Pipeline with Incremental Indexing

3.3.1. Background

Bell is a Canadian telecommunications operator whose document system is large and continuously updated. A one-time full rebuild of the index is unacceptable in both cost and latency.

3.3.2. Approach

Based on Evidently AI's summary (R2, original source is a conference talk video):

  • Build a modular document-embedding pipeline supporting batch processing and incremental updates.
  • Automatically update the index when documents are added or removed, avoiding full rebuilds.
  • Each component is built and maintained as an independent service following DevOps principles.
3.3.3. Results

This source does not disclose quantitative metrics (R2). Its engineering value lies in clarifying RAG's operations-side requirements: the index is not a one-time artifact but a service that needs continuous evolution; incremental updates and component decoupling are necessary conditions for putting RAG into production.

Other reference cases (R2):

CaseKey Points
Harvard Business School · ChatLTVCorpus of about 200 documents, 15 million words (cases, teaching notes, books, blogs, course Slack historical Q&A); integrated into the course Slack channel, supporting both private and public modes
VimeoVideo transcription → processing → passage retrieval → video Q&A; needs to handle speaker identification and long-context retrieval problems
GrabReport Summarizer calls the Data-Arks API to generate tabular data which the LLM then summarizes; saves about 3~4 hours per report
PinterestTable-description vector index → semantic retrieval of candidate tables → LLM selects a refined subset → generates SQL
Glean (enterprise AI platform)Vendor claims: 110 hours/user/year saved, 20% fewer internal support tickets, 93% enterprise adoption within 2 years, ROI period <6 months, 30% lower token usage compared to off-the-shelf MCP tools

The Glean figures are vendor claims and must be labeled when cited.


4. Practical Standards

4.1. AGENTS.md Specification

The following is the full, copyable industry-standard AGENTS.md for the RAG direction, reflecting the specialized toolchain of vector stores, chunkers, embedding models, rerankers, and evaluation frameworks.

# AGENTS.md —— RAG(检索增强生成)

## 角色与边界
- 你是检索增强智能体,负责切分、索引、召回、重排、装配与评测。
- 你可以:读写索引、执行检索与重排、调整检索参数并提交评测、生成评测报告、调用脚本做跑分。
- 你不可以:在未见评测结果的情况下把参数变更推到生产、绕过权限过滤、
  把评测集混入索引语料、删除历史索引快照、编造评测数值。
- 判定原则:任何影响检索结果的变更,必须先评测后上线。

## 环境假设
- 运行环境提供:向量库、BM25 索引、嵌入服务、重排服务、切分器、
  评测框架(RAGAS / 自有黄金集)、对象存储(索引快照)、Trace 与审计日志。
- 语料具备版本/快照标识;索引具备版本号,可与语料版本对应。
- 检索服务在召回阶段即完成权限过滤(permission-aware retrieval)。
- 具备上下文预算读数(已装配 Token 数 / 上限)。

## 上下文加载顺序(Context Budget)
1. 查询与任务契约(常驻,不压缩)
2. 检索配置:切分参数、嵌入模型、融合方式、重排模型、Top-K(常驻)
3. 候选集(重排后的 Top-K,附 doc_id + 版本 + 片段定位)
4. 会话历史(多轮时,压缩为要点)
5. 参考示例(按需)
- 装配上限:默认低于 8K tokens;关键信息置于首部或尾部,不得埋在中部。
- 关键片段优先,低相关片段先被裁剪。

## 工具契约
- 切分器:递归 / 语义 / 层次化;参数必须由自有黄金集实测确定,不得照搬默认值。
- 向量库:写入前校验幂等;索引变更生成新版本快照;旧快照保留可回溯。
- 检索:默认向量 + BM25 混合召回 + RRF 融合;纯向量检索不得作为生产默认。
- 重排:候选集进入上下文前做 cross-encoder 重排;重排阈值写入配置。
- 评测:RAGAS(context_recall / context_precision / faithfulness / answer_correctness /
  noise_sensitivity)+ 自有黄金集;需要 Judge LLM 与 Judge Embeddings。
- 脚本:切分、跑分、批量评测、格式转换必须调用 scripts/,不得用生成方式替代。

## 任务执行流程(SOP)
1. 定界:语料白名单与版本、密级、查询类型(事实型 / 分析型 / 多跳)。
2. 判定模式:语料 < 200,000 tokens 走全量装配,不做切分召回。
3. 检索:向量 + BM25 → 权限过滤 → RRF 融合 → 重排 → Top-K。
4. 装配:按预算组织上下文;关键信息首尾放置;标注来源标识。
5. 生成:逐条陈述挂来源标识;无召回支撑不输出事实性结论。
6. 评测:任何参数变更先在黄金集上跑 Recall@K 与忠实度,与基线对比。
7. 上线:指标不低于基线方可上线;记录基线变化与变更日志。
8. 回流:把失败查询与人工修正加入评测集与负样本。

## 验证与证据要求
- 证据包:来源清单(doc_id + 版本 + 片段定位)、检索记录(查询、召回数、
  重排 Top-K、权限过滤命中数)、评测报告(指标 + 基线对比)、变更日志、索引版本快照。
- 引用外部基准时须注明所跑子集与指标口径(不同 BEIR 子集的均值不可直接比较)。
- 嵌入模型选型须引用 retrieval 子分,而非 MTEB/MMTEB 总排名。

## 失败与升级策略
| 失败 | 处置 |
|---|---|
| 检索零命中 | 输出"未检索到可靠依据";检查索引版本与查询改写;禁止自由生成 |
| 召回正确但答案错误 | 检查装配位置与上下文预算;提高忠实度阈值 |
| 参数变更导致指标回退 | 回退上一配置;重新评测;记录归因 |
| 权限过滤命中异常 | 停止检索,记录审计事件,升级至权限管理员 |
| 索引与语料版本不匹配 | 暂停服务,重新索引并校验 |
| 评测不可比 | 固定子集、模型版本与随机种子;注明口径 |
- 每个循环必须有步数上限与 Token 预算上限;超限即停并升级。

## 安全与合规红线
- 不得绕过权限过滤;不得使用高权限账号做检索。
- 不得把评测集混入索引语料造成数据污染。
- 不得删除历史索引快照;不得编造评测数值。
- 未经评测通过的检索配置不得推到生产。

## 禁止事项
- 禁止照搬默认切分参数(400~512 tokens + 10%~20% 重叠仅为线索,须实测)。
- 禁止把 `[待核实]` 的聚合站数值当作设计依据。
- 禁止用纯向量检索作为生产默认。
- 禁止在无召回支撑时输出事实性结论。
- 禁止编造 doc_id、版本号、评测指标与 URL。
- 禁止使用 XX / XXX / ___ 等非标准占位符(统一用 [待填写] / [待核实])。
- 禁止使用 emoji 与署名。

## 输出格式
- 检索结果:片段内容 + 来源标识 + 相似度/重排分 + 用于哪一论点。
- 评测报告:指标 / 基线 / 当前值 / 差值 / 是否通过门控 / 口径说明。
- 数值带单位;范围用 ~ 连接;中文全角标点;中英文之间加空格。

## 评估与自检
- 九项自检:来源可回溯 / 无无源断言 / 数值一致 / 无非标占位符 / 数量与表格一致 /
  编号可核实 / 密级正确 / 评测已跑且可比 / 信息缺口已声明。
- 每次索引或模型变更后重跑评测集;指标回退视为缺陷。
- 失败查询与人工修正必须回流进评测集。

4.2. SKILL.md Specification

The following is the full, copyable industry-standard SKILL.md for the RAG direction.

---
name: rag-eval
description: 构建 RAG 评测集并在切分/嵌入/融合/重排/装配变更后跑分,产出与基线的对比报告与上线门控结论。当用户要求"评测这套检索""对比两种切分策略""换嵌入模型后效果如何""生成 RAG 回归报告""为什么这条查询召回不到"时触发。
version: 1.0
created: 2026-09-12
---

# RAG 评测与调优(RAG Evaluation & Tuning)

## 适用场景
- 构建与维护黄金集(Golden Dataset)与负样本集。
- 切分参数、嵌入模型、融合方式、重排模型、Top-K 的对比评测。
- 索引或语料大版本更新后的回归验证。
- 单条查询召回失败的归因分析。

## 前置条件
- 已有基线指标(Recall@K、context_recall、faithfulness 等)与对应的配置快照。
- 评测集版本固定,且未混入索引语料。
- 运行环境提供 Judge LLM 与 Judge Embeddings。
- 已确定指标口径与门控阈值。

## 输入
- 待评测配置:切分参数 / 嵌入模型 / 融合方式 / 重排模型 / Top-K / 装配预算
- 评测集版本与门控阈值
- 可选:失败查询清单(用于归因分析)

## 输出
- 评测报告:指标 / 基线 / 当前值 / 差值 / 是否通过门控 / 口径说明
- 归因分析:失败查询归类(切分问题 / 召回问题 / 重排问题 / 装配问题)
- 上线建议:通过 / 回退 / 需人工裁决
- 评测集更新建议:新增负样本与边界用例

## 执行步骤
1. 固定变量:一次只变更一个因素,其余保持不变。
2. 跑检索侧指标:Recall@K、context_recall、context_precision、context_entity_recall。
3. 跑生成侧指标:faithfulness、response_relevancy、response_groundedness。
4. 跑综合与鲁棒性:answer_correctness、noise_sensitivity。
5. 与外部基准对齐(可选):BEIR 子集(注明子集与 nDCG@10 口径)、
   MTEB/MMTEB retrieval 子分(**引用子分,不引用总排名**)。
6. 对比基线,输出是否通过门控;未通过则回退并归因。
7. 把失败查询归类:切分 / 召回 / 重排 / 装配,并加入评测集。
8. 产出报告与变更日志;记录索引版本快照。

## 质量标准(DoD)
- 一次只变更一个变量,配置快照完整可复现。
- 指标口径明确;引用外部基准时注明子集。
- 与基线的差值、统计窗口、样本量均已披露。
- 失败查询已归类并进入评测集。
- 通过门控才可给出上线建议。

## 常见失败与处理
| 失败 | 根因 | 处置 |
|---|---|---|
| 指标不可比 | 子集/口径/模型版本变化 | 固定变量重跑;注明口径 |
| 检索好但生成差 | 装配位置或预算问题 | 关键信息首尾放置;压缩低相关片段 |
| 生成好但检索差 | 评测集过易或样本偏差 | 补充难样本与负样本 |
| 评测结果不稳定 | Judge 模型/随机性 | 固定 Judge 模型与种子;多次取均值 |
| 线上与离线不一致 | 索引版本不匹配 | 校验线上索引版本与评测时一致 |
| 换嵌入模型无提升 | 引用了总排名而非 retrieval 子分 | 改看 retrieval 子分与域内召回 |

## 示例
用户请求:把切分从 512 tokens/15% 重叠改为 1024 tokens/20% 重叠,评估是否可上线。
执行:
1. 固定嵌入模型、融合方式、重排模型、Top-K=20、装配预算 8K tokens。
2. 在黄金集(v1.3,300 条)上跑 Recall@20、context_recall、context_precision。
3. 跑生成侧 faithfulness 与 answer_correctness(Judge 模型固定)。
4. 对比基线:Recall@20 由 0.81 变为 0.79(-0.02),faithfulness 由 0.92 变为 0.91。
5. 结论:未通过门控(Recall 回退),建议回退 512/15% 配置。
6. 归因:长块稀释了事实型查询的信号;把 12 条失败查询加入评测集。
7. 输出报告:指标表 + 口径说明 + 回退建议 + 变更日志。
约束:禁止在未跑评测的情况下给出"建议上线"结论;禁止引用聚合站数值作为依据。

4.3. Implementation Checklist

#Check ItemPass CriteriaFrequency
1Small-corpus exemption decidedWhen the corpus is < 200,000 tokens, use full assembly and do not build an indexProject start
2Chunking parameters measuredParameters set by Recall@K on your own golden dataset, not copied from defaultsEvery parameter decision
3Index version traceableIndex has a version number corresponding one-to-one with the corpus versionEvery index
4Hybrid recall is the defaultVector + BM25 + RRF fusion; pure vector is not the production defaultEvery release
5Reranking enabledCandidate set is cross-encoder reranked before entering contextEvery release
6Permission filtering at retrieval stageResults already passed permission filtering, not post-hoc maskingEvery retrieval
7Assembly budget controlledAssembled context below 8K tokens; key information placed at the head or tailEvery call
8Source identifiers completeEach passage carries doc_id + version + passage locationEvery delivery
9Evaluation run and comparableFixed variables, fixed subsets, fixed measureEvery change
10Gate setThresholds exist for faithfulness and Recall@K; if not met, do not releaseEvery change
11Cite sub-score, not overall rankModel selection looks at the MTEB/MMTEB retrieval sub-scoreEvery selection
12Evaluation set unpollutedEvaluation set is not mixed into the indexed corpusEvery index
13Incremental indexing availableSupports incremental updates for added/removed documents without full rebuildsOngoing
14Failures recycledFailed queries are categorized and enter the evaluation set and negative samplesWeekly
15Deterministic operations scriptedChunking, benchmarking, and batch evaluation go through scripts/Every execution

5. Summary

RAG is the direction in the knowledge-collaboration group with the highest technical density, the most complete evaluation system, and the most easily underestimated engineering complexity.

The five most important conclusions of this direction:

  1. The returns from engineering means can be quantified and are enormous. Anthropic's official data shows Contextual Retrieval lowers the Top-20 retrieval failure rate from 5.7% to 2.9% (-49%), and to 1.9% (-67%) when combined with reranking, while the one-time preprocessing cost is only about $1.02 per million document tokens (R1). This was achieved with the model unchanged.
  2. Long context is not a substitute for RAG. The U-shaped effect of Lost in the Middle, RULER's evaluation of 17 long-context models (only about half maintain satisfactory performance at 32K), and NoLiMa's evaluation of 13 models supporting ≥128K (11 drop below half of their own short-context accuracy at 32K) together show that piling on context can produce a net negative return.
  3. Small corpora should go directly and entirely into the prompt. Anthropic officially states: < 200,000 tokens (about 500 pages) requires no RAG. This rule can significantly reduce complexity for small teams.
  4. Evaluation is a release gate, not an after-the-fact report. RAGAS measures the retrieval side and the generation side separately, so that "retrieval went wrong" and "generation went wrong" can be distinguished and located; BEIR emphasizes zero-shot cross-domain evaluation, and MTEB/MMTEB emphasizes citing the retrieval sub-score rather than the overall rank.
  5. Chunking is the first source of failure, but there is no universal optimum. In public data, "semantic chunking has better recall" and "semantic chunking has lower end-to-end accuracy" point in opposite directions, showing that you must measure on your own golden dataset.

We must candidly point out that, in the parameter space of chunking and retrieval for this direction, the vast majority of public figures come from SEO/aggregation content sites and have not been traced back to Chroma Research, Weaviate, NVIDIA FinanceBench, OpenAI's official pricing pages, or the MTEB official leaderboard. These figures have been fully marked [To be verified] in 1.4.1 and 1.4.2 and should not be used as design basis; they can only serve as clues for tuning direction. The only truly trustworthy quantitative anchors are the Anthropic Contextual Retrieval set (R1).

Information-Gap Statement

#GapStatus
1Formal ISO/IEEE standard for RAGNo authoritative standard/specification yet
2Chroma Research's official chunking evaluation reportOriginal page not obtained; related figures
3Weaviate 2025 official chunking benchmarkOriginal page not obtained; related figures
4OpenAI official embedding model pricing and dimensionsOfficial pricing page not traced back; related figures
5Current values on the MTEB / C-MTEB official leaderboardOfficial leaderboard not traced back; related figures
6Original source of the NVIDIA FinanceBench chunking experimentNot obtained
7Original source of the "February 2026 benchmark across 50 papers: recursive 69% vs semantic 54%"No result
8Original statistical source for production context length 16K~50K tokens
9Original research on the 2,500-token "context cliff"
10Original research on Top-K heuristics (20 > 10 > 5)

6. References

  1. Introducing Contextual Retrieval — Anthropic, 2024. https://www.anthropic.com/news/contextual-retrieval
  2. RAGAS Metrics — NVIDIA NeMo Microservices documentation. https://docs.nvidia.com/nemo/microservices/latest/evaluator/metrics/rag.html
  3. Evaluating AI Systems (RAGAS practical usage) — Red Hat OpenShift AI documentation. https://docs.redhat.com/en/documentation/red_hat_openshift_ai_self-managed/3.3/html-single/evaluating_ai_systems/
  4. BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models — Thakur et al., NeurIPS 2021 (arXiv:2104.08663). https://arxiv.org/abs/2104.08663
  5. MTEB: Massive Text Embedding Benchmark — Muennighoff et al. (arXiv:2210.07316). https://arxiv.org/abs/2210.07316
  6. MMTEB: Massive Multilingual Text Embedding Benchmark — Enevoldsen et al., ICLR 2025 (arXiv:2502.13595). https://arxiv.org/abs/2502.13595
  7. Lost in the Middle: How Language Models Use Long Contexts — Liu et al., TACL 2024. https://arxiv.org/abs/2307.03172
  8. RULER: What's the Real Context Size of Your Long-Context Models? — Hsieh et al., COLM 2024. https://arxiv.org/abs/2404.06654
  9. NoLiMa: Long-Context Evaluation Beyond Literal Matching — Modarressi et al., ICML 2025. https://arxiv.org/abs/2502.05167
  10. Needle in a Haystack — Pressure Testing LLMs — gkamradt/LLMTest_NeedleInAHaystack. https://github.com/gkamradt/LLMTest_NeedleInAHaystack
  11. Knowledge Graph-Enhanced RAG for Customer Service QA (LinkedIn case) — ZenML LLMOps Database. https://zenml.io/llmops-database/knowledge-graph-enhanced-rag-for-customer-service-question-answering
  12. RAG Examples (DoorDash / Bell / HBS / Vimeo / Grab / Pinterest roundup) — Evidently AI. https://evidently.ai/blog/rag-examples
  13. Glean — Enterprise AI platform official site (vendor-claimed data). https://www.glean.com
  14. Qdrant — Vector database (used in the LinkedIn case). https://qdrant.tech/