RAG · 索引、切分、召回、重排与上下文装配
1. 介绍
1.1. 背景
检索增强生成(Retrieval-Augmented Generation,RAG)的出现,是为了解决大模型在企业落地中最致命的一个问题:模型的知识是参数化的、静态的、不可追溯的。
RAG 的基本思想很朴素:先在外部语料中检索出与问题相关的片段,再把这些片段作为上下文交给模型生成答案。但在工程实践中,RAG 的失败率极高,且失败模式隐蔽——答案看起来流畅专业,实际上召回的片段与问题无关,或者相关片段被埋在上下文中部而未被利用。
RAG 的失败来源按影响程度排序,通常为:
- 切分(Chunking):语义单元被切断,导致片段本身不可理解;
- 召回(Retrieval):嵌入模型域外、纯向量召回丢失关键词命中;
- 装配(Assembly):召回正确但放置位置错误,或上下文超预算;
- 生成(Generation):模型脱离上下文自由发挥。
前三项都属于 Harness 工程范畴,只有第四项与模型能力直接相关。这正是本方向的核心论点:RAG 质量的绝大部分改进空间在工程层,而非模型层。
1.2. 定义
RAG(检索增强生成):在生成答案之前,先从外部语料中检索相关片段并装配进上下文,使生成过程建立在可追溯证据之上的技术体系。
一个完整的 RAG 管线包含五个阶段:
| 阶段 | 输入 → 输出 | 关键决策 |
|---|---|---|
| 切分 | 原始文档 → 片段集合 | 块大小、重叠比例、切分策略(递归/语义/层次) |
| 索引 | 片段 → 向量索引 + 关键词索引 | 嵌入模型、维度、量化、元数据字段 |
| 召回 | 查询 → 候选集 | 稠密/稀疏/混合策略、融合算法、Top-N |
| 重排 | 候选集 → 有序候选集 | 重排模型、候选数、阈值 |
| 装配 | 有序候选集 → 上下文 | Top-K、预算控制、放置顺序、引用标注 |
1.3. 在 AI Harness 体系中的定位
图 1-1|RAG 六层能力模型:主层 L1 上下文工程、次层 L5 评估观测
数据来源:基于本文分析绘制的示意图。
RAG 在 Harness 六层能力模型中的定位为:主层 L1 上下文工程层,次层 L5 评估与观测层(该映射为本文分析)。
| 层 | 在 RAG 中的体现 | 关键机制 |
|---|---|---|
| L1 上下文工程(主) | 切分、索引、召回、重排、装配的全部环节 | 上下文预算、预算裁剪优先级、关键信息首尾放置 |
| L2 工具与执行 | 语料连接器、脚本化切分与跑分 | MCP 连接器、scripts/ |
| L3 编排与控制 | 多阶段检索(改写查询→多路召回→融合→重排) | 检索流水线编排 |
| L4 记忆与状态 | 索引版本、查询缓存、会话历史 | 索引快照、KV Cache 复用 |
| L5 评估与观测(次) | 检索评测与生成评测 | RAGAS、BEIR、MTEB/MMTEB、Trace |
| L6 治理与安全 | 权限感知检索、数据分级、审计 | 召回阶段权限过滤 |
瓶颈所在层:RAG 的瓶颈全部在 L1。理由如下:
- RAG 的本质是"决定模型看到什么",这正是 L1 的职责定义;
- RAG 的所有可调参数(切分、嵌入、融合、重排、Top-K、预算)都是上下文工程参数;
- RAG 的失败绝大多数不可在生成层修复——一旦相关片段未被召回,再强的模型也无能为力。
为什么长上下文不能替代 RAG:这是本方向最常被误解的问题。三个权威证据链给出了否定答案:
- Lost in the Middle(Liu et al., TACL 2024):多文档问答中固定文档数、移动含答案文档的位置,得到 U 形曲线——答案在首部或尾部时准确率最高,中部显著下降;某些配置下把相关信息埋在中部,表现甚至低于该模型的闭卷(closed-book)表现,即召回的上下文是净负收益。
- RULER(Hsieh et al., NVIDIA,COLM 2024):原文批评 NIAH "is indicative of only a superficial form of long-context understanding";在评测的 17 个长上下文模型中,只有约一半能在 32K 长度维持令人满意的表现。
- NoLiMa(Modarressi et al., ICML 2025):改写 needle 以最小化字面重叠后,在 13 个宣称支持 ≥128K 的模型中,11 个在 32K 时就跌破其自身短上下文准确率的一半;GPT-4o 从 99.3% 降到 69.7%。
反过来的边界(何时不用 RAG):Anthropic 官方给出的一条极具工程价值的规则——"If your knowledge base is smaller than 200,000 tokens (about 500 pages of material), you can just include the entire knowledge base in the prompt... with no need for RAG or similar methods."
价值:RAG 是 L1 上下文工程层最成熟、最可度量、最可优化的实现。它把"模型知道什么"这个不可控问题,转化成了"我们给模型看了什么"这个可控问题。
1.4. 技术全景与关键参数
来源可靠性提示:1.4.1 与 1.4.2 中的量化数值绝大多数来自 SEO/聚合内容站,未回溯到 Chroma Research、Weaviate、NVIDIA FinanceBench、OpenAI 官方定价页、MTEB 官方榜单等原始来源,因此全部标注 ,不得作为设计依据直接引用,只能作为调优方向的线索。1.4.3 节(Anthropic Contextual Retrieval)为一手官方数据(R1),可直接引用。
1.4.1. 切分(Chunking)
| 参数/结论 | 数值 | 来源级别 |
|---|---|---|
| 生产默认基线 | 递归字符切分,400~512 tokens + 10%~20% 重叠 | |
| 重叠比例经验值 | 块大小的 10%~20% 为通行默认;NVIDIA FinanceBench 在 1024-token 块下发现 15% 最优;超过 30% 收益迅速递减而索引线性增长 | |
| 按查询类型的块大小 | 事实型查询 256~512 tokens;分析/多跳查询 512~1024 tokens;混合负载起点 400~512 tokens;500-token 块建议 50~100 tokens 重叠 | |
| 递归 vs 语义(召回口径) | Chroma 研究:递归字符切分在 400 tokens 下达 85%~90% 召回;语义切分达 91%~92% 召回(代价是切分阶段需对每个句子做一次嵌入调用) | |
| 递归 vs 语义(端到端口径) | 2026 年 2 月一份跨 50 篇学术论文的基准:递归 512-token 切分端到端准确率 69%,语义切分 54%(差 15 个百分点) | ,原始出处未找到 |
| 切分方法差距上限 | Weaviate 2025 基准:同一语料、同一嵌入模型与检索器下,最优与最差切分方法召回差距最高 9 个百分点 | |
| 层次化/父子切分 | 小"子块"用于检索、大"父块"送入 LLM;典型配置 子块 256 / 父块 2048;Amazon Bedrock Knowledge Bases 支持两级层次切分(注意:多个子块命中同一父块时返回数会少于请求数) | |
| 装配上下文上限 | 每次调用装配上下文建议低于 8K tokens;有分析指出 2,500 tokens 附近存在"上下文悬崖(context cliff)" | |
| 主流库默认值 | LlamaIndex SentenceSplitter 默认 chunk_size=1024, chunk_overlap=200;LangChain 推荐 RecursiveCharacterTextSplitter 为通用默认;OpenAI Assistants 默认 800 tokens + 400 tokens 重叠(在 Chroma 评测中表现不佳) |
可写入正文的分析结论:切分是 RAG 失败的第一来源,但其收益高度依赖语料——必须以自有黄金集实测 Recall@K 定参,不得照搬默认值。注意上表中"语义切分召回更优"与"语义切分端到端准确率更低"两组数据方向相反,这正说明不同口径下结论可能反转,进一步印证了"必须实测"的必要性。
1.4.2. 检索与重排
| 参数/结论 | 数值 | 来源级别 |
|---|---|---|
| 稠密 vs BM25 vs 混合 | 关键词密集查询:纯稠密 0.58 NDCG、纯 BM25 0.88 NDCG、混合 RRF 0.89 NDCG;复杂混合查询:混合 RRF 0.85,叠加 cross-encoder 重排后 0.93 | |
| 混合检索实现方式 | Dense(向量)+ Sparse(BM25)+ RRF(Reciprocal Rank Fusion) 融合;RRF 无参数,先把原始分转为排名再合并 | |
| 嵌入模型(闭源 API) | text-embedding-3-large:MTEB 约 64.6%、3072 维、8191 上下文、$0.13/MTok;text-embedding-3-small:62.3%、1536 维、$0.02/MTok;ada-002:61.0%,不建议新项目使用 | (价格与维度可到 OpenAI 官网核实) |
| 开源/自托管 | BGE-M3(568M 参数、1024 维、8192 token、dense+sparse+multi-vector 三路输出);bge-large-zh-v1.5(326M、1024 维、512 token);Qwen3-Embedding(0.6B/4B/8B,Apache 2.0) | (规格建议以官方模型卡为准) |
| 检索子榜(C-MTEB 中文) | bge-m3 综合 71.4 / Retrieval ndcg@10 73.9;bge-large-zh-v1.5 68.6 / 71.5;bge-base-zh-v1.5 67.5 / 70.2;m3e-base 61.5 / 64.3 | (随榜单更新会变化) |
| 量化代价 | FP16 召回损失 <0.1%、吞吐 1.5~2 倍;INT8 损失 0.5%~1.5%、2~3 倍;INT4 损失 2%~5%、3~4 倍 | |
| 生产上下文长度 | 团队通常把生产上下文压到 16K~50K tokens/请求,文档分析偶尔 80K;RAG 管线通常只给模型 3K~10K token 的高信号切片 | |
| Top-K 经验 | 送入 20 个块优于 10 个和 5 个,但存在上限 |
1.4.3. Anthropic Contextual Retrieval(一手官方数据,R1)
来源:Anthropic 官方博客 https://www.anthropic.com/news/contextual-retrieval
| 方案 | Top-20 检索失败率 | 相对基线的降低 |
|---|---|---|
| 基线(仅 Embedding) | 5.7% | — |
| 标准混合(Embedding + 原生 BM25) | 5.0% | 小幅优化(该数值来自中文转述,官方博客主表未列 → ) |
| Contextual Embeddings | 3.7% | 35% |
| Contextual Embeddings + Contextual BM25 | 2.9% | 49% |
| 再叠加 Reranking | 1.9% | 67% |
官方原文表述:"This method can reduce the number of failed retrievals by 49% and, when combined with reranking, by 67%."
指标定义:失败率 = 1 - recall@20,即"前 20 个检索结果中找不到相关文档的失败率";测试域覆盖代码库、小说、ArXiv 论文、科学文献。
两阶段工程流程(官方原文):
- 离线预处理(一次性):切分为数百 token 的块 → 对每个块向 Claude 传入完整文档 + 该块,生成 50~100 token 的上下文描述 → 前置拼接到块上 → 对拼接文本生成语义嵌入 → 对拼接文本构建 BM25 索引。
- 在线推理:BM25 与向量双路召回 → Rank Fusion 融合去重 →(可选)对 Top 150 候选做重排 → 取 Top 20 送入 LLM。
成本控制(官方原文):利用 Prompt Caching,完整文档只加载一次,各块共享缓存;预处理一次性成本约 $1.02 / 百万文档 token。
这条数据是本方向最有价值的一手证据:在模型不变的前提下,纯工程手段把检索失败率降低了 67%。
1.4.4. 评测方法学
RAGAS(R1/R2,NVIDIA NeMo 微服务文档、Red Hat OpenShift AI 文档):
| 类别 | 指标 |
|---|---|
| 检索侧 | context_recall(参考信息被召回的比例)、context_precision(召回块与问题的相关度)、context_relevance、context_entity_recall |
| 生成侧 | faithfulness(回答与召回上下文的事实一致性)、response_relevancy(需 embeddings_model)、response_groundedness |
| 综合侧 | answer_correctness(语义相似度 + 事实相似度)、answer_similarity、noise_sensitivity(对噪声/无关上下文的鲁棒性) |
- 分值区间:
context_recall为 0~1,越高越好。 - 依赖:需要一个 Judge LLM;部分指标额外需要 Judge Embeddings。
- 支持离线(预生成回答)与在线(自动生成回答)两种评测模式。
- 落地用法:自动化质量门(每次 commit 或模型更新后触发,防止回归)、评估驱动开发(对比不同切分策略/嵌入模型/检索算法)、事实一致性阈值(对 faithfulness 设阈值)、生产规模化(remote provider 模式下分布式跑数千条样本)。
BEIR(R2,原始论文 arXiv:2104.08663,Thakur et al., NeurIPS 2021 Datasets and Benchmarks):
- 18 个数据集、9 类检索任务:fact-checking、citation prediction、duplicate-question retrieval、argument retrieval、news retrieval、QA、tweet retrieval、bio-medical IR、entity retrieval。
- 主指标 nDCG@10,同时计算 MAP、Recall@k、Precision@k、MRR(k 从 1 到 1000)。
- 核心方法论贡献:零样本(zero-shot)评估——稠密检索器在域内常优于 BM25,跨域则常落败,BEIR 暴露了这一泛化鸿沟。
- 实用提醒:部分数据集需授权或手动下载(TREC-NEWS、Robust04、Signal-1M、BioASQ),公开可跑子集约 15~17 个,不同子集算出的"BEIR 均值"不可直接比较。
MTEB / MMTEB(R2,原始论文 arXiv:2210.07316,Muennighoff et al.):
- 原始 MTEB:8 类任务、58 个数据集、112 种语言;指标分别为 retrieval(nDCG@10)、classification(accuracy)、clustering(v-measure)、pair classification(AP)、reranking(MAP)、STS(Spearman)、summarization(Spearman)、bitext mining(F1)。
- MMTEB(Enevoldsen et al., ICLR 2025,arXiv:2502.13595):扩展至 500+ 评测任务、250+ 语言;榜单按命名切分(English v2 / multilingual)并改用 Borda 计数排名。
- 对 RAG 选模型的关键建议(R1 级方法论):"Quote the retrieval sub-score, not the overall MTEB rank."——RAG 场景应看 retrieval 子分,而非总排名。
2. 名词解释
| 术语 | 英文/缩写 | 释义 |
|---|---|---|
| 检索增强生成 | RAG | Retrieval-Augmented Generation,先检索后生成的技术体系 |
| 切分 | Chunking | 把文档切为可检索片段的过程 |
| 递归字符切分 | Recursive Character Splitting | 按字符层级(段落→句子→词)递归切分的通用策略 |
| 语义切分 | Semantic Chunking | 按语义相似度边界切分的策略,需对句子做嵌入 |
| 父子块 | Parent-Child Chunk | 小子块用于检索、大父块送入生成的层次化结构 |
| 重叠 | Overlap / Chunk Overlap | 相邻块之间重复的内容,用于缓解边界语义丢失 |
| 稠密检索 | Dense Retrieval | 基于向量相似度的检索 |
| 稀疏检索 | Sparse Retrieval | 基于词频统计(如 BM25)的检索 |
| 混合检索 | Hybrid Search | 稠密与稀疏两路召回的融合 |
| 倒数排名融合 | RRF | Reciprocal Rank Fusion,无参数的多路排名融合算法 |
| 重排 | Reranking | 用 cross-encoder 等模型对候选集精排序 |
| 交叉编码器 | Cross-encoder | 同时编码查询与文档的相关性打分模型 |
| 上下文装配 | Context Assembly | 把重排结果组织为最终送入模型的上下文 |
| 上下文悬崖 | Context Cliff | 上下文超过某一长度后质量急剧下降的现象 |
| 召回率 | Recall@K | 前 K 个结果中包含相关文档的比例 |
| 归一化折损累积增益 | nDCG@10 | 前 10 个结果的排序质量指标 |
| 忠实度 | Faithfulness | 生成内容与所提供上下文的事实一致性 |
| 上下文召回 | Context Recall | 参考信息被成功召回的比例(RAGAS 指标) |
| 噪声敏感度 | Noise Sensitivity | 对噪声/无关上下文的鲁棒性(RAGAS 指标) |
| 提示词缓存 | Prompt Caching | 复用已缓存前缀以降低重复输入的成本 |
| 上下文检索 | Contextual Retrieval | 为每个块生成上下文描述后再嵌入与索引的方法 |
| 干草堆找针 | NIAH | Needle in a Haystack,长上下文检索能力评测 |
| 中间迷失 | Lost in the Middle | 长上下文中部信息利用率显著下降的现象 |
| 权限感知检索 | Permission-aware Retrieval | 在召回阶段按权限过滤结果的检索方式 |
| 黄金集 | Golden Dataset | 组织内部标注的高质量评测集 |
3. 案例
3.1. LinkedIn:知识图谱增强的客服 RAG
3.1.1. 背景
LinkedIn 的客服团队需要处理大量技术性问题,历史解决方案散落在 Jira 工单中。纯文本 RAG 的难点在于:工单之间的关系(克隆自、关联、导致)承载了大量语义信息,而这些信息在文本切分后会丢失。
3.1.2. 方案
依据 ZenML LLMOps 数据库与 Evidently AI 的汇总(R2,原始来源为工程分享):
- 将历史 Jira 工单构建为树状知识图谱:节点 = Summary / Description / Priority / Steps to Reproduce / Fix Solution。
- 显式关系来自 Jira 元数据(cloned from / related to / caused by)。
- 隐式关系用 E5 嵌入余弦相似度 + 阈值建立。
- GPT-4 负责解析与生成;Qdrant 存储向量。
- 已部署至多条产品线,生产运行约 6 个月。
3.1.3. 效果
依据上述来源(R2):
| 指标 | 数值 |
|---|---|
| MRR | 提升 77.6% |
| BLEU | 提升 0.32 |
| 每问题中位解决时间 | 降低 28.6% |
该案例的工程启示是:结构化关系 + 向量检索的组合,比纯向量 RAG 更能捕捉工单类语料的语义结构。
3.2. DoorDash:护栏与 LLM Judge 构成的评测闭环
3.2.1. 背景
DoorDash 的客服聊天机器人直接面向终端用户,一次错误回答会造成实际业务损失。因此其技术重点不只是"答得准",还包括"能发现答得不准"。
3.2.2. 方案
依据 Evidently AI 汇总(R2,原始来源为 DoorDash 工程博客):
- 系统架构 = RAG 系统 + LLM Guardrail:在线监控每一条回答的准确性与合规性。
- LLM Judge 从五个维度评测:检索正确性、回答准确性、语法语言准确性、上下文连贯性、与用户请求的相关性。
3.2.3. 效果
该来源未披露具体的指标提升数值(R2)。其价值在于给出了一套可复制的线上评测结构:把检索正确性与回答准确性分开评测,使得"检索错了"与"检索对了但生成错了"可以被区分定位——这正是 RAGAS 等评测框架的设计思想在生产中的对应实现。
3.3. Bell:模块化文档嵌入管线与增量索引
3.3.1. 背景
Bell 是加拿大电信运营商,其文档体系庞大且持续更新。一次性全量重建索引在成本与时延上均不可接受。
3.3.2. 方案
依据 Evidently AI 汇总(R2,原始来源为会议演讲视频):
- 构建模块化文档嵌入管线,支持批处理与增量更新。
- 文档增删时自动更新索引,避免全量重建。
- 每个组件作为独立服务,按 DevOps 原则建设与维护。
3.3.3. 效果
该来源未披露量化指标(R2)。其工程价值在于明确了 RAG 的运维侧要求:索引不是一次性产物,而是需要持续演进的服务;增量更新与组件解耦是 RAG 上生产的必要条件。
其他可参考案例(R2):
| 案例 | 要点 |
|---|---|
| Harvard Business School · ChatLTV | 语料约 200 份文档、1,500 万词(案例、教学笔记、书籍、博客、课程 Slack 历史问答);集成在课程 Slack 频道,支持私聊与公开两种模式 |
| Vimeo | 视频转写 → 处理 → 片段检索 → 视频问答;需处理说话人识别与长上下文检索问题 |
| Grab | Report Summarizer 调用 Data-Arks API 生成表格数据再由 LLM 总结;每份报告节省约 3~4 小时 |
| 表描述向量索引 → 语义检索候选表 → LLM 精选子集 → 生成 SQL | |
| Glean(企业 AI 平台) | 厂商宣称:110 小时/用户/年节省、内部支持工单减少 20%、2 年内企业采用率 93%、ROI 周期 <6 个月、相比现成 MCP 工具 token 用量降低 30% |
Glean 数据为厂商宣称,引用时须标注。
4. 实践标准
4.1. AGENTS.md 规范
以下为 RAG 方向的行业标准 AGENTS.md 完整可复制原文,体现向量库、切分器、嵌入模型、重排模型、评测框架等专有工具链。
# AGENTS.md —— RAG(检索增强生成)
## 角色与边界
- 你是检索增强智能体,负责切分、索引、召回、重排、装配与评测。
- 你可以:读写索引、执行检索与重排、调整检索参数并提交评测、生成评测报告、调用脚本做跑分。
- 你不可以:在未见评测结果的情况下把参数变更推到生产、绕过权限过滤、
把评测集混入索引语料、删除历史索引快照、编造评测数值。
- 判定原则:任何影响检索结果的变更,必须先评测后上线。
## 环境假设
- 运行环境提供:向量库、BM25 索引、嵌入服务、重排服务、切分器、
评测框架(RAGAS / 自有黄金集)、对象存储(索引快照)、Trace 与审计日志。
- 语料具备版本/快照标识;索引具备版本号,可与语料版本对应。
- 检索服务在召回阶段即完成权限过滤(permission-aware retrieval)。
- 具备上下文预算读数(已装配 Token 数 / 上限)。
## 上下文加载顺序(Context Budget)
1. 查询与任务契约(常驻,不压缩)
2. 检索配置:切分参数、嵌入模型、融合方式、重排模型、Top-K(常驻)
3. 候选集(重排后的 Top-K,附 doc_id + 版本 + 片段定位)
4. 会话历史(多轮时,压缩为要点)
5. 参考示例(按需)
- 装配上限:默认低于 8K tokens;关键信息置于首部或尾部,不得埋在中部。
- 关键片段优先,低相关片段先被裁剪。
## 工具契约
- 切分器:递归 / 语义 / 层次化;参数必须由自有黄金集实测确定,不得照搬默认值。
- 向量库:写入前校验幂等;索引变更生成新版本快照;旧快照保留可回溯。
- 检索:默认向量 + BM25 混合召回 + RRF 融合;纯向量检索不得作为生产默认。
- 重排:候选集进入上下文前做 cross-encoder 重排;重排阈值写入配置。
- 评测:RAGAS(context_recall / context_precision / faithfulness / answer_correctness /
noise_sensitivity)+ 自有黄金集;需要 Judge LLM 与 Judge Embeddings。
- 脚本:切分、跑分、批量评测、格式转换必须调用 scripts/,不得用生成方式替代。
## 任务执行流程(SOP)
1. 定界:语料白名单与版本、密级、查询类型(事实型 / 分析型 / 多跳)。
2. 判定模式:语料 < 200,000 tokens 走全量装配,不做切分召回。
3. 检索:向量 + BM25 → 权限过滤 → RRF 融合 → 重排 → Top-K。
4. 装配:按预算组织上下文;关键信息首尾放置;标注来源标识。
5. 生成:逐条陈述挂来源标识;无召回支撑不输出事实性结论。
6. 评测:任何参数变更先在黄金集上跑 Recall@K 与忠实度,与基线对比。
7. 上线:指标不低于基线方可上线;记录基线变化与变更日志。
8. 回流:把失败查询与人工修正加入评测集与负样本。
## 验证与证据要求
- 证据包:来源清单(doc_id + 版本 + 片段定位)、检索记录(查询、召回数、
重排 Top-K、权限过滤命中数)、评测报告(指标 + 基线对比)、变更日志、索引版本快照。
- 引用外部基准时须注明所跑子集与指标口径(不同 BEIR 子集的均值不可直接比较)。
- 嵌入模型选型须引用 retrieval 子分,而非 MTEB/MMTEB 总排名。
## 失败与升级策略
| 失败 | 处置 |
|---|---|
| 检索零命中 | 输出"未检索到可靠依据";检查索引版本与查询改写;禁止自由生成 |
| 召回正确但答案错误 | 检查装配位置与上下文预算;提高忠实度阈值 |
| 参数变更导致指标回退 | 回退上一配置;重新评测;记录归因 |
| 权限过滤命中异常 | 停止检索,记录审计事件,升级至权限管理员 |
| 索引与语料版本不匹配 | 暂停服务,重新索引并校验 |
| 评测不可比 | 固定子集、模型版本与随机种子;注明口径 |
- 每个循环必须有步数上限与 Token 预算上限;超限即停并升级。
## 安全与合规红线
- 不得绕过权限过滤;不得使用高权限账号做检索。
- 不得把评测集混入索引语料造成数据污染。
- 不得删除历史索引快照;不得编造评测数值。
- 未经评测通过的检索配置不得推到生产。
## 禁止事项
- 禁止照搬默认切分参数(400~512 tokens + 10%~20% 重叠仅为线索,须实测)。
- 禁止把 `[待核实]` 的聚合站数值当作设计依据。
- 禁止用纯向量检索作为生产默认。
- 禁止在无召回支撑时输出事实性结论。
- 禁止编造 doc_id、版本号、评测指标与 URL。
- 禁止使用 XX / XXX / ___ 等非标准占位符(统一用 [待填写] / [待核实])。
- 禁止使用 emoji 与署名。
## 输出格式
- 检索结果:片段内容 + 来源标识 + 相似度/重排分 + 用于哪一论点。
- 评测报告:指标 / 基线 / 当前值 / 差值 / 是否通过门控 / 口径说明。
- 数值带单位;范围用 ~ 连接;中文全角标点;中英文之间加空格。
## 评估与自检
- 九项自检:来源可回溯 / 无无源断言 / 数值一致 / 无非标占位符 / 数量与表格一致 /
编号可核实 / 密级正确 / 评测已跑且可比 / 信息缺口已声明。
- 每次索引或模型变更后重跑评测集;指标回退视为缺陷。
- 失败查询与人工修正必须回流进评测集。 4.2. SKILL.md 规范
以下为 RAG 方向的行业标准 SKILL.md 完整可复制原文。
---
name: rag-eval
description: 构建 RAG 评测集并在切分/嵌入/融合/重排/装配变更后跑分,产出与基线的对比报告与上线门控结论。当用户要求"评测这套检索""对比两种切分策略""换嵌入模型后效果如何""生成 RAG 回归报告""为什么这条查询召回不到"时触发。
version: 1.0
created: 2026-09-12
---
# RAG 评测与调优(RAG Evaluation & Tuning)
## 适用场景
- 构建与维护黄金集(Golden Dataset)与负样本集。
- 切分参数、嵌入模型、融合方式、重排模型、Top-K 的对比评测。
- 索引或语料大版本更新后的回归验证。
- 单条查询召回失败的归因分析。
## 前置条件
- 已有基线指标(Recall@K、context_recall、faithfulness 等)与对应的配置快照。
- 评测集版本固定,且未混入索引语料。
- 运行环境提供 Judge LLM 与 Judge Embeddings。
- 已确定指标口径与门控阈值。
## 输入
- 待评测配置:切分参数 / 嵌入模型 / 融合方式 / 重排模型 / Top-K / 装配预算
- 评测集版本与门控阈值
- 可选:失败查询清单(用于归因分析)
## 输出
- 评测报告:指标 / 基线 / 当前值 / 差值 / 是否通过门控 / 口径说明
- 归因分析:失败查询归类(切分问题 / 召回问题 / 重排问题 / 装配问题)
- 上线建议:通过 / 回退 / 需人工裁决
- 评测集更新建议:新增负样本与边界用例
## 执行步骤
1. 固定变量:一次只变更一个因素,其余保持不变。
2. 跑检索侧指标:Recall@K、context_recall、context_precision、context_entity_recall。
3. 跑生成侧指标:faithfulness、response_relevancy、response_groundedness。
4. 跑综合与鲁棒性:answer_correctness、noise_sensitivity。
5. 与外部基准对齐(可选):BEIR 子集(注明子集与 nDCG@10 口径)、
MTEB/MMTEB retrieval 子分(**引用子分,不引用总排名**)。
6. 对比基线,输出是否通过门控;未通过则回退并归因。
7. 把失败查询归类:切分 / 召回 / 重排 / 装配,并加入评测集。
8. 产出报告与变更日志;记录索引版本快照。
## 质量标准(DoD)
- 一次只变更一个变量,配置快照完整可复现。
- 指标口径明确;引用外部基准时注明子集。
- 与基线的差值、统计窗口、样本量均已披露。
- 失败查询已归类并进入评测集。
- 通过门控才可给出上线建议。
## 常见失败与处理
| 失败 | 根因 | 处置 |
|---|---|---|
| 指标不可比 | 子集/口径/模型版本变化 | 固定变量重跑;注明口径 |
| 检索好但生成差 | 装配位置或预算问题 | 关键信息首尾放置;压缩低相关片段 |
| 生成好但检索差 | 评测集过易或样本偏差 | 补充难样本与负样本 |
| 评测结果不稳定 | Judge 模型/随机性 | 固定 Judge 模型与种子;多次取均值 |
| 线上与离线不一致 | 索引版本不匹配 | 校验线上索引版本与评测时一致 |
| 换嵌入模型无提升 | 引用了总排名而非 retrieval 子分 | 改看 retrieval 子分与域内召回 |
## 示例
用户请求:把切分从 512 tokens/15% 重叠改为 1024 tokens/20% 重叠,评估是否可上线。
执行:
1. 固定嵌入模型、融合方式、重排模型、Top-K=20、装配预算 8K tokens。
2. 在黄金集(v1.3,300 条)上跑 Recall@20、context_recall、context_precision。
3. 跑生成侧 faithfulness 与 answer_correctness(Judge 模型固定)。
4. 对比基线:Recall@20 由 0.81 变为 0.79(-0.02),faithfulness 由 0.92 变为 0.91。
5. 结论:未通过门控(Recall 回退),建议回退 512/15% 配置。
6. 归因:长块稀释了事实型查询的信号;把 12 条失败查询加入评测集。
7. 输出报告:指标表 + 口径说明 + 回退建议 + 变更日志。
约束:禁止在未跑评测的情况下给出"建议上线"结论;禁止引用聚合站数值作为依据。 4.3. 落地检查清单
| # | 检查项 | 判定标准 | 频次 |
|---|---|---|---|
| 1 | 小语料豁免已判定 | 语料 < 200,000 tokens 时走全量装配,不建索引 | 项目启动 |
| 2 | 切分参数已实测 | 以自有黄金集 Recall@K 定参,未照搬默认值 | 每次定参 |
| 3 | 索引版本可追溯 | 索引有版本号,与语料版本一一对应 | 每次索引 |
| 4 | 混合召回为默认 | 向量 + BM25 + RRF 融合;纯向量不作为生产默认 | 每次上线 |
| 5 | 重排已启用 | 候选集进入上下文前做 cross-encoder 重排 | 每次上线 |
| 6 | 权限过滤在召回阶段 | 结果已过权限过滤,非事后脱敏 | 每次检索 |
| 7 | 装配预算可控 | 装配上下文低于 8K tokens;关键信息首尾放置 | 每次调用 |
| 8 | 来源标识完整 | 每条片段附 doc_id + 版本 + 片段定位 | 每次交付 |
| 9 | 评测已跑且可比 | 固定变量、固定子集、固定口径 | 每次变更 |
| 10 | 门控已设置 | faithfulness 与 Recall@K 有阈值,未达标不上线 | 每次变更 |
| 11 | 引用子分而非总排名 | 嵌入模型选型看 MTEB/MMTEB retrieval 子分 | 每次选型 |
| 12 | 评测集未污染 | 评测集未混入索引语料 | 每次索引 |
| 13 | 增量索引可用 | 支持文档增删的增量更新,无需全量重建 | 持续 |
| 14 | 失败已回流 | 失败查询归类并进入评测集与负样本 | 每周 |
| 15 | 确定性操作已脚本化 | 切分、跑分、批量评测走 scripts/ | 每次执行 |
5. 总结
RAG 是知识协同组中技术密度最高、评测体系最完备、也最容易被低估工程复杂度的方向。
本方向最重要的五条结论:
- 工程手段的收益可以量化且巨大。Anthropic 官方数据显示,Contextual Retrieval 将 Top-20 检索失败率从 5.7% 降至 2.9%(-49%),叠加 Reranking 后降至 1.9%(-67%),而预处理一次性成本仅约 $1.02 / 百万文档 token(R1)。这是在模型不变前提下取得的。
- 长上下文不是 RAG 的替代品。Lost in the Middle 的 U 形效应、RULER 对 17 个长上下文模型的评测(仅约一半在 32K 维持满意表现)、NoLiMa 对 13 个 ≥128K 模型的评测(11 个在 32K 跌破自身短上下文准确率的一半),共同说明堆上下文可能带来净负收益。
- 小语料应直接全量入提示词。Anthropic 官方明确:< 200,000 tokens(约 500 页)无需 RAG。这条规则能显著降低小团队复杂度。
- 评测是上线门控,不是事后报告。RAGAS 把检索侧与生成侧分开度量,使得"检索错了"与"生成错了"可被区分定位;BEIR 强调零样本跨域评估,MTEB/MMTEB 强调引用 retrieval 子分而非总排名。
- 切分是第一失败来源,但无通用最优解。公开数据中"语义切分召回更优"与"语义切分端到端准确率更低"方向相反,说明必须以自有黄金集实测。
必须坦率指出:本方向在切分与检索的参数区,绝大多数公开数值来自 SEO/聚合内容站,未回溯到 Chroma Research、Weaviate、NVIDIA FinanceBench、OpenAI 官方定价页与 MTEB 官方榜单。这些数值已在 1.4.1 与 1.4.2 中全部标注 ,不应作为设计依据,只能作为调优方向线索。真正可信的量化锚点只有 Anthropic Contextual Retrieval 一组(R1)。
信息缺口声明
| # | 缺口 | 状态 |
|---|---|---|
| 1 | RAG 的 ISO/IEEE 正式标准 | 暂无权威标准/规范 |
| 2 | Chroma Research 官方切分评测报告(chunking evaluation) | 未获取到原始页面,相关数值 |
| 3 | Weaviate 2025 官方切分基准 | 未获取到原始页面,相关数值 |
| 4 | OpenAI 官方嵌入模型定价与维度 | 未回溯官方定价页,相关数值 |
| 5 | MTEB / C-MTEB 官方榜单当前值 | 未回溯官方榜单,相关数值 |
| 6 | NVIDIA FinanceBench 切分实验原始出处 | 未获取到, |
| 7 | "2026 年 2 月跨 50 篇论文基准:递归 69% vs 语义 54%" 的原始出处 | 无结果 |
| 8 | 生产上下文长度 16K~50K tokens 的原始统计来源 | |
| 9 | 2,500 tokens "上下文悬崖" 的原始研究 | |
| 10 | Top-K 经验值(20 > 10 > 5)的原始研究 |
6. 参考资料
- Introducing Contextual Retrieval — Anthropic,2024。https://www.anthropic.com/news/contextual-retrieval
- RAGAS Metrics — NVIDIA NeMo Microservices 文档。https://docs.nvidia.com/nemo/microservices/latest/evaluator/metrics/rag.html
- Evaluating AI Systems(RAGAS 落地用法)— Red Hat OpenShift AI 文档。https://docs.redhat.com/en/documentation/red_hat_openshift_ai_self-managed/3.3/html-single/evaluating_ai_systems/
- BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models — Thakur et al., NeurIPS 2021(arXiv:2104.08663)。https://arxiv.org/abs/2104.08663
- MTEB: Massive Text Embedding Benchmark — Muennighoff et al.(arXiv:2210.07316)。https://arxiv.org/abs/2210.07316
- MMTEB: Massive Multilingual Text Embedding Benchmark — Enevoldsen et al., ICLR 2025(arXiv:2502.13595)。https://arxiv.org/abs/2502.13595
- Lost in the Middle: How Language Models Use Long Contexts — Liu et al., TACL 2024。https://arxiv.org/abs/2307.03172
- RULER: What's the Real Context Size of Your Long-Context Models? — Hsieh et al., COLM 2024。https://arxiv.org/abs/2404.06654
- NoLiMa: Long-Context Evaluation Beyond Literal Matching — Modarressi et al., ICML 2025。https://arxiv.org/abs/2502.05167
- Needle in a Haystack — Pressure Testing LLMs — gkamradt/LLMTest_NeedleInAHaystack。https://github.com/gkamradt/LLMTest_NeedleInAHaystack
- Knowledge Graph-Enhanced RAG for Customer Service QA(LinkedIn 案例)— ZenML LLMOps Database。https://zenml.io/llmops-database/knowledge-graph-enhanced-rag-for-customer-service-question-answering
- RAG Examples(DoorDash / Bell / HBS / Vimeo / Grab / Pinterest 汇总)— Evidently AI。https://evidently.ai/blog/rag-examples
- Glean — 企业 AI 平台官网(厂商宣称数据)。https://www.glean.com
- Qdrant — 向量数据库(LinkedIn 案例使用)。https://qdrant.tech/
RAG · Indexing, Chunking, Retrieval, Reranking & Context Assembly
1. Introduction
1.1. Background
Retrieval-Augmented Generation (RAG) emerged to solve the most critical problem of deploying large models in enterprises: the model's knowledge is parameterized, static, and untraceable.
The basic idea of RAG is simple: first retrieve passages relevant to the question from external corpora, then feed those passages to the model as context to generate an answer. But in engineering practice, RAG has an extremely high failure rate, and the failure modes are subtle — answers look fluent and professional, yet the retrieved passages are actually unrelated to the question, or the relevant passages are buried in the middle of the context and never used.
The sources of RAG failure, ranked by impact, are usually:
- Chunking: semantic units get severed, making the passages themselves unintelligible;
- Retrieval: out-of-domain embedding models, or pure vector retrieval that misses keyword hits;
- Assembly: retrieval is correct but placement is wrong, or the context exceeds budget;
- Generation: the model free-wheels beyond the context.
The first three items all fall within the scope of Harness engineering; only the fourth is directly related to model capability. This is precisely the core thesis of this direction: most of the room for improving RAG quality lies in the engineering layer, not the model layer.
1.2. Definition
RAG (Retrieval-Augmented Generation): a technical system that, before generating an answer, first retrieves relevant passages from external corpora and assembles them into the context, grounding the generation process on traceable evidence.
A complete RAG pipeline consists of five stages:
| Stage | Input → Output | Key Decisions |
|---|---|---|
| Chunking | Raw document → passage set | Chunk size, overlap ratio, chunking strategy (recursive/semantic/hierarchical) |
| Indexing | Passage → vector index + keyword index | Embedding model, dimensions, quantization, metadata fields |
| Retrieval | Query → candidate set | Dense/sparse/hybrid strategy, fusion algorithm, Top-N |
| Reranking | Candidate set → ordered candidate set | Reranker model, candidate count, threshold |
| Assembly | Ordered candidate set → context | Top-K, budget control, placement order, citation annotation |
1.3. Positioning in the AI Harness System
图 1-1|RAG 六层能力模型:主层 L1 上下文工程、次层 L5 评估观测
数据来源:基于本文分析绘制的示意图。
In the Harness six-layer capability model, RAG is positioned as: primary layer L1 (Context Engineering), secondary layer L5 (Evaluation & Observability) (this mapping is the analysis of this document).
| Layer | How It Manifests in RAG | Key Mechanisms |
|---|---|---|
| L1 Context Engineering (primary) | All stages of chunking, indexing, retrieval, reranking, and assembly | Context budget, budget-trimming priority, placing key information at the head/tail |
| L2 Tools & Execution | Corpus connectors, scripted chunking and benchmarking | MCP connectors, scripts/ |
| L3 Orchestration & Control | Multi-stage retrieval (query rewriting → multi-recall → fusion → reranking) | Retrieval pipeline orchestration |
| L4 Memory & State | Index versions, query cache, session history | Index snapshots, KV Cache reuse |
| L5 Evaluation & Observability (secondary) | Retrieval evaluation and generation evaluation | RAGAS, BEIR, MTEB/MMTEB, Trace |
| L6 Governance & Security | Permission-aware retrieval, data classification, auditing | Permission filtering at the retrieval stage |
Bottleneck layer: RAG's bottlenecks are entirely in L1. The reasons are:
- The essence of RAG is "deciding what the model sees," which is precisely the definition of L1's responsibility;
- All of RAG's tunable parameters (chunking, embedding, fusion, reranking, Top-K, budget) are context-engineering parameters;
- The vast majority of RAG failures cannot be fixed at the generation layer — once a relevant passage is not retrieved, even the strongest model can do nothing about it.
Why long context cannot replace RAG: this is the most misunderstood question in this direction. Three authoritative evidence chains give a negative answer:
- Lost in the Middle (Liu et al., TACL 2024): in multi-document QA, fixing the document count and moving the position of the answer-bearing document yields a U-shaped curve — accuracy is highest when the answer is at the head or tail and drops significantly in the middle; in some configurations, burying relevant information in the middle performs even worse than the model's closed-book performance, i.e., the retrieved context is a net negative.
- RULER (Hsieh et al., NVIDIA, COLM 2024): the original paper criticizes NIAH as being "indicative of only a superficial form of long-context understanding"; among the 17 long-context models evaluated, only about half maintain satisfactory performance at 32K length.
- NoLiMa (Modarressi et al., ICML 2025): after rewriting the needle to minimize literal overlap, among 13 models claiming support for ≥128K, 11 dropped below half of their own short-context accuracy at 32K; GPT-4o fell from 99.3% to 69.7%.
The boundary in reverse (when not to use RAG): Anthropic officially gives a rule of great engineering value — "If your knowledge base is smaller than 200,000 tokens (about 500 pages of material), you can just include the entire knowledge base in the prompt... with no need for RAG or similar methods."
Value: RAG is the most mature, most measurable, and most optimizable implementation in the L1 Context Engineering layer. It converts the uncontrollable question of "what the model knows" into the controllable question of "what we let the model see."
1.4. Technical Landscape & Key Parameters
Source reliability note: the quantitative values in 1.4.1 and 1.4.2 mostly come from SEO/aggregation content sites and have not been traced back to original sources such as Chroma Research, Weaviate, NVIDIA FinanceBench, OpenAI's official pricing pages, or the MTEB official leaderboard. They are therefore all marked
[To be verified]and must not be directly cited as design basis; they can only serve as clues for tuning direction. Section 1.4.3 (Anthropic Contextual Retrieval) is first-party official data (R1) and can be cited directly.
1.4.1. Chunking
| Parameter / Conclusion | Value | Source Level |
|---|---|---|
| Production default baseline | Recursive character splitting, 400~512 tokens + 10%~20% overlap | |
| Overlap ratio heuristics | 10%~20% of chunk size is the common default; NVIDIA FinanceBench found 15% optimal with 1024-token chunks; beyond 30%, returns diminish quickly while indexing grows linearly | |
| Chunk size by query type | Factual queries 256~512 tokens; analytical/multi-hop queries 512~1024 tokens; mixed-load starting point 400~512 tokens; 500-token chunks suggest 50~100 tokens overlap | |
| Recursive vs semantic (recall measure) | Chroma research: recursive character splitting reaches 85%~90% recall at 400 tokens; semantic chunking reaches 91%~92% recall (at the cost of one embedding call per sentence during chunking) | |
| Recursive vs semantic (end-to-end measure) | A February 2026 benchmark across 50 academic papers: recursive 512-token chunking achieves 69% end-to-end accuracy, semantic chunking 54% (a 15-percentage-point gap) | , original source not found |
| Upper bound of chunking-method gap | Weaviate 2025 benchmark: with the same corpus, embedding model, and retriever, the recall gap between the best and worst chunking methods is at most 9 percentage points | |
| Hierarchical / parent-child chunking | Small "child chunks" for retrieval, large "parent chunks" fed to the LLM; typical config child 256 / parent 2048; Amazon Bedrock Knowledge Bases supports two-level hierarchical chunking (note: when multiple child chunks hit the same parent chunk, the returned count can be lower than requested) | |
| Assembled context upper bound | Keep assembled context per call below 8K tokens; some analyses point to a "context cliff" near 2,500 tokens | |
| Mainstream library defaults | LlamaIndex SentenceSplitter defaults to chunk_size=1024, chunk_overlap=200; LangChain recommends RecursiveCharacterTextSplitter as the general default; OpenAI Assistants defaults to 800 tokens + 400 tokens overlap (performs poorly in Chroma evaluations) |
Analysis conclusion suitable for the main text: chunking is the first source of RAG failure, but its returns depend heavily on the corpus — parameters must be determined by measuring Recall@K on your own golden dataset, not by copying defaults. Note that "semantic chunking has better recall" and "semantic chunking has lower end-to-end accuracy" in the table point in opposite directions, which precisely shows that conclusions can reverse under different measures, further confirming the necessity of "must measure."
1.4.2. Retrieval & Reranking
| Parameter / Conclusion | Value | Source Level |
|---|---|---|
| Dense vs BM25 vs hybrid | Keyword-dense queries: pure dense 0.58 NDCG, pure BM25 0.88 NDCG, hybrid RRF 0.89 NDCG; complex mixed queries: hybrid RRF 0.85, rising to 0.93 after adding cross-encoder reranking | |
| How to implement hybrid retrieval | Dense (vector) + Sparse (BM25) + RRF (Reciprocal Rank Fusion) fusion; RRF is parameter-free, converting raw scores to ranks before merging | |
| Embedding models (closed-source API) | text-embedding-3-large: MTEB ~64.6%, 3072 dims, 8191 context, $0.13/MTok; text-embedding-3-small: 62.3%, 1536 dims, $0.02/MTok; ada-002: 61.0%, not recommended for new projects | (price and dims can be verified on the OpenAI website) |
| Open-source / self-hosted | BGE-M3 (568M params, 1024 dims, 8192 token, triple outputs of dense+sparse+multi-vector); bge-large-zh-v1.5 (326M, 1024 dims, 512 token); Qwen3-Embedding (0.6B/4B/8B, Apache 2.0) | (specs should follow the official model card) |
| Retrieval sub-leaderboard (C-MTEB Chinese) | bge-m3 composite 71.4 / Retrieval ndcg@10 73.9; bge-large-zh-v1.5 68.6 / 71.5; bge-base-zh-v1.5 67.5 / 70.2; m3e-base 61.5 / 64.3 | (changes as the leaderboard updates) |
| Quantization cost | FP16 retrieval loss <0.1%, throughput 1.5~2×; INT8 loss 0.5%~1.5%, 2~3×; INT4 loss 2%~5%, 3~4× | |
| Production context length | Teams typically compress production context to 16K~50K tokens/request, occasionally 80K for document analysis; RAG pipelines usually give the model only 3K~10K token high-signal slices | |
| Top-K heuristics | Feeding 20 chunks beats 10 and 5, but there is an upper limit |
1.4.3. Anthropic Contextual Retrieval (first-party official data, R1)
Source: Anthropic official blog https://www.anthropic.com/news/contextual-retrieval
| Approach | Top-20 retrieval failure rate | Reduction vs baseline |
|---|---|---|
| Baseline (Embedding only) | 5.7% | — |
| Standard hybrid (Embedding + native BM25) | 5.0% | Small improvement (this figure comes from a Chinese retelling; the official blog's main table does not list it →) |
| Contextual Embeddings | 3.7% | 35% |
| Contextual Embeddings + Contextual BM25 | 2.9% | 49% |
| Plus Reranking | 1.9% | 67% |
The official text states: "This method can reduce the number of failed retrievals by 49% and, when combined with reranking, by 67%."
Metric definition: failure rate = 1 - recall@20, i.e., the "failure rate of not finding a relevant document in the top 20 retrieval results"; the test domains cover codebases, novels, ArXiv papers, and scientific literature.
Two-stage engineering workflow (official text):
- Offline preprocessing (one-time): chunk into blocks of a few hundred tokens → for each block, pass the full document + that block to Claude to generate a 50~100 token context description → prepend it to the block → generate a semantic embedding for the concatenated text → build a BM25 index on the concatenated text.
- Online inference: dual-recall with BM25 and vectors → Rank Fusion to merge and deduplicate → (optional) rerank the Top 150 candidates → take the Top 20 into the LLM.
Cost control (official text): using Prompt Caching, the full document is loaded only once and all blocks share the cache; the one-time preprocessing cost is about $1.02 per million document tokens.
This data point is the most valuable first-hand evidence in this direction: with the model unchanged, purely engineering means reduced the retrieval failure rate by 67%.
1.4.4. Evaluation Methodology
RAGAS (R1/R2, NVIDIA NeMo microservices documentation, Red Hat OpenShift AI documentation):
| Category | Metrics |
|---|---|
| Retrieval side | context_recall (proportion of reference information that is recalled), context_precision (relevance of the recalled blocks to the question), context_relevance, context_entity_recall |
| Generation side | faithfulness (factual consistency between the answer and the recalled context), response_relevancy (requires embeddings_model), response_groundedness |
| Composite side | answer_correctness (semantic similarity + factual similarity), answer_similarity, noise_sensitivity (robustness to noisy/irrelevant context) |
- Score range:
context_recallranges from 0 to 1, the higher the better. - Dependencies: needs a Judge LLM; some metrics additionally require Judge Embeddings.
- Supports both offline (pre-generated answers) and online (auto-generated answers) evaluation modes.
- Practical usage: automated quality gates (triggered after every commit or model update to prevent regression), evaluation-driven development (comparing different chunking strategies/embedding models/retrieval algorithms), factual-consistency thresholds (setting a threshold on faithfulness), production scale-out (running thousands of samples in a distributed manner under the remote provider mode).
BEIR (R2, original paper arXiv:2104.08663, Thakur et al., NeurIPS 2021 Datasets and Benchmarks):
- 18 datasets, 9 categories of retrieval tasks: fact-checking, citation prediction, duplicate-question retrieval, argument retrieval, news retrieval, QA, tweet retrieval, bio-medical IR, entity retrieval.
- Primary metric nDCG@10, also computing MAP, Recall@k, Precision@k, MRR (k from 1 to 1000).
- Core methodological contribution: zero-shot evaluation — dense retrievers often beat BM25 in-domain but often lose cross-domain; BEIR exposes this generalization gap.
- Practical reminder: some datasets require authorization or manual download (TREC-NEWS, Robust04, Signal-1M, BioASQ); the publicly runnable subset is about 15~17, and "BEIR averages" computed over different subsets are not directly comparable.
MTEB / MMTEB (R2, original paper arXiv:2210.07316, Muennighoff et al.):
- Original MTEB: 8 task categories, 58 datasets, 112 languages; the metrics are respectively retrieval (nDCG@10), classification (accuracy), clustering (v-measure), pair classification (AP), reranking (MAP), STS (Spearman), summarization (Spearman), bitext mining (F1).
- MMTEB (Enevoldsen et al., ICLR 2025, arXiv:2502.13595): expanded to 500+ evaluation tasks, 250+ languages; the leaderboard is split by naming (English v2 / multilingual) and now uses Borda count ranking.
- Key advice for choosing models for RAG (R1-level methodology): "Quote the retrieval sub-score, not the overall MTEB rank." — for RAG scenarios you should look at the retrieval sub-score, not the overall rank.
2. Glossary
| Term | English / Abbreviation | Definition |
|---|---|---|
| Retrieval-Augmented Generation | RAG | Retrieval-Augmented Generation, a technical system that retrieves first and generates after |
| Chunking | Chunking | The process of splitting a document into searchable passages |
| Recursive character splitting | Recursive Character Splitting | A general strategy of splitting recursively by character level (paragraph → sentence → word) |
| Semantic chunking | Semantic Chunking | A strategy that splits at semantically similar boundaries, requiring embedding sentences |
| Parent-child chunk | Parent-Child Chunk | A hierarchical structure where small child chunks are used for retrieval and large parent chunks are fed into generation |
| Overlap | Overlap / Chunk Overlap | Content repeated between adjacent chunks, used to mitigate semantic loss at boundaries |
| Dense retrieval | Dense Retrieval | Retrieval based on vector similarity |
| Sparse retrieval | Sparse Retrieval | Retrieval based on term-frequency statistics (e.g., BM25) |
| Hybrid search | Hybrid Search | Fusion of dense and sparse dual-path recall |
| Reciprocal rank fusion | RRF | Reciprocal Rank Fusion, a parameter-free multi-path ranking fusion algorithm |
| Reranking | Reranking | Fine-ranking the candidate set using models such as cross-encoders |
| Cross-encoder | Cross-encoder | A relevance-scoring model that encodes query and document jointly |
| Context assembly | Context Assembly | Organizing the reranked results into the context finally sent to the model |
| Context cliff | Context Cliff | The phenomenon where quality drops sharply once the context exceeds a certain length |
| Recall | Recall@K | The proportion of the top K results that contain the relevant document |
| Normalized discounted cumulative gain | nDCG@10 | The ranking-quality metric of the top 10 results |
| Faithfulness | Faithfulness | Factual consistency between generated content and the provided context |
| Context recall | Context Recall | The proportion of reference information successfully recalled (a RAGAS metric) |
| Noise sensitivity | Noise Sensitivity | Robustness to noisy/irrelevant context (a RAGAS metric) |
| Prompt caching | Prompt Caching | Reusing cached prefixes to reduce the cost of repeated input |
| Contextual retrieval | Contextual Retrieval | A method that generates a context description for each block before embedding and indexing |
| Needle in a haystack | NIAH | Needle in a Haystack, evaluating long-context retrieval capability |
| Lost in the middle | Lost in the Middle | The phenomenon where information utilization in the middle of long context drops significantly |
| Permission-aware retrieval | Permission-aware Retrieval | An approach that filters results by permissions at the retrieval stage |
| Golden dataset | Golden Dataset | A high-quality internally annotated evaluation set |
3. Case Studies
3.1. LinkedIn: Knowledge-Graph-Enhanced Customer Service RAG
3.1.1. Background
LinkedIn's customer service team needs to handle a large number of technical questions, and historical solutions are scattered across Jira tickets. The difficulty of plain-text RAG is that the relationships between tickets (cloned from, related to, caused by) carry a great deal of semantic information, which is lost after text chunking.
3.1.2. Approach
Based on the ZenML LLMOps database and Evidently AI's summary (R2, original source is an engineering talk):
- Build historical Jira tickets into a tree-shaped knowledge graph: nodes = Summary / Description / Priority / Steps to Reproduce / Fix Solution.
- Explicit relationships come from Jira metadata (cloned from / related to / caused by).
- Implicit relationships are built using E5 embedding cosine similarity + a threshold.
- GPT-4 handles parsing and generation; Qdrant stores the vectors.
- Deployed across multiple product lines, running in production for about 6 months.
3.1.3. Results
Based on the above source (R2):
| Metric | Value |
|---|---|
| MRR | Up 77.6% |
| BLEU | Up 0.32 |
| Median resolution time per question | Down 28.6% |
The engineering takeaway of this case is: the combination of structured relationships + vector retrieval captures the semantic structure of ticket-like corpora better than pure vector RAG.
3.2. DoorDash: An Evaluation Closed Loop Built on Guardrails and an LLM Judge
3.2.1. Background
DoorDash's customer service chatbot faces end users directly, so a single wrong answer causes real business loss. Its technical focus is therefore not only on "answering accurately" but also on "being able to detect when it answered incorrectly."
3.2.2. Approach
Based on Evidently AI's summary (R2, original source is the DoorDash engineering blog):
- System architecture = RAG system + LLM Guardrail: monitoring the accuracy and compliance of every answer online.
- The LLM Judge evaluates along five dimensions: retrieval correctness, answer accuracy, grammatical/language accuracy, context coherence, and relevance to the user's request.
3.2.3. Results
This source does not disclose specific metric-improvement figures (R2). Its value lies in providing a replicable online evaluation structure: evaluating retrieval correctness and answer accuracy separately, so that "retrieval went wrong" and "retrieval was right but generation went wrong" can be distinguished and located — this is precisely the production counterpart of the design ideas behind evaluation frameworks such as RAGAS.
3.3. Bell: A Modular Document-Embedding Pipeline with Incremental Indexing
3.3.1. Background
Bell is a Canadian telecommunications operator whose document system is large and continuously updated. A one-time full rebuild of the index is unacceptable in both cost and latency.
3.3.2. Approach
Based on Evidently AI's summary (R2, original source is a conference talk video):
- Build a modular document-embedding pipeline supporting batch processing and incremental updates.
- Automatically update the index when documents are added or removed, avoiding full rebuilds.
- Each component is built and maintained as an independent service following DevOps principles.
3.3.3. Results
This source does not disclose quantitative metrics (R2). Its engineering value lies in clarifying RAG's operations-side requirements: the index is not a one-time artifact but a service that needs continuous evolution; incremental updates and component decoupling are necessary conditions for putting RAG into production.
Other reference cases (R2):
| Case | Key Points |
|---|---|
| Harvard Business School · ChatLTV | Corpus of about 200 documents, 15 million words (cases, teaching notes, books, blogs, course Slack historical Q&A); integrated into the course Slack channel, supporting both private and public modes |
| Vimeo | Video transcription → processing → passage retrieval → video Q&A; needs to handle speaker identification and long-context retrieval problems |
| Grab | Report Summarizer calls the Data-Arks API to generate tabular data which the LLM then summarizes; saves about 3~4 hours per report |
| Table-description vector index → semantic retrieval of candidate tables → LLM selects a refined subset → generates SQL | |
| Glean (enterprise AI platform) | Vendor claims: 110 hours/user/year saved, 20% fewer internal support tickets, 93% enterprise adoption within 2 years, ROI period <6 months, 30% lower token usage compared to off-the-shelf MCP tools |
The Glean figures are vendor claims and must be labeled when cited.
4. Practical Standards
4.1. AGENTS.md Specification
The following is the full, copyable industry-standard AGENTS.md for the RAG direction, reflecting the specialized toolchain of vector stores, chunkers, embedding models, rerankers, and evaluation frameworks.
# AGENTS.md —— RAG(检索增强生成)
## 角色与边界
- 你是检索增强智能体,负责切分、索引、召回、重排、装配与评测。
- 你可以:读写索引、执行检索与重排、调整检索参数并提交评测、生成评测报告、调用脚本做跑分。
- 你不可以:在未见评测结果的情况下把参数变更推到生产、绕过权限过滤、
把评测集混入索引语料、删除历史索引快照、编造评测数值。
- 判定原则:任何影响检索结果的变更,必须先评测后上线。
## 环境假设
- 运行环境提供:向量库、BM25 索引、嵌入服务、重排服务、切分器、
评测框架(RAGAS / 自有黄金集)、对象存储(索引快照)、Trace 与审计日志。
- 语料具备版本/快照标识;索引具备版本号,可与语料版本对应。
- 检索服务在召回阶段即完成权限过滤(permission-aware retrieval)。
- 具备上下文预算读数(已装配 Token 数 / 上限)。
## 上下文加载顺序(Context Budget)
1. 查询与任务契约(常驻,不压缩)
2. 检索配置:切分参数、嵌入模型、融合方式、重排模型、Top-K(常驻)
3. 候选集(重排后的 Top-K,附 doc_id + 版本 + 片段定位)
4. 会话历史(多轮时,压缩为要点)
5. 参考示例(按需)
- 装配上限:默认低于 8K tokens;关键信息置于首部或尾部,不得埋在中部。
- 关键片段优先,低相关片段先被裁剪。
## 工具契约
- 切分器:递归 / 语义 / 层次化;参数必须由自有黄金集实测确定,不得照搬默认值。
- 向量库:写入前校验幂等;索引变更生成新版本快照;旧快照保留可回溯。
- 检索:默认向量 + BM25 混合召回 + RRF 融合;纯向量检索不得作为生产默认。
- 重排:候选集进入上下文前做 cross-encoder 重排;重排阈值写入配置。
- 评测:RAGAS(context_recall / context_precision / faithfulness / answer_correctness /
noise_sensitivity)+ 自有黄金集;需要 Judge LLM 与 Judge Embeddings。
- 脚本:切分、跑分、批量评测、格式转换必须调用 scripts/,不得用生成方式替代。
## 任务执行流程(SOP)
1. 定界:语料白名单与版本、密级、查询类型(事实型 / 分析型 / 多跳)。
2. 判定模式:语料 < 200,000 tokens 走全量装配,不做切分召回。
3. 检索:向量 + BM25 → 权限过滤 → RRF 融合 → 重排 → Top-K。
4. 装配:按预算组织上下文;关键信息首尾放置;标注来源标识。
5. 生成:逐条陈述挂来源标识;无召回支撑不输出事实性结论。
6. 评测:任何参数变更先在黄金集上跑 Recall@K 与忠实度,与基线对比。
7. 上线:指标不低于基线方可上线;记录基线变化与变更日志。
8. 回流:把失败查询与人工修正加入评测集与负样本。
## 验证与证据要求
- 证据包:来源清单(doc_id + 版本 + 片段定位)、检索记录(查询、召回数、
重排 Top-K、权限过滤命中数)、评测报告(指标 + 基线对比)、变更日志、索引版本快照。
- 引用外部基准时须注明所跑子集与指标口径(不同 BEIR 子集的均值不可直接比较)。
- 嵌入模型选型须引用 retrieval 子分,而非 MTEB/MMTEB 总排名。
## 失败与升级策略
| 失败 | 处置 |
|---|---|
| 检索零命中 | 输出"未检索到可靠依据";检查索引版本与查询改写;禁止自由生成 |
| 召回正确但答案错误 | 检查装配位置与上下文预算;提高忠实度阈值 |
| 参数变更导致指标回退 | 回退上一配置;重新评测;记录归因 |
| 权限过滤命中异常 | 停止检索,记录审计事件,升级至权限管理员 |
| 索引与语料版本不匹配 | 暂停服务,重新索引并校验 |
| 评测不可比 | 固定子集、模型版本与随机种子;注明口径 |
- 每个循环必须有步数上限与 Token 预算上限;超限即停并升级。
## 安全与合规红线
- 不得绕过权限过滤;不得使用高权限账号做检索。
- 不得把评测集混入索引语料造成数据污染。
- 不得删除历史索引快照;不得编造评测数值。
- 未经评测通过的检索配置不得推到生产。
## 禁止事项
- 禁止照搬默认切分参数(400~512 tokens + 10%~20% 重叠仅为线索,须实测)。
- 禁止把 `[待核实]` 的聚合站数值当作设计依据。
- 禁止用纯向量检索作为生产默认。
- 禁止在无召回支撑时输出事实性结论。
- 禁止编造 doc_id、版本号、评测指标与 URL。
- 禁止使用 XX / XXX / ___ 等非标准占位符(统一用 [待填写] / [待核实])。
- 禁止使用 emoji 与署名。
## 输出格式
- 检索结果:片段内容 + 来源标识 + 相似度/重排分 + 用于哪一论点。
- 评测报告:指标 / 基线 / 当前值 / 差值 / 是否通过门控 / 口径说明。
- 数值带单位;范围用 ~ 连接;中文全角标点;中英文之间加空格。
## 评估与自检
- 九项自检:来源可回溯 / 无无源断言 / 数值一致 / 无非标占位符 / 数量与表格一致 /
编号可核实 / 密级正确 / 评测已跑且可比 / 信息缺口已声明。
- 每次索引或模型变更后重跑评测集;指标回退视为缺陷。
- 失败查询与人工修正必须回流进评测集。 4.2. SKILL.md Specification
The following is the full, copyable industry-standard SKILL.md for the RAG direction.
---
name: rag-eval
description: 构建 RAG 评测集并在切分/嵌入/融合/重排/装配变更后跑分,产出与基线的对比报告与上线门控结论。当用户要求"评测这套检索""对比两种切分策略""换嵌入模型后效果如何""生成 RAG 回归报告""为什么这条查询召回不到"时触发。
version: 1.0
created: 2026-09-12
---
# RAG 评测与调优(RAG Evaluation & Tuning)
## 适用场景
- 构建与维护黄金集(Golden Dataset)与负样本集。
- 切分参数、嵌入模型、融合方式、重排模型、Top-K 的对比评测。
- 索引或语料大版本更新后的回归验证。
- 单条查询召回失败的归因分析。
## 前置条件
- 已有基线指标(Recall@K、context_recall、faithfulness 等)与对应的配置快照。
- 评测集版本固定,且未混入索引语料。
- 运行环境提供 Judge LLM 与 Judge Embeddings。
- 已确定指标口径与门控阈值。
## 输入
- 待评测配置:切分参数 / 嵌入模型 / 融合方式 / 重排模型 / Top-K / 装配预算
- 评测集版本与门控阈值
- 可选:失败查询清单(用于归因分析)
## 输出
- 评测报告:指标 / 基线 / 当前值 / 差值 / 是否通过门控 / 口径说明
- 归因分析:失败查询归类(切分问题 / 召回问题 / 重排问题 / 装配问题)
- 上线建议:通过 / 回退 / 需人工裁决
- 评测集更新建议:新增负样本与边界用例
## 执行步骤
1. 固定变量:一次只变更一个因素,其余保持不变。
2. 跑检索侧指标:Recall@K、context_recall、context_precision、context_entity_recall。
3. 跑生成侧指标:faithfulness、response_relevancy、response_groundedness。
4. 跑综合与鲁棒性:answer_correctness、noise_sensitivity。
5. 与外部基准对齐(可选):BEIR 子集(注明子集与 nDCG@10 口径)、
MTEB/MMTEB retrieval 子分(**引用子分,不引用总排名**)。
6. 对比基线,输出是否通过门控;未通过则回退并归因。
7. 把失败查询归类:切分 / 召回 / 重排 / 装配,并加入评测集。
8. 产出报告与变更日志;记录索引版本快照。
## 质量标准(DoD)
- 一次只变更一个变量,配置快照完整可复现。
- 指标口径明确;引用外部基准时注明子集。
- 与基线的差值、统计窗口、样本量均已披露。
- 失败查询已归类并进入评测集。
- 通过门控才可给出上线建议。
## 常见失败与处理
| 失败 | 根因 | 处置 |
|---|---|---|
| 指标不可比 | 子集/口径/模型版本变化 | 固定变量重跑;注明口径 |
| 检索好但生成差 | 装配位置或预算问题 | 关键信息首尾放置;压缩低相关片段 |
| 生成好但检索差 | 评测集过易或样本偏差 | 补充难样本与负样本 |
| 评测结果不稳定 | Judge 模型/随机性 | 固定 Judge 模型与种子;多次取均值 |
| 线上与离线不一致 | 索引版本不匹配 | 校验线上索引版本与评测时一致 |
| 换嵌入模型无提升 | 引用了总排名而非 retrieval 子分 | 改看 retrieval 子分与域内召回 |
## 示例
用户请求:把切分从 512 tokens/15% 重叠改为 1024 tokens/20% 重叠,评估是否可上线。
执行:
1. 固定嵌入模型、融合方式、重排模型、Top-K=20、装配预算 8K tokens。
2. 在黄金集(v1.3,300 条)上跑 Recall@20、context_recall、context_precision。
3. 跑生成侧 faithfulness 与 answer_correctness(Judge 模型固定)。
4. 对比基线:Recall@20 由 0.81 变为 0.79(-0.02),faithfulness 由 0.92 变为 0.91。
5. 结论:未通过门控(Recall 回退),建议回退 512/15% 配置。
6. 归因:长块稀释了事实型查询的信号;把 12 条失败查询加入评测集。
7. 输出报告:指标表 + 口径说明 + 回退建议 + 变更日志。
约束:禁止在未跑评测的情况下给出"建议上线"结论;禁止引用聚合站数值作为依据。 4.3. Implementation Checklist
| # | Check Item | Pass Criteria | Frequency |
|---|---|---|---|
| 1 | Small-corpus exemption decided | When the corpus is < 200,000 tokens, use full assembly and do not build an index | Project start |
| 2 | Chunking parameters measured | Parameters set by Recall@K on your own golden dataset, not copied from defaults | Every parameter decision |
| 3 | Index version traceable | Index has a version number corresponding one-to-one with the corpus version | Every index |
| 4 | Hybrid recall is the default | Vector + BM25 + RRF fusion; pure vector is not the production default | Every release |
| 5 | Reranking enabled | Candidate set is cross-encoder reranked before entering context | Every release |
| 6 | Permission filtering at retrieval stage | Results already passed permission filtering, not post-hoc masking | Every retrieval |
| 7 | Assembly budget controlled | Assembled context below 8K tokens; key information placed at the head or tail | Every call |
| 8 | Source identifiers complete | Each passage carries doc_id + version + passage location | Every delivery |
| 9 | Evaluation run and comparable | Fixed variables, fixed subsets, fixed measure | Every change |
| 10 | Gate set | Thresholds exist for faithfulness and Recall@K; if not met, do not release | Every change |
| 11 | Cite sub-score, not overall rank | Model selection looks at the MTEB/MMTEB retrieval sub-score | Every selection |
| 12 | Evaluation set unpolluted | Evaluation set is not mixed into the indexed corpus | Every index |
| 13 | Incremental indexing available | Supports incremental updates for added/removed documents without full rebuilds | Ongoing |
| 14 | Failures recycled | Failed queries are categorized and enter the evaluation set and negative samples | Weekly |
| 15 | Deterministic operations scripted | Chunking, benchmarking, and batch evaluation go through scripts/ | Every execution |
5. Summary
RAG is the direction in the knowledge-collaboration group with the highest technical density, the most complete evaluation system, and the most easily underestimated engineering complexity.
The five most important conclusions of this direction:
- The returns from engineering means can be quantified and are enormous. Anthropic's official data shows Contextual Retrieval lowers the Top-20 retrieval failure rate from 5.7% to 2.9% (-49%), and to 1.9% (-67%) when combined with reranking, while the one-time preprocessing cost is only about $1.02 per million document tokens (R1). This was achieved with the model unchanged.
- Long context is not a substitute for RAG. The U-shaped effect of Lost in the Middle, RULER's evaluation of 17 long-context models (only about half maintain satisfactory performance at 32K), and NoLiMa's evaluation of 13 models supporting ≥128K (11 drop below half of their own short-context accuracy at 32K) together show that piling on context can produce a net negative return.
- Small corpora should go directly and entirely into the prompt. Anthropic officially states: < 200,000 tokens (about 500 pages) requires no RAG. This rule can significantly reduce complexity for small teams.
- Evaluation is a release gate, not an after-the-fact report. RAGAS measures the retrieval side and the generation side separately, so that "retrieval went wrong" and "generation went wrong" can be distinguished and located; BEIR emphasizes zero-shot cross-domain evaluation, and MTEB/MMTEB emphasizes citing the retrieval sub-score rather than the overall rank.
- Chunking is the first source of failure, but there is no universal optimum. In public data, "semantic chunking has better recall" and "semantic chunking has lower end-to-end accuracy" point in opposite directions, showing that you must measure on your own golden dataset.
We must candidly point out that, in the parameter space of chunking and retrieval for this direction, the vast majority of public figures come from SEO/aggregation content sites and have not been traced back to Chroma Research, Weaviate, NVIDIA FinanceBench, OpenAI's official pricing pages, or the MTEB official leaderboard. These figures have been fully marked [To be verified] in 1.4.1 and 1.4.2 and should not be used as design basis; they can only serve as clues for tuning direction. The only truly trustworthy quantitative anchors are the Anthropic Contextual Retrieval set (R1).
Information-Gap Statement
| # | Gap | Status |
|---|---|---|
| 1 | Formal ISO/IEEE standard for RAG | No authoritative standard/specification yet |
| 2 | Chroma Research's official chunking evaluation report | Original page not obtained; related figures |
| 3 | Weaviate 2025 official chunking benchmark | Original page not obtained; related figures |
| 4 | OpenAI official embedding model pricing and dimensions | Official pricing page not traced back; related figures |
| 5 | Current values on the MTEB / C-MTEB official leaderboard | Official leaderboard not traced back; related figures |
| 6 | Original source of the NVIDIA FinanceBench chunking experiment | Not obtained |
| 7 | Original source of the "February 2026 benchmark across 50 papers: recursive 69% vs semantic 54%" | No result |
| 8 | Original statistical source for production context length 16K~50K tokens | |
| 9 | Original research on the 2,500-token "context cliff" | |
| 10 | Original research on Top-K heuristics (20 > 10 > 5) |
6. References
- Introducing Contextual Retrieval — Anthropic, 2024. https://www.anthropic.com/news/contextual-retrieval
- RAGAS Metrics — NVIDIA NeMo Microservices documentation. https://docs.nvidia.com/nemo/microservices/latest/evaluator/metrics/rag.html
- Evaluating AI Systems (RAGAS practical usage) — Red Hat OpenShift AI documentation. https://docs.redhat.com/en/documentation/red_hat_openshift_ai_self-managed/3.3/html-single/evaluating_ai_systems/
- BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models — Thakur et al., NeurIPS 2021 (arXiv:2104.08663). https://arxiv.org/abs/2104.08663
- MTEB: Massive Text Embedding Benchmark — Muennighoff et al. (arXiv:2210.07316). https://arxiv.org/abs/2210.07316
- MMTEB: Massive Multilingual Text Embedding Benchmark — Enevoldsen et al., ICLR 2025 (arXiv:2502.13595). https://arxiv.org/abs/2502.13595
- Lost in the Middle: How Language Models Use Long Contexts — Liu et al., TACL 2024. https://arxiv.org/abs/2307.03172
- RULER: What's the Real Context Size of Your Long-Context Models? — Hsieh et al., COLM 2024. https://arxiv.org/abs/2404.06654
- NoLiMa: Long-Context Evaluation Beyond Literal Matching — Modarressi et al., ICML 2025. https://arxiv.org/abs/2502.05167
- Needle in a Haystack — Pressure Testing LLMs — gkamradt/LLMTest_NeedleInAHaystack. https://github.com/gkamradt/LLMTest_NeedleInAHaystack
- Knowledge Graph-Enhanced RAG for Customer Service QA (LinkedIn case) — ZenML LLMOps Database. https://zenml.io/llmops-database/knowledge-graph-enhanced-rag-for-customer-service-question-answering
- RAG Examples (DoorDash / Bell / HBS / Vimeo / Grab / Pinterest roundup) — Evidently AI. https://evidently.ai/blog/rag-examples
- Glean — Enterprise AI platform official site (vendor-claimed data). https://www.glean.com
- Qdrant — Vector database (used in the LinkedIn case). https://qdrant.tech/