Haystack 2.x Retrievers API 深度解析從 BM25、向量檢索到句子窗口與自動合并【免費(fèi)下載鏈接】haystackOpen-source AI orchestration framework for building context-engineered, production-ready LLM applications. Design modular pipelines and agent workflows with explicit control over retrieval, routing, memory, and generation. Built for scalable agents, RAG, multimodal applications, semantic search, and conversational systems.項(xiàng)目地址: https://gitcode.com/GitHub_Trending/ha/haystack本篇技術(shù)指南以 Haystack 2.x 的 Retrievers API 文檔docs-website/reference_versioned_docs/version-2.18/haystack-api/retrievers_api.md為核心骨架系統(tǒng)講解AutoMergingRetriever、InMemoryBM25Retriever、InMemoryEmbeddingRetriever、FilterRetriever與SentenceWindowRetriever五大組件的設(shè)計(jì)原理、完整參數(shù)語義、代碼級調(diào)用鏈與實(shí)戰(zhàn)用法。讀完本文你將能夠根據(jù)檢索場景關(guān)鍵詞、語義、過濾、父子文檔合并、上下文窗口擴(kuò)展正確選型并組裝可運(yùn)行的 Haystack 檢索流水線。檢索器在 Haystack 中的定位在 Haystack 中Retriever檢索器是一個掃過 Document Store、返回與查詢相關(guān)的候選文檔集合的組件。它與Reader等組件的核心區(qū)別在于檢索器只負(fù)責(zé)召回候選不做精讀與答案生成它通常位于 RAG 流水線的前端將召回結(jié)果交給后續(xù)的 Prompt Builder、Generator 或 Evaluator 消費(fèi)。從當(dāng)前倉庫源碼看所有內(nèi)置檢索器都位于 haystack/components/retrievers/并通過component裝飾器注冊為可序列化、可連線的管道組件auto_merging_retriever.py —AutoMergingRetrieverin_memory/bm25_retriever.py —InMemoryBM25Retrieverin_memory/embedding_retriever.py —InMemoryEmbeddingRetrieverfilter_retriever.py —FilterRetrieversentence_window_retriever.py —SentenceWindowRetriever此外還有MultiQueryEmbeddingRetriever、MultiQueryTextRetriever、MultiRetriever、TextEmbeddingRetriever等組件見 haystack/components/retrievers/init.py本文聚焦 API 文檔中展開講解的五個組件。InMemoryBM25Retriever基于關(guān)鍵詞的經(jīng)典檢索InMemoryBM25Retriever使用 BM25 關(guān)鍵詞算法檢索與查詢最相似的文檔必須搭配InMemoryDocumentStore使用。最小可用示例API 文檔給出的最小示例完整可運(yùn)行from haystack import Document from haystack.components.retrievers.in_memory import InMemoryBM25Retriever from haystack.document_stores.in_memory import InMemoryDocumentStore docs [ Document(contentPython is a popular programming language), Document(contentpython ist eine beliebte Programmiersprache), ] doc_store InMemoryDocumentStore() doc_store.write_documents(docs) retriever InMemoryBM25Retriever(doc_store) result retriever.run(queryProgrammiersprache) print(result[documents])構(gòu)造參數(shù)與底層校驗(yàn)構(gòu)造函數(shù)簽名見 in_memory/bm25_retriever.pydef __init__(document_store: InMemoryDocumentStore, filters: Optional[dict[str, Any]] None, top_k: int 10, scale_score: bool False, filter_policy: FilterPolicy FilterPolicy.REPLACE)參數(shù)默認(rèn)值語義document_store必填搜索目標(biāo)必須是InMemoryDocumentStore實(shí)例否則拋TypeError源碼第 70-71 行filtersNone用于縮小搜索空間的過濾字典元數(shù)據(jù)過濾top_k10返回的最大文檔數(shù)必須 0否則拋ValueError源碼第 75-76 行scale_scoreFalse為True時將得分歸一化到 0-11 表示極相關(guān)為False返回原始相似度得分filter_policyREPLACE過濾策略見下文FilterPolicy 詳解run 與 run_async運(yùn)行期覆蓋初始化參數(shù)run方法簽名component.output_types(documentslist[Document]) def run(query: str, filters: Optional[dict[str, Any]] None, top_k: Optional[int] None, scale_score: Optional[bool] None)run_async擁有完全一致的簽名用于異步執(zhí)行。底層調(diào)用鏈非常直接源碼第 146-153 行先通過apply_filter_policy(self.filter_policy, self.filters, filters)依據(jù)策略合并/替換初始化過濾器與運(yùn)行期過濾器top_k、scale_score若未在run中傳入則回退到構(gòu)造函數(shù)中的值最終委托給document_store.bm25_retrieval(query..., filters..., top_k..., scale_score...)。bm25_retrieval是InMemoryDocumentStore的原生方法haystack/document_stores/in_memory/document_store.py異步版本為bm25_retrieval_async同文件第 1054 行附近由run_async調(diào)用。也就是說同步/異步只發(fā)生在 Document Store 層檢索器本身是薄封裝。InMemoryEmbeddingRetriever語義向量檢索InMemoryEmbeddingRetriever檢索與查詢在語義上最相似的文檔同樣只搭配InMemoryDocumentStore。使用前必須保證文檔與查詢兩側(cè)都有 embedding 可用索引流水線中用DocumentEmbedder為文檔生成向量查詢流水線中用TextEmbedder為查詢生成向量再傳給檢索器。完整用法示例Sentence Transformers 版API 文檔給出的完整示例使用SentenceTransformersDocumentEmbedder/SentenceTransformersTextEmbedderfrom haystack import Document from haystack.components.embedders import SentenceTransformersDocumentEmbedder, SentenceTransformersTextEmbedder from haystack.components.retrievers.in_memory import InMemoryEmbeddingRetriever from haystack.document_stores.in_memory import InMemoryDocumentStore docs [ Document(contentPython is a popular programming language), Document(contentpython ist eine beliebte Programmiersprache), ] doc_embedder SentenceTransformersDocumentEmbedder() doc_embedder.warm_up() docs_with_embeddings doc_embedder.run(docs)[documents] doc_store InMemoryDocumentStore() doc_store.write_documents(docs_with_embeddings) retriever InMemoryEmbeddingRetriever(doc_store) queryProgrammiersprache text_embedder SentenceTransformersTextEmbedder() text_embedder.warm_up() query_embedding text_embedder.run(query)[embedding] result retriever.run(query_embeddingquery_embedding) print(result[documents])注意與 BM25 的關(guān)鍵差異run的入?yún)⑹莙uery_embedding向量列表而非query字符串向量化完全由外部 Embedder 負(fù)責(zé)。構(gòu)造參數(shù)與 run 參數(shù)構(gòu)造函數(shù)in_memory/embedding_retriever.pydef __init__(document_store: InMemoryDocumentStore, filters: Optional[dict[str, Any]] None, top_k: int 10, scale_score: bool False, return_embedding: bool False, filter_policy: FilterPolicy FilterPolicy.REPLACE)參數(shù)默認(rèn)值語義return_embeddingFalse為True時檢索結(jié)果文檔中附帶各自的 embedding為False時僅返回文檔本體減少內(nèi)存占用與傳輸開銷run簽名component.output_types(documentslist[Document]) def run(query_embedding: list[float], filters: Optional[dict[str, Any]] None, top_k: Optional[int] None, scale_score: Optional[bool] None, return_embedding: Optional[bool] None)run_async簽名與之完全相同。底層委托鏈源碼第 166-182 行為apply_filter_policy(...)→ 參數(shù)回退 →document_store.embedding_retrieval(query_embedding..., filters..., top_k..., scale_score..., return_embedding...)異步版本調(diào)用embedding_retrieval_asyncdocument_store.py。構(gòu)造函數(shù)與 BM25 版一致地對document_store做類型檢查、對top_k 0拋ValueError。FilterPolicy 詳解REPLACE 與 MERGEFilterPolicy是 Haystack 檢索器統(tǒng)一的過濾策略枚舉定義于 haystack/document_stores/types/filter_policy.pyREPLACE默認(rèn)運(yùn)行期傳入的filters直接覆蓋初始化時的過濾器。適用于需要針對不同查詢動態(tài)切換過濾條件的場景。MERGE運(yùn)行期過濾器與初始化過濾器合并重疊字段以運(yùn)行期值為準(zhǔn)從而進(jìn)一步收窄搜索范圍。合并邏輯由apply_filter_policy(filter_policy, init_filters, runtime_filters, default_logical_operator)實(shí)現(xiàn)同文件第 287 行起在MERGE模式下若兩側(cè)都是比較型過濾器則用combine_two_comparison_filters組合為邏輯表達(dá)式否則按默認(rèn)邏輯操作符合并。由于filter_policy在to_dict中被序列化為字符串值、在from_dict中通過FilterPolicy.from_str還原見兩個 InMemory 檢索器的from_dict因此該配置可以完整地在 YAML/JSON 管道描述中往返。FilterRetriever純元數(shù)據(jù)過濾檢索FilterRetriever不計(jì)算任何相似度只按過濾器條件從 Document Store 中取出匹配的文檔。它接受通用的DocumentStore不僅限 InMemory是搭建先過濾、再檢索流水線的輕量工具。用法示例與運(yùn)行期覆蓋API 文檔示例from haystack import Document from haystack.components.retrievers import FilterRetriever from haystack.document_stores.in_memory import InMemoryDocumentStore docs [ Document(contentPython is a popular programming language, meta{lang: en}), Document(contentpython ist eine beliebte Programmiersprache, meta{lang: de}), ] doc_store InMemoryDocumentStore() doc_store.write_documents(docs) retriever FilterRetriever(doc_store, filters{field: lang, operator: , value: en}) # 若在 run 中傳入 filters將覆蓋初始化時的過濾器 result retriever.run(filters{field: lang, operator: , value: de}) print(result[documents])__init__(document_store, filtersNone)與run(filtersNone)的參數(shù)語義完全一致從源碼filter_retriever.py看run中的解析規(guī)則是resolved_filters filters if filters is not None else self.filters——運(yùn)行期過濾器一旦給出就整體覆蓋初始化值與FilterPolicy.REPLACE行為一致但該組件自身不攜帶 FilterPolicy 枚舉。異步版run_async委托filter_documents_async同步版委托filter_documents。過濾器語法Comparison 與 Logic過濾器是嵌套字典支持兩種類型規(guī)范見 haystack/document_stores/types/protocol.py比較型Comparison必須含field、operator、value三個鍵operator取值、!、、、、、in、not in。邏輯型Logic必須含operator與conditionsoperator取值NOT、OR、ANDconditions為比較型或邏輯型字典的列表。簡單示例{field: meta.type, operator: , value: article}。復(fù)合示例filters { operator: AND, conditions: [ {field: meta.type, operator: , value: article}, {field: meta.date, operator: , value: 1420066800}, {field: meta.date, operator: , value: 1609455600}, {field: meta.rating, operator: , value: 3}, { operator: OR, conditions: [ {field: meta.genre, operator: in, value: [economy, politics]}, {field: meta.publisher, operator: , value: nytimes}, ], }, ], }SentenceWindowRetriever句子窗口上下文擴(kuò)展SentenceWindowRetriever從 Document Store 中取回檢索命中文檔的鄰近分塊為查詢結(jié)果補(bǔ)充上下文。它設(shè)計(jì)為接在某個基礎(chǔ) Retriever如 BM25、Embedding Retriever之后使用輸出context_windows合并后的上下文文本列表與context_documents含鄰近塊的文檔列表兩類結(jié)果。前置條件分塊元數(shù)據(jù)約定使用該組件的前提是文檔攜帶兩塊元數(shù)據(jù)source_id標(biāo)識分塊所屬的原始文檔用于將同一來源的句子分塊歸組split_id分塊在原始文檔中的位置/順序。這兩個字段名可分別通過source_id_meta_field與split_id_meta_field自定義。與 DocumentSplitter 組合的端到端示例API 文檔給出完整流水線示例DocumentSplitter生成的source_id/split_id/split_idx_start元數(shù)據(jù)正好滿足約定from haystack import Document, Pipeline from haystack.components.retrievers.in_memory import InMemoryBM25Retriever from haystack.components.retrievers import SentenceWindowRetriever from haystack.components.preprocessors import DocumentSplitter from haystack.document_stores.in_memory import InMemoryDocumentStore splitter DocumentSplitter(split_length10, split_overlap5, split_byword) text ( This is a text with some words. There is a second sentence. And there is also a third sentence. It also contains a fourth sentence. And a fifth sentence. And a sixth sentence. And a seventh sentence ) doc Document(contenttext) docs splitter.run([doc]) doc_store InMemoryDocumentStore() doc_store.write_documents(docs[documents]) rag Pipeline() rag.add_component(bm25_retriever, InMemoryBM25Retriever(doc_store, top_k1)) rag.add_component(sentence_window_retriever, SentenceWindowRetriever(document_storedoc_store, window_size2)) rag.connect(bm25_retriever, sentence_window_retriever) rag.run({bm25_retriever: {query:third}})運(yùn)行后sentence_window_retriever的輸出形如{sentence_window_retriever: {context_windows: [some words. There is a second sentence. And there is also a third sentence. It also contains a fourth sentence. And a fifth sentence. And a sixth sentence. And a], context_documents: [Document(id..., content: some words. There is a second sentence. And there is , meta: {source_id: ..., page_number: 1, split_id: 1, split_idx_start: 20, ...}), Document(id..., content: second sentence. And there is also a third sentence. It , meta: {source_id: ..., split_id: 2, split_idx_start: 43, ...}), ...]}}命中third的分塊split_id3位于窗口中心window_size2使其前后各取 2 個分塊最終context_documents含 5 個文檔并按split_idx_start排序。構(gòu)造參數(shù)與 run 參數(shù)構(gòu)造函數(shù)sentence_window_retriever.pydef __init__(document_store: DocumentStore, window_size: int 3, *, source_id_meta_field: Union[str, list[str]] source_id, split_id_meta_field: str split_id, raise_on_missing_meta_fields: bool True)參數(shù)默認(rèn)值語義window_size3命中文檔前后各取多少個鄰近文檔如2表示取前 2 個、后 2 個必須 0_validate_window_size會對 0拋ValueError源碼第 247-250 行source_id_meta_fieldsource_id源 ID 元數(shù)據(jù)字段可傳字符串或列表傳列表時多個字段須全部匹配才視為同一來源split_id_meta_fieldsplit_id分塊位置元數(shù)據(jù)字段raise_on_missing_meta_fieldsTrue為True時缺少必要元數(shù)據(jù)即拋ValueError為False時跳過缺失文檔的上下文檢索但仍把原文檔放進(jìn)結(jié)果并在日志中告警run簽名component.output_types(context_windowslist[str], context_documentslist[Document]) def run(retrieved_documents: list[Document], window_size: Optional[int] None)運(yùn)行期傳入的window_size會覆蓋構(gòu)造時的值源碼第 200 行。輸出字典兩個鍵context_windows與retrieved_documents一一對應(yīng)的合并上下文文本每個字符串即一個上下文窗口context_documents原檢索文檔 其上下文文檔按split_idx_start排序。底層實(shí)現(xiàn)過濾條件構(gòu)造與去重疊合并從源碼看_retrieve_context_for_document第 265-285 行每個命中文檔的上下文獲取分三步讀取source_id可多字段與split_id任一缺失則跳過上下文檢索并告警返回(doc.content or , [doc])。用_build_filter_conditions(split_id, window_size, source_ids)構(gòu)造 AND 過濾條件第 310-322 行min_before split_id - window_size max_after split_id window_size conditions [ {field: fmeta.{split_id_meta_field}, operator: , value: min_before}, {field: fmeta.{split_id_meta_field}, operator: , value: max_after}, *source_id_filters, # 每個源字段一個 {field: fmeta.{...}, operator: , value: source_id} ] return {operator: AND, conditions: conditions}調(diào)用document_store.filter_documents(filter_conditions)取回候選再由靜態(tài)方法merge_documents_text合并文本。merge_documents_text靜態(tài)方法第 119-151 行的處理細(xì)節(jié)值得注意若文檔元數(shù)據(jù)中都沒有split_idx_start則直接拼接所有文本否則按split_idx_start排序后利用max(start, last_idx_end)跳過重疊區(qū)間實(shí)現(xiàn)去重疊合并——這正是DocumentSplitter(split_overlap0)場景下窗口文本不重復(fù)的關(guān)鍵。run_async走完全等價(jià)的異步路徑_retrieve_context_for_document_async。兼容的 Document Store按 API 文檔聲明SentenceWindowRetriever與以下 Document Store 兼容Astra、Elasticsearch、OpenSearch、Pgvector、Pinecone、Qdrant這些 Store 均需實(shí)現(xiàn)filter_documents/filter_documents_async。AutoMergingRetriever父子層級文檔自動合并AutoMergingRetriever基于閾值設(shè)定將命中的葉子節(jié)點(diǎn)文檔替換為其父文檔返回。它假設(shè)文檔庫中存在層級樹狀結(jié)構(gòu)葉子節(jié)點(diǎn)被索引進(jìn) Document Store父節(jié)點(diǎn)保留層級關(guān)系。設(shè)計(jì)動機(jī)其設(shè)計(jì)動機(jī)非常直觀一個段落被切成多個葉子分塊后若某個查詢命中了同一父節(jié)點(diǎn)下的多個分塊那么整段父文檔可能比單獨(dú)的幾個分塊更有信息量。此時返回父文檔能顯著提升下游 LLM 的上下文質(zhì)量。構(gòu)造參數(shù)與閾值校驗(yàn)def __init__(document_store: DocumentStore, threshold: float 0.5)document_store從中檢索父文檔的 Document Storethreshold決定返回父文檔還是保留子文檔的閾值。必須滿足0 threshold 1否則拋ValueError源碼 auto_merging_retriever.py。合并判定公式對同一父節(jié)點(diǎn)的命中子文檔數(shù)c與該父節(jié)點(diǎn)的全部子文檔數(shù)n當(dāng)c / n threshold時合并為父文檔。例如父節(jié)點(diǎn)有 3 個子文檔、命中其中 2 個時2/3 ≈ 0.67 0.5于是返回父文檔。完整示例與 HierarchicalDocumentSplitter 搭配from haystack import Document from haystack.components.preprocessors import HierarchicalDocumentSplitter from haystack.components.retrievers.auto_merging_retriever import AutoMergingRetriever from haystack.document_stores.in_memory import InMemoryDocumentStore # 創(chuàng)建 3 層層級文檔結(jié)構(gòu)父文檔有 3 個子文檔 text The sun rose early in the morning. It cast a warm glow over the trees. Birds began to sing. original_document Document(contenttext) builder HierarchicalDocumentSplitter(block_sizes[10, 3], split_overlap0, split_byword) docs builder.run([original_document])[documents] # 存儲 level-1 父文檔并初始化檢索器 doc_store_parents InMemoryDocumentStore() for doc in docs[documents]: if doc.meta[children_ids] and doc.meta[level] 1: doc_store_parents.write_documents([doc]) retriever AutoMergingRetriever(doc_store_parents, threshold0.5) # 假設(shè)檢索到同一父節(jié)點(diǎn)下的 2 個葉子文檔 # 父節(jié)點(diǎn)有 3 個子文檔命中 2 個2/3 0.66(6)超過閾值 0.5應(yīng)返回父文檔 leaf_docs [doc for doc in docs[documents] if not doc.meta[children_ids]] docs retriever.run(leaf_docs[4:6])預(yù)期輸出返回父文檔而非兩個葉子分塊{documents: [Document(id538..), content: warm glow over the trees. Birds began to sing., meta: {block_size: 10, parent_id: 835.., children_ids: [c17..., 3ff..., 352...], level: 1, source_id: 835..., page_number: 1, split_id: 1, split_idx_start: 45})]}層級元數(shù)據(jù)約定與輸入校驗(yàn)該組件依賴HierarchicalDocumentSplitter寫入的元數(shù)據(jù)字段見 haystack/components/preprocessors/hierarchical_document_splitter.py__block_size當(dāng)前節(jié)點(diǎn)塊的尺寸根節(jié)點(diǎn)為 0__parent_id父文檔 ID根節(jié)點(diǎn)為None__children_ids子文檔 ID 列表葉子為空列表__level層級號根節(jié)點(diǎn)為 0逐層 1。run之前的_check_valid_documents源碼第 101-113 行會校驗(yàn)每個輸入文檔必須含__parent_id、__level、__block_size三個字段缺失即拋ValueError。注意__level與__block_size使用存在性檢查in doc.meta而非真值檢查因?yàn)楦?jié)點(diǎn)的__level0、__block_size0本身是假值。遞歸合并算法run以及等價(jià)的run_async的核心是遞歸函數(shù)_try_merge_level源碼第 141-167 行按__parent_id用defaultdict(list)對輸入文檔分組沒有父節(jié)點(diǎn)的文檔直接進(jìn)入返回列表。對每個父組通過filter_documents({field: id, operator: , value: parent_id})從 Document Store 取回父文檔要求恰好命中 1 個且父文檔必須有子節(jié)點(diǎn)否則拋ValueError。計(jì)算score len(child_docs) / len(parent_doc.meta[__children_ids])若score threshold則合并為父文檔否則子文檔原樣保留。若本輪產(chǎn)生了新的合并結(jié)果則遞歸地嘗試向更高一層合并因?yàn)楦肝臋n本身也有__parent_id直到?jīng)]有任何合并發(fā)生才返回。因此輸出可能是不同層級文檔的混合列表。兼容的 Document Store按 API 文檔聲明AutoMergingRetriever目前僅支持AstraDB、ElasticSearch、OpenSearch、PGVector、Qdrant這些 Store 需支持按 ID 過濾父文檔。序列化to_dict / from_dict 契約上述五個組件全部實(shí)現(xiàn)統(tǒng)一的序列化契約to_dict() - dict[str, Any]將組件序列化為字典。InMemory 兩個檢索器還會把filter_policy序列化為字符串值filter_policy.value便于 YAML/JSON 表示。from_dict(cls, data) - 組件實(shí)例類方法反序列化。InMemory 兩個檢索器在from_dict中用FilterPolicy.from_str將字符串還原為枚舉后再走default_from_dict。這使得整個檢索組件可以被完整嵌入 Haystack 的 YAML 管道描述見 haystack/marshal/yaml.py與Pipeline.loads/Pipeline.dumps流程實(shí)現(xiàn)聲明式定義、可復(fù)現(xiàn)部署。選型速查五個檢索器怎么選場景推薦組件關(guān)鍵理由關(guān)鍵詞/詞法匹配無需模型InMemoryBM25Retriever純 BM25 算法無需 embedding速度快、零模型依賴語義相似檢索已具備向量InMemoryEmbeddingRetriever接收query_embedding配合TextEmbedder/DocumentEmbedder只按元數(shù)據(jù)條件過濾FilterRetriever不做相似度計(jì)算僅執(zhí)行過濾器Comparison/Logic 語法檢索后需要上下文擴(kuò)展SentenceWindowRetriever基于source_id/split_id拉取鄰近分塊并去重疊合并層級分塊下避免碎片化召回AutoMergingRetriever命中同一父節(jié)點(diǎn)分塊比例超閾值時自動升級為父文檔結(jié)語Haystack 2.x 的 Retrievers API 以組件化、可序列化、可異步為設(shè)計(jì)主線BM25 與 Embedding 檢索器通過run/run_async雙入口薄封裝 Document Store 的原生檢索方法FilterPolicy統(tǒng)一了運(yùn)行期與初始化過濾器的 REPLACE/MERGE 語義SentenceWindowRetriever與AutoMergingRetriever則分別解決了碎片化上下文與碎片化召回兩類實(shí)際問題。理解這些組件的參數(shù)語義與底層調(diào)用鏈你就能依據(jù)業(yè)務(wù)場景快速搭建出高質(zhì)量的 RAG 檢索階段?！久赓M(fèi)下載鏈接】haystackOpen-source AI orchestration framework for building context-engineered, production-ready LLM applications. Design modular pipelines and agent workflows with explicit control over retrieval, routing, memory, and generation. Built for scalable agents, RAG, multimodal applications, semantic search, and conversational systems.項(xiàng)目地址: https://gitcode.com/GitHub_Trending/ha/haystack創(chuàng)作聲明：本文部分內(nèi)容由AI輔助生成（AIGC），僅供參考