如何检索块的整个文档

先决条件

本指南假定您熟悉以下概念

在拆分文档以进行检索时，通常存在冲突的需求

您可能希望拥有小文档，以便它们的嵌入可以最准确地反映其含义。如果文档太长，则嵌入可能会失去意义。
您希望拥有足够长的文档，以便保留每个块的上下文。

ParentDocumentRetriever 通过拆分和存储小数据块来平衡这种需求。在检索期间，它首先获取小块，然后查找这些块的父 ID 并返回那些更大的文档。

请注意，“父文档”指的是小块的原始文档。这可以是整个原始文档，也可以是更大的块。

这是为每个文档生成多个嵌入的更具体形式。

用法

提示

请参阅本节，了解有关安装集成包的通用说明。

npm
Yarn
pnpm

npm install @langchain/openai @langchain/core

yarn add @langchain/openai @langchain/core

pnpm add @langchain/openai @langchain/core

import { OpenAIEmbeddings } from "@langchain/openai";
import { MemoryVectorStore } from "langchain/vectorstores/memory";
import { ParentDocumentRetriever } from "langchain/retrievers/parent_document";
import { RecursiveCharacterTextSplitter } from "@langchain/textsplitters";
import { TextLoader } from "langchain/document_loaders/fs/text";
import { InMemoryStore } from "@langchain/core/stores";

const vectorstore = new MemoryVectorStore(new OpenAIEmbeddings());
const byteStore = new InMemoryStore<Uint8Array>();

const retriever = new ParentDocumentRetriever({
  vectorstore,
  byteStore,
  // Optional, not required if you're already passing in split documents
  parentSplitter: new RecursiveCharacterTextSplitter({
    chunkOverlap: 0,
    chunkSize: 500,
  }),
  childSplitter: new RecursiveCharacterTextSplitter({
    chunkOverlap: 0,
    chunkSize: 50,
  }),
  // Optional `k` parameter to search for more child documents in VectorStore.
  // Note that this does not exactly correspond to the number of final (parent) documents
  // retrieved, as multiple child documents can point to the same parent.
  childK: 20,
  // Optional `k` parameter to limit number of final, parent documents returned from this
  // retriever and sent to LLM. This is an upper-bound, and the final count may be lower than this.
  parentK: 5,
});
const textLoader = new TextLoader("../examples/state_of_the_union.txt");
const parentDocuments = await textLoader.load();

// We must add the parent documents via the retriever's addDocuments method
await retriever.addDocuments(parentDocuments);

const retrievedDocs = await retriever.invoke("justice breyer");

// Retrieved chunks are the larger parent chunks
console.log(retrievedDocs);
/*
  [
    Document {
      pageContent: 'Tonight, I call on the Senate to pass — pass the Freedom to Vote Act. Pass the John Lewis Act — Voting Rights Act. And while you’re at it, pass the DISCLOSE Act so Americans know who is funding our elections.\n' +
        '\n' +
        'Look, tonight, I’d — I’d like to honor someone who has dedicated his life to serve this country: Justice Breyer — an Army veteran, Constitutional scholar, retiring Justice of the United States Supreme Court.',
      metadata: { source: '../examples/state_of_the_union.txt', loc: [Object] }
    },
    Document {
      pageContent: 'As I did four days ago, I’ve nominated a Circuit Court of Appeals — Ketanji Brown Jackson. One of our nation’s top legal minds who will continue in just Brey- — Justice Breyer’s legacy of excellence. A former top litigator in private practice, a former federal public defender from a family of public-school educators and police officers — she’s a consensus builder.',
      metadata: { source: '../examples/state_of_the_union.txt', loc: [Object] }
    },
    Document {
      pageContent: 'Justice Breyer, thank you for your service. Thank you, thank you, thank you. I mean it. Get up. Stand — let me see you. Thank you.\n' +
        '\n' +
        'And we all know — no matter what your ideology, we all know one of the most serious constitutional responsibilities a President has is nominating someone to serve on the United States Supreme Court.',
      metadata: { source: '../examples/state_of_the_union.txt', loc: [Object] }
    }
  ]
*/

API 参考

OpenAIEmbeddings 来自 @langchain/openai
MemoryVectorStore 来自 langchain/vectorstores/memory
ParentDocumentRetriever 来自 langchain/retrievers/parent_document
RecursiveCharacterTextSplitter 来自 @langchain/textsplitters
TextLoader 来自 langchain/document_loaders/fs/text
InMemoryStore 来自 @langchain/core/stores

带分数阈值

通过在 scoreThresholdOptions 中设置选项，我们可以强制 ParentDocumentRetriever 在后台使用 ScoreThresholdRetriever。这会将 ScoreThresholdRetriever 内部的向量存储设置为我们在初始化 ParentDocumentRetriever 时传递的向量存储，同时还允许我们为检索器设置分数阈值。

当您不确定需要多少文档时（或者如果您确定，只需设置 maxK 选项），但您想确保您获得的文档在一定的相关性阈值内时，这会很有帮助。

注意：如果传递了检索器，则 ParentDocumentRetriever 将默认使用它来检索小块，以及通过 addDocuments 方法添加文档。

import { OpenAIEmbeddings } from "@langchain/openai";
import { MemoryVectorStore } from "langchain/vectorstores/memory";
import { InMemoryStore } from "@langchain/core/stores";
import { ParentDocumentRetriever } from "langchain/retrievers/parent_document";
import { RecursiveCharacterTextSplitter } from "@langchain/textsplitters";
import { TextLoader } from "langchain/document_loaders/fs/text";
import { ScoreThresholdRetriever } from "langchain/retrievers/score_threshold";

const vectorstore = new MemoryVectorStore(new OpenAIEmbeddings());
const byteStore = new InMemoryStore<Uint8Array>();

const childDocumentRetriever = ScoreThresholdRetriever.fromVectorStore(
  vectorstore,
  {
    minSimilarityScore: 0.01, // Essentially no threshold
    maxK: 1, // Only return the top result
  }
);
const retriever = new ParentDocumentRetriever({
  vectorstore,
  byteStore,
  childDocumentRetriever,
  // Optional, not required if you're already passing in split documents
  parentSplitter: new RecursiveCharacterTextSplitter({
    chunkOverlap: 0,
    chunkSize: 500,
  }),
  childSplitter: new RecursiveCharacterTextSplitter({
    chunkOverlap: 0,
    chunkSize: 50,
  }),
});
const textLoader = new TextLoader("../examples/state_of_the_union.txt");
const parentDocuments = await textLoader.load();

// We must add the parent documents via the retriever's addDocuments method
await retriever.addDocuments(parentDocuments);

const retrievedDocs = await retriever.invoke("justice breyer");

// Retrieved chunk is the larger parent chunk
console.log(retrievedDocs);
/*
  [
    Document {
      pageContent: 'Tonight, I call on the Senate to pass — pass the Freedom to Vote Act. Pass the John Lewis Act — Voting Rights Act. And while you’re at it, pass the DISCLOSE Act so Americans know who is funding our elections.\n' +
        '\n' +
        'Look, tonight, I’d — I’d like to honor someone who has dedicated his life to serve this country: Justice Breyer — an Army veteran, Constitutional scholar, retiring Justice of the United States Supreme Court.',
      metadata: { source: '../examples/state_of_the_union.txt', loc: [Object] }
    },
  ]
*/

API 参考

OpenAIEmbeddings 来自 @langchain/openai
MemoryVectorStore 来自 langchain/vectorstores/memory
InMemoryStore 来自 @langchain/core/stores
ParentDocumentRetriever 来自 langchain/retrievers/parent_document
RecursiveCharacterTextSplitter 来自 @langchain/textsplitters
TextLoader 来自 langchain/document_loaders/fs/text
ScoreThresholdRetriever 来自 langchain/retrievers/score_threshold

带上下文块头

考虑这样一种情况：您想将文档集合存储在向量存储中，并对它们执行问答任务。仅使用重叠文本拆分文档可能无法为 LLM 提供足够的上下文来确定多个块是否引用相同的信息，或者如何解决来自矛盾来源的信息。

如果您知道要针对什么进行过滤，则使用元数据标记每个文档是一种解决方案，但您可能无法提前知道您的向量存储需要处理哪种类型的查询。在每个块中以标头的形式直接包含额外的上下文信息可以帮助处理任意查询。

如果您有几个需要从向量存储中正确检索的细粒度子块，这一点尤其重要。

import { OpenAIEmbeddings } from "@langchain/openai";
import { HNSWLib } from "@langchain/community/vectorstores/hnswlib";
import { InMemoryStore } from "@langchain/core/stores";
import { ParentDocumentRetriever } from "langchain/retrievers/parent_document";
import { RecursiveCharacterTextSplitter } from "@langchain/textsplitters";

const splitter = new RecursiveCharacterTextSplitter({
  chunkSize: 1500,
  chunkOverlap: 0,
});

const jimDocs = await splitter.createDocuments([`My favorite color is blue.`]);
const jimChunkHeaderOptions = {
  chunkHeader: "DOC NAME: Jim Interview\n---\n",
  appendChunkOverlapHeader: true,
};

const pamDocs = await splitter.createDocuments([`My favorite color is red.`]);
const pamChunkHeaderOptions = {
  chunkHeader: "DOC NAME: Pam Interview\n---\n",
  appendChunkOverlapHeader: true,
};

const vectorstore = await HNSWLib.fromDocuments([], new OpenAIEmbeddings());
const byteStore = new InMemoryStore<Uint8Array>();

const retriever = new ParentDocumentRetriever({
  vectorstore,
  byteStore,
  // Very small chunks for demo purposes.
  // Use a bigger chunk size for serious use-cases.
  childSplitter: new RecursiveCharacterTextSplitter({
    chunkSize: 10,
    chunkOverlap: 0,
  }),
  childK: 50,
  parentK: 5,
});

// We pass additional option `childDocChunkHeaderOptions`
// that will add the chunk header to child documents
await retriever.addDocuments(jimDocs, {
  childDocChunkHeaderOptions: jimChunkHeaderOptions,
});
await retriever.addDocuments(pamDocs, {
  childDocChunkHeaderOptions: pamChunkHeaderOptions,
});

// This will search child documents in vector store with the help of chunk header,
// returning the unmodified parent documents
const retrievedDocs = await retriever.invoke("What is Pam's favorite color?");

// Pam's favorite color is returned first!
console.log(JSON.stringify(retrievedDocs, null, 2));
/*
  [
    {
      "pageContent": "My favorite color is red.",
      "metadata": {
        "loc": {
          "lines": {
            "from": 1,
            "to": 1
          }
        }
      }
    },
    {
      "pageContent": "My favorite color is blue.",
      "metadata": {
        "loc": {
          "lines": {
            "from": 1,
            "to": 1
          }
        }
      }
    }
  ]
*/

const rawDocs = await vectorstore.similaritySearch(
  "What is Pam's favorite color?"
);

// Raw docs in vectorstore are short but have chunk headers
console.log(JSON.stringify(rawDocs, null, 2));

/*
  [
    {
      "pageContent": "DOC NAME: Pam Interview\n---\n(cont'd) color is",
      "metadata": {
        "loc": {
          "lines": {
            "from": 1,
            "to": 1
          }
        },
        "doc_id": "affdcbeb-6bfb-42e9-afe5-80f4f2e9f6aa"
      }
    },
    {
      "pageContent": "DOC NAME: Pam Interview\n---\n(cont'd) favorite",
      "metadata": {
        "loc": {
          "lines": {
            "from": 1,
            "to": 1
          }
        },
        "doc_id": "affdcbeb-6bfb-42e9-afe5-80f4f2e9f6aa"
      }
    },
    {
      "pageContent": "DOC NAME: Pam Interview\n---\n(cont'd) red.",
      "metadata": {
        "loc": {
          "lines": {
            "from": 1,
            "to": 1
          }
        },
        "doc_id": "affdcbeb-6bfb-42e9-afe5-80f4f2e9f6aa"
      }
    },
    {
      "pageContent": "DOC NAME: Pam Interview\n---\nMy",
      "metadata": {
        "loc": {
          "lines": {
            "from": 1,
            "to": 1
          }
        },
        "doc_id": "affdcbeb-6bfb-42e9-afe5-80f4f2e9f6aa"
      }
    }
  ]
*/

API 参考

OpenAIEmbeddings 来自 @langchain/openai
HNSWLib 来自 @langchain/community/vectorstores/hnswlib
InMemoryStore 来自 @langchain/core/stores
ParentDocumentRetriever 来自 langchain/retrievers/parent_document
RecursiveCharacterTextSplitter 来自 @langchain/textsplitters

带重新排序

当将来自向量存储的许多文档传递给 LLM 时，最终答案有时包含来自不相关块的信息，这使其不太精确，有时甚至不正确。此外，传递多个不相关的文档会使其更加昂贵。因此，使用重新排序有两个原因 - 精度和成本。

import { OpenAIEmbeddings } from "@langchain/openai";
import { CohereRerank } from "@langchain/cohere";
import { HNSWLib } from "@langchain/community/vectorstores/hnswlib";
import { InMemoryStore } from "@langchain/core/stores";
import {
  ParentDocumentRetriever,
  type SubDocs,
} from "langchain/retrievers/parent_document";
import { RecursiveCharacterTextSplitter } from "@langchain/textsplitters";

// init Cohere Rerank. Remember to add COHERE_API_KEY to your .env
const reranker = new CohereRerank({
  topN: 50,
  model: "rerank-multilingual-v2.0",
});

export function documentCompressorFiltering({
  relevanceScore,
}: { relevanceScore?: number } = {}) {
  return (docs: SubDocs) => {
    let outputDocs = docs;

    if (relevanceScore) {
      const docsRelevanceScoreValues = docs.map(
        (doc) => doc?.metadata?.relevanceScore
      );
      outputDocs = docs.filter(
        (_doc, index) =>
          (docsRelevanceScoreValues?.[index] || 1) >= relevanceScore
      );
    }

    return outputDocs;
  };
}

const splitter = new RecursiveCharacterTextSplitter({
  chunkSize: 500,
  chunkOverlap: 0,
});

const jimDocs = await splitter.createDocuments([`Jim favorite color is blue.`]);

const pamDocs = await splitter.createDocuments([`Pam favorite color is red.`]);

const vectorstore = await HNSWLib.fromDocuments([], new OpenAIEmbeddings());
const byteStore = new InMemoryStore<Uint8Array>();

const retriever = new ParentDocumentRetriever({
  vectorstore,
  byteStore,
  // Very small chunks for demo purposes.
  // Use a bigger chunk size for serious use-cases.
  childSplitter: new RecursiveCharacterTextSplitter({
    chunkSize: 10,
    chunkOverlap: 0,
  }),
  childK: 50,
  parentK: 5,
  // We add Reranker
  documentCompressor: reranker,
  documentCompressorFilteringFn: documentCompressorFiltering({
    relevanceScore: 0.3,
  }),
});

const docs = jimDocs.concat(pamDocs);
await retriever.addDocuments(docs);

// This will search for documents in vector store and return for LLM already reranked and sorted document
// with appropriate minimum relevance score
const retrievedDocs = await retriever.invoke("What is Pam's favorite color?");

// Pam's favorite color is returned first!
console.log(JSON.stringify(retrievedDocs, null, 2));
/*
  [
    {
      "pageContent": "My favorite color is red.",
      "metadata": {
        "relevanceScore": 0.9
        "loc": {
          "lines": {
            "from": 1,
            "to": 1
          }
        }
      }
    }
  ]
*/

API 参考

OpenAIEmbeddings 来自 @langchain/openai
CohereRerank 来自 @langchain/cohere
HNSWLib 来自 @langchain/community/vectorstores/hnswlib
InMemoryStore 来自 @langchain/core/stores
ParentDocumentRetriever 来自 langchain/retrievers/parent_document
SubDocs 来自 langchain/retrievers/parent_document
RecursiveCharacterTextSplitter 来自 @langchain/textsplitters

下一步

您现在已经学习了如何使用 ParentDocumentRetriever。

接下来，查看为每个文档生成多个嵌入的更通用形式，关于 RAG 的更广泛教程，或本节了解如何在任何数据源上创建您自己的自定义检索器。

如何检索块的整个文档

用法

API 参考

带分数阈值

API 参考

带上下文块头

API 参考

带重新排序

API 参考

下一步

此页对您有帮助吗？

您也可以留下详细的反馈在 GitHub 上.

如何检索块的整个文档

用法​

API 参考

带分数阈值​

API 参考

带上下文块头​

API 参考

带重新排序​

API 参考

下一步​

此页对您有帮助吗？

您也可以留下详细的反馈 在 GitHub 上.

用法

带分数阈值

带上下文块头

带重新排序

下一步

您也可以留下详细的反馈在 GitHub 上.