如何拆分代码
先决条件
本指南假定您熟悉以下概念
RecursiveCharacterTextSplitter 包含预构建的分隔符列表，这些分隔符对于在特定编程语言中拆分文本非常有用。
支持的语言包括
"html" | "cpp" | "go" | "java" | "js" | "php" | "proto" | "python" | "rst" | "ruby" | "rust" | "scala" | "swift" | "markdown" | "latex" | "sol"
要查看给定语言的分隔符列表，请将上面列表中的一个值传递到 getSeparatorsForLanguage() 静态方法中
import { RecursiveCharacterTextSplitter } from "@langchain/textsplitters";
RecursiveCharacterTextSplitter.getSeparatorsForLanguage("js");
[
"\nfunction ", "\nconst ",
"\nlet ", "\nvar ",
"\nclass ", "\nif ",
"\nfor ", "\nwhile ",
"\nswitch ", "\ncase ",
"\ndefault ", "\n\n",
"\n", " ",
""
]
JS
以下是一个使用 JS 文本拆分器的示例
const JS_CODE = `
function helloWorld() {
console.log("Hello, World!");
}
// Call the function
helloWorld();
`;
const jsSplitter = RecursiveCharacterTextSplitter.fromLanguage("js", {
chunkSize: 60,
chunkOverlap: 0,
});
const jsDocs = await jsSplitter.createDocuments([JS_CODE]);
jsDocs;
[
Document {
pageContent: 'function helloWorld() {\n console.log("Hello, World!");\n}',
metadata: { loc: { lines: { from: 2, to: 4 } } }
},
Document {
pageContent: "// Call the function\nhelloWorld();",
metadata: { loc: { lines: { from: 6, to: 7 } } }
}
]
Python
以下是一个 Python 示例
const PYTHON_CODE = `
def hello_world():
print("Hello, World!")
# Call the function
hello_world()
`;
const pythonSplitter = RecursiveCharacterTextSplitter.fromLanguage("python", {
chunkSize: 50,
chunkOverlap: 0,
});
const pythonDocs = await pythonSplitter.createDocuments([PYTHON_CODE]);
pythonDocs;
[
Document {
pageContent: 'def hello_world():\n print("Hello, World!")',
metadata: { loc: { lines: { from: 2, to: 3 } } }
},
Document {
pageContent: "# Call the function\nhello_world()",
metadata: { loc: { lines: { from: 5, to: 6 } } }
}
]
Markdown
以下是在 markdown 分隔符上拆分的示例
const markdownText = `
# 🦜️🔗 LangChain
⚡ Building applications with LLMs through composability ⚡
## Quick Install
\`\`\`bash
# Hopefully this code block isn't split
pip install langchain
\`\`\`
As an open-source project in a rapidly developing field, we are extremely open to contributions.
`;
const mdSplitter = RecursiveCharacterTextSplitter.fromLanguage("markdown", {
chunkSize: 60,
chunkOverlap: 0,
});
const mdDocs = await mdSplitter.createDocuments([markdownText]);
mdDocs;
[
Document {
pageContent: "# 🦜️🔗 LangChain",
metadata: { loc: { lines: { from: 2, to: 2 } } }
},
Document {
pageContent: "⚡ Building applications with LLMs through composability ⚡",
metadata: { loc: { lines: { from: 4, to: 4 } } }
},
Document {
pageContent: "## Quick Install",
metadata: { loc: { lines: { from: 6, to: 6 } } }
},
Document {
pageContent: "```bash\n# Hopefully this code block isn't split",
metadata: { loc: { lines: { from: 8, to: 9 } } }
},
Document {
pageContent: "pip install langchain",
metadata: { loc: { lines: { from: 10, to: 10 } } }
},
Document {
pageContent: "```",
metadata: { loc: { lines: { from: 11, to: 11 } } }
},
Document {
pageContent: "As an open-source project in a rapidly developing field, we",
metadata: { loc: { lines: { from: 13, to: 13 } } }
},
Document {
pageContent: "are extremely open to contributions.",
metadata: { loc: { lines: { from: 13, to: 13 } } }
}
]
Latex
以下是在 Latex 文本上的示例
const latexText = `
\documentclass{article}
\begin{document}
\maketitle
\section{Introduction}
Large language models (LLMs) are a type of machine learning model that can be trained on vast amounts of text data to generate human-like language. In recent years, LLMs have made significant advances in a variety of natural language processing tasks, including language translation, text generation, and sentiment analysis.
\subsection{History of LLMs}
The earliest LLMs were developed in the 1980s and 1990s, but they were limited by the amount of data that could be processed and the computational power available at the time. In the past decade, however, advances in hardware and software have made it possible to train LLMs on massive datasets, leading to significant improvements in performance.
\subsection{Applications of LLMs}
LLMs have many applications in industry, including chatbots, content creation, and virtual assistants. They can also be used in academia for research in linguistics, psychology, and computational linguistics.
\end{document}
`;
const latexSplitter = RecursiveCharacterTextSplitter.fromLanguage("latex", {
chunkSize: 60,
chunkOverlap: 0,
});
const latexDocs = await latexSplitter.createDocuments([latexText]);
latexDocs;
[
Document {
pageContent: "documentclass{article}\n\n\begin{document}\n\nmaketitle",
metadata: { loc: { lines: { from: 2, to: 6 } } }
},
Document {
pageContent: "section{Introduction}",
metadata: { loc: { lines: { from: 8, to: 8 } } }
},
Document {
pageContent: "Large language models (LLMs) are a type of machine learning",
metadata: { loc: { lines: { from: 9, to: 9 } } }
},
Document {
pageContent: "model that can be trained on vast amounts of text data to",
metadata: { loc: { lines: { from: 9, to: 9 } } }
},
Document {
pageContent: "generate human-like language. In recent years, LLMs have",
metadata: { loc: { lines: { from: 9, to: 9 } } }
},
Document {
pageContent: "made significant advances in a variety of natural language",
metadata: { loc: { lines: { from: 9, to: 9 } } }
},
Document {
pageContent: "processing tasks, including language translation, text",
metadata: { loc: { lines: { from: 9, to: 9 } } }
},
Document {
pageContent: "generation, and sentiment analysis.",
metadata: { loc: { lines: { from: 9, to: 9 } } }
},
Document {
pageContent: "subsection{History of LLMs}",
metadata: { loc: { lines: { from: 11, to: 11 } } }
},
Document {
pageContent: "The earliest LLMs were developed in the 1980s and 1990s,",
metadata: { loc: { lines: { from: 12, to: 12 } } }
},
Document {
pageContent: "but they were limited by the amount of data that could be",
metadata: { loc: { lines: { from: 12, to: 12 } } }
},
Document {
pageContent: "processed and the computational power available at the",
metadata: { loc: { lines: { from: 12, to: 12 } } }
},
Document {
pageContent: "time. In the past decade, however, advances in hardware and",
metadata: { loc: { lines: { from: 12, to: 12 } } }
},
Document {
pageContent: "software have made it possible to train LLMs on massive",
metadata: { loc: { lines: { from: 12, to: 12 } } }
},
Document {
pageContent: "datasets, leading to significant improvements in",
metadata: { loc: { lines: { from: 12, to: 12 } } }
},
Document {
pageContent: "performance.",
metadata: { loc: { lines: { from: 12, to: 12 } } }
},
Document {
pageContent: "subsection{Applications of LLMs}",
metadata: { loc: { lines: { from: 14, to: 14 } } }
},
Document {
pageContent: "LLMs have many applications in industry, including",
metadata: { loc: { lines: { from: 15, to: 15 } } }
},
Document {
pageContent: "chatbots, content creation, and virtual assistants. They",
metadata: { loc: { lines: { from: 15, to: 15 } } }
},
Document {
pageContent: "can also be used in academia for research in linguistics,",
metadata: { loc: { lines: { from: 15, to: 15 } } }
},
Document {
pageContent: "psychology, and computational linguistics.",
metadata: { loc: { lines: { from: 15, to: 15 } } }
},
Document {
pageContent: "end{document}",
metadata: { loc: { lines: { from: 17, to: 17 } } }
}
]
HTML
以下是一个使用 HTML 文本拆分器的示例
const htmlText = `
<!DOCTYPE html>
<html>
<head>
<title>🦜️🔗 LangChain</title>
<style>
body {
font-family: Arial, sans-serif;
}
h1 {
color: darkblue;
}
</style>
</head>
<body>
<div>
<h1>🦜️🔗 LangChain</h1>
<p>⚡ Building applications with LLMs through composability ⚡</p>
</div>
<div>
As an open-source project in a rapidly developing field, we are extremely open to contributions.
</div>
</body>
</html>
`;
const htmlSplitter = RecursiveCharacterTextSplitter.fromLanguage("html", {
chunkSize: 60,
chunkOverlap: 0,
});
const htmlDocs = await htmlSplitter.createDocuments([htmlText]);
htmlDocs;
[
Document {
pageContent: "<!DOCTYPE html>\n<html>",
metadata: { loc: { lines: { from: 2, to: 3 } } }
},
Document {
pageContent: "<head>\n <title>🦜️🔗 LangChain</title>",
metadata: { loc: { lines: { from: 4, to: 5 } } }
},
Document {
pageContent: "<style>\n body {\n font-family:",
metadata: { loc: { lines: { from: 6, to: 8 } } }
},
Document {
pageContent: "Arial, sans-serif;\n }\n h1 {",
metadata: { loc: { lines: { from: 8, to: 10 } } }
},
Document {
pageContent: "color: darkblue;\n }\n </style>",
metadata: { loc: { lines: { from: 11, to: 13 } } }
},
Document {
pageContent: "</head>",
metadata: { loc: { lines: { from: 14, to: 14 } } }
},
Document {
pageContent: "<body>",
metadata: { loc: { lines: { from: 15, to: 15 } } }
},
Document {
pageContent: "<div>\n <h1>🦜️🔗 LangChain</h1>",
metadata: { loc: { lines: { from: 16, to: 17 } } }
},
Document {
pageContent: "<p>⚡ Building applications with LLMs through composability",
metadata: { loc: { lines: { from: 18, to: 18 } } }
},
Document {
pageContent: "⚡</p>\n </div>",
metadata: { loc: { lines: { from: 18, to: 19 } } }
},
Document {
pageContent: "<div>\n As an open-source project in a rapidly",
metadata: { loc: { lines: { from: 20, to: 21 } } }
},
Document {
pageContent: "developing field, we are extremely open to contributions.",
metadata: { loc: { lines: { from: 21, to: 21 } } }
},
Document {
pageContent: "</div>\n </body>\n</html>",
metadata: { loc: { lines: { from: 22, to: 24 } } }
}
]
Solidity
以下是在 Solidity 代码上拆分的示例
const SOL_CODE = `
pragma solidity ^0.8.20;
contract HelloWorld {
function add(uint a, uint b) pure public returns(uint) {
return a + b;
}
}
`;
const solSplitter = RecursiveCharacterTextSplitter.fromLanguage("sol", {
chunkSize: 128,
chunkOverlap: 0,
});
const solDocs = await solSplitter.createDocuments([SOL_CODE]);
solDocs;
[
Document {
pageContent: "pragma solidity ^0.8.20;",
metadata: { loc: { lines: { from: 2, to: 2 } } }
},
Document {
pageContent: "contract HelloWorld {\n" +
" function add(uint a, uint b) pure public returns(uint) {\n" +
" return a + "... 9 more characters,
metadata: { loc: { lines: { from: 3, to: 7 } } }
}
]
PHP
以下是在 PHP 代码上拆分的示例
const PHP_CODE = `<?php
namespace foo;
class Hello {
public function __construct() { }
}
function hello() {
echo "Hello World!";
}
interface Human {
public function breath();
}
trait Foo { }
enum Color
{
case Red;
case Blue;
}`;
const phpSplitter = RecursiveCharacterTextSplitter.fromLanguage("php", {
chunkSize: 50,
chunkOverlap: 0,
});
const phpDocs = await phpSplitter.createDocuments([PHP_CODE]);
phpDocs;
[
Document {
pageContent: "<?php\nnamespace foo;",
metadata: { loc: { lines: { from: 1, to: 2 } } }
},
Document {
pageContent: "class Hello {",
metadata: { loc: { lines: { from: 3, to: 3 } } }
},
Document {
pageContent: "public function __construct() { }\n}",
metadata: { loc: { lines: { from: 4, to: 5 } } }
},
Document {
pageContent: 'function hello() {\n echo "Hello World!";\n}',
metadata: { loc: { lines: { from: 6, to: 8 } } }
},
Document {
pageContent: "interface Human {\n public function breath();\n}",
metadata: { loc: { lines: { from: 9, to: 11 } } }
},
Document {
pageContent: "trait Foo { }\nenum Color\n{\n case Red;",
metadata: { loc: { lines: { from: 12, to: 15 } } }
},
Document {
pageContent: "case Blue;\n}",
metadata: { loc: { lines: { from: 16, to: 17 } } }
}
]
下一步
您现在已经学习了一种在特定于代码的分隔符上拆分文本的方法。
接下来，查看有关 检索增强生成的完整教程。