topic: RAG Chunking, Step by Step
Recursive Chunking
How recursive chunking preserves document structure through progressively smaller separators, with size constraints, overlap, and a Gemini token-counting example.
Table of contents
Introduction
Recursive Chunking is one of the most commonly used chunking strategies in RAG systems.
Similar to Fixed Chunking, we still need to define:
- Chunk size: the maximum size of each chunk.
- Overlap size: the amount of content shared between consecutive chunks.
The size can be measured in characters or tokens, depending on the implementation.
The main difference from Fixed Chunking is how the chunk boundaries are selected.
Fixed Chunking mainly splits text according to size. This means a chunk boundary may occur in the middle of a paragraph or even a sentence.
Recursive Chunking tries to avoid this problem by splitting the document using progressively smaller structural boundaries.
How It Works
Suppose we set:
chunk_size = 500 tokens
chunk_overlap = 50 tokens
Instead of immediately cutting the document every 500 tokens, Recursive Chunking first tries to split the document using larger and more meaningful separators.
Conceptually, we can think about a document as having several levels of structure:
Document
↓
Page
↓
Paragraph
↓
Sentence
↓
Clause
↓
Word
These structures may be represented by separators such as:
Page → \f
Paragraph → \n\n
Line → \n
Sentence → ". "
Clause → ", "
Word → " "
The exact separators depend on the document format and splitter configuration.
The key idea is that the splitter tries larger boundaries first.
For example:
1. Try to split by page
2. If a page is still too large, split it by paragraph
3. If a paragraph is still too large, split it by lines or sentences
4. If a sentence is still too large, split it by smaller separators
5. Continue until the content fits within the chunk size
This is why the method is called Recursive Chunking.
For example, imagine the following document:
Paragraph A: 180 tokens
Paragraph B: 210 tokens
Paragraph C: 170 tokens
With:
chunk_size = 500 tokens
the splitter may combine Paragraph A and Paragraph B:
Chunk 1
├── Paragraph A: 180 tokens
└── Paragraph B: 210 tokens
Total ≈ 390 tokens
Adding Paragraph C would make the chunk larger than the target:
180 + 210 + 170 = 560 tokens
Therefore, Paragraph C would be placed into the next chunk instead of unnecessarily cutting Paragraph B in the middle.
The result may look like:
Chunk 1
┌─────────────────────────────┐
│ Paragraph A │
│ Paragraph B │
└─────────────────────────────┘
Chunk 2
┌─────────────────────────────┐
│ overlap from Chunk 1 │
│ Paragraph C │
└─────────────────────────────┘
The overlap helps preserve context around the boundary between two chunks.
Compared with Fixed Chunking, the chunks may not all have exactly the same size. The goal is to remain within the configured size while preserving natural document boundaries whenever possible.
This generally produces cleaner chunks for retrieval because paragraphs and sentences are less likely to be unnecessarily broken apart.
Recursive Chunking vs. Fixed Chunking
The main difference can be summarized as:
Fixed Chunking
Document
──────────────────────────────────────────────
↓ 500 ↓ 500 ↓ 500
──────────────────────────────────────────────
Chunk 1 Chunk 2 Chunk 3
The boundary is primarily determined by size.
Recursive Chunking instead tries to find a better structural boundary:
Document
──────────────────────────────────────────────
Paragraph Paragraph
↓ ↓
──────────────────────────────────────────────
Chunk 1 Chunk 2
In other words:
Fixed Chunking
Size first
Recursive Chunking
Structure first
↓
Size constraint
Recursive Chunking does not understand the meaning of the text. It uses document structure and separators to choose better boundaries.
Downsides
The main limitation of Recursive Chunking is semantic awareness.
The splitter can preserve structural boundaries such as paragraphs or sentences, but it does not actually understand what those sentences mean.
For example:
Sentence 1: The company reported strong revenue growth.
Sentence 2: The database uses PostgreSQL.
Sentence 3: Operating margin increased by 8%.
Structurally, all three sentences may fit into the same chunk.
However, Sentence 2 is semantically unrelated to Sentences 1 and 3.
The opposite problem can also happen:
Chunk 1:
The company introduced a new recommendation model...
Chunk 2:
...which increased click-through rate by 12%.
The two pieces are semantically related, but they may still end up in different chunks because of the configured size and structural boundaries.
Therefore:
Recursive Chunking
understands structure
but
does not understand semantics
This limitation motivates more advanced approaches such as Semantic Chunking, where chunk boundaries are determined partly by changes in meaning rather than only document structure.
Practical Code Quick View
If our RAG application uses a Google Gemini model, we can use Gemini's tokenizer to measure the size of each chunk instead of using OpenAI's tiktoken.
from langchain_google_genai import ChatGoogleGenerativeAI
from langchain_text_splitters import RecursiveCharacterTextSplitter
# Google Gemini model
llm = ChatGoogleGenerativeAI(
model="gemini-3.8-flash"
)
# Recursive chunking using Gemini token count
text_splitter = RecursiveCharacterTextSplitter(
chunk_size=500,
chunk_overlap=50,
length_function=llm.get_num_tokens,
separators=[
"\f", # page
"\n\n", # paragraph
"\n", # line
". ", # sentence
", ", # clause
" ", # word
"", # character fallback
],
)
chunks = text_splitter.split_text(document_text)
Here:
chunk_size = 500 tokens
chunk_overlap = 50 tokens
The important part is:
length_function=llm.get_num_tokens
By default, RecursiveCharacterTextSplitter can measure length using characters. By providing Gemini's get_num_tokens() function, the chunk size is instead measured according to the Gemini model's tokenizer.
Therefore:
Document
↓
RecursiveCharacterTextSplitter
↓
Try separators recursively
↓
Gemini tokenizer checks token size
↓
Keep each chunk around ≤ 500 tokens
One important distinction is that Gemini is not deciding where the semantic boundaries are.
The recursive splitter still determines boundaries using separators:
\f → \n\n → \n → ". " → ", " → " " → character
Gemini is only being used here to answer:
"How many tokens does this piece of text contain?"
If the input is already represented as LangChain Document objects, we can split them directly:
chunks = text_splitter.split_documents(documents)
Each resulting chunk remains a Document, so its text and metadata can continue through the rest of the RAG pipeline.