7 Chunking Strategies for RAG Systems (Examples)

When building RAG systems, one design decision consistently has a bigger impact than most people expect: how documents are chunked before retrieval. Chunking determines what the retriever can actually find and pass to the model. Get it wrong and you fragment context, retrieve irrelevant passages, and increase hallucination rates.

This guide covers seven chunking strategies that improve RAG retrieval quality, with examples of when to use each one.

What Is Chunking in RAG?

Chunking in RAG is the process of breaking large documents into smaller pieces before they are embedded and stored for retrieval. Instead of searching an entire document, the system retrieves only the most relevant chunks and passes them to the language model to generate an answer.

This step is necessary because embedding models and LLMs have context limits. Splitting documents into structured chunks makes retrieval faster, more targeted, and more accurate.

Chunking determines what information the retriever can find, which directly affects answer quality, hallucination risk, and token usage across the entire RAG pipeline.

Why Chunking Matters for RAG Performance

Retrieval quality in a RAG system depends on how well documents are segmented. Chunking plays a central role because it shapes how information is stored and retrieved.

Good chunking improves retrieval precision by returning exactly the passage the query needs instead of unrelated text. It also preserves meaningful context by keeping related ideas together rather than splitting them across segments.

When the model receives complete and relevant context, it is less likely to hallucinate. Smaller, well-structured chunks also reduce embedding size, speed up vector search, and lower token usage during generation. In production RAG systems, chunking is not a preprocessing afterthought. It directly influences reliability at every stage of the pipeline.

7 Chunking Strategies for RAG

1. Fixed-Size Chunking

Fixed-size chunking splits documents into equal-length segments based on characters, words, or tokens. Every chunk has a predefined size, making it one of the simplest methods used in RAG pipelines.

The document is treated as a continuous stream of text and divided into uniform windows. To avoid losing context at boundaries, many systems introduce chunk overlap, allowing some content from one chunk to carry over into the next.

Example: A 20,000-character document with a chunk size of 1,000 characters and 200-character overlap produces:

• Chunk 1: characters 1 to 1000
• Chunk 2: characters 801 to 1800
• Chunk 3: characters 1601 to 2600

Works well for: logs, transcripts, flat knowledge bases, and early RAG prototypes.

Limitations: ignores sentence and paragraph boundaries, which can split important ideas mid-thought and return incomplete context during retrieval.

Fixed-size chunking is typically used as a baseline before moving to more structure-aware approaches.

2. Recursive Chunking

Recursive chunking splits documents by following natural structure rather than cutting by length alone. It uses a hierarchy of separators such as sections, paragraphs, sentences, and tokens to create chunks that respect logical boundaries.

The system attempts to split the document using the largest meaningful separator first. If a section is too large, the algorithm moves to smaller separators like paragraphs, then sentences, then tokens if necessary. This recursive process keeps coherent units of information together while still enforcing chunk size limits.

Example: In a Python documentation page, recursive chunking keeps an entire class or function block intact instead of splitting it mid-definition.

Works well for: technical documentation, API references, source code, and structured manuals.

Limitations: slower than fixed-size chunking and depends on clear formatting and separators, which can produce uneven chunk sizes.

Recursive chunking is one of the most commonly used strategies in production RAG systems because it balances structure preservation with size control.

3. Document-Based Chunking

Document-based chunking splits content based on large logical sections such as chapters, clauses, or sections, rather than aggressively breaking text into smaller pieces. Each chunk represents a complete conceptual unit.

Instead of optimizing for small chunk sizes, the system keeps entire sections of a document intact. These sections are embedded and stored as individual retrieval units. This approach prioritizes context preservation over granularity.

Example: In a legal contract, the document splits like this:

• Clause 1: Definitions
• Clause 2: Payment Terms
• Clause 3: Termination

Each clause becomes its own retrieval unit, even if it runs several hundred tokens.

Works well for: legal documents, medical reports, scientific papers, and policy manuals.

Limitations: large chunks can reduce retrieval precision and push more tokens into the model context per query.

Document-based chunking is best when contextual completeness matters more than granularity.

4. Semantic Chunking

Semantic chunking splits text based on changes in meaning or topic rather than fixed length or structural separators. The goal is to keep related ideas together so each chunk represents a coherent concept.

This method analyzes the semantic similarity between sentences using embeddings. When similarity between consecutive sentences drops below a threshold, the system creates a new chunk. The result is chunks that reflect the natural flow of ideas rather than arbitrary text lengths.

Example: In a research paper, semantic chunking may produce:

• Discussion of Transformer Architecture: one chunk
• Transition to Training Data: new chunk
• Shift to Evaluation Metrics: another chunk

Works well for: research papers, knowledge bases, technical documentation, and long-form educational content.

Limitations: requires embedding calculations during preprocessing and needs careful threshold tuning. It is more computationally expensive than simpler methods.

Semantic chunking often delivers higher retrieval relevance in multi-topic documents.

5. Token-Based Chunking

Token-based chunking splits text based on token limits defined by the embedding model or LLM. Each chunk is created to stay within the model's maximum token capacity.

The system converts the document into tokens and divides it into segments that do not exceed the model's token limit. This ensures every chunk can be safely embedded and passed to the language model without hitting context constraints. Some implementations add small overlaps between chunks to preserve continuity.

Example: With an embedding model supporting 512 tokens per input:

• Chunk 1: tokens 1 to 512
• Chunk 2: tokens 513 to 1024
• Chunk 3: tokens 1025 to 1536

Works well for: large-scale indexing pipelines, streaming data ingestion, and systems with strict context limits.

Limitations: ignores sentence and paragraph structure, which can split ideas mid-sentence and reduce semantic coherence during retrieval.

Token-based chunking is often used as a safety layer in RAG pipelines, even when other strategies are applied first.

6. Sentence-Based Chunking

Sentence-based chunking groups text into chunks by combining complete sentences rather than splitting by characters or tokens. This ensures each chunk contains grammatically complete thoughts.

The document is first divided into individual sentences using sentence boundary detection. Sentences are then grouped together until the chunk reaches a target size while preserving natural language boundaries. This keeps ideas intact and improves the coherence of retrieved context.

Example: A tutorial paragraph may be grouped like this:

• Chunk 1: Sentences 1 to 5
• Chunk 2: Sentences 6 to 10
• Chunk 3: Sentences 11 to 15

Works well for: tutorials, conversational datasets, narrative content, and explanation-focused question answering.

Limitations: sentence lengths vary significantly, some sentences may exceed token limits, and chunk sizes can become inconsistent.

Sentence-based chunking often improves readability and contextual coherence in retrieval results.

7. Agentic Chunking

Agentic chunking organizes content based on tasks, roles, or reasoning objectives rather than text structure alone. Each chunk is designed to support a specific function in an AI workflow, such as answering questions, planning steps, or validating outputs.

Instead of storing raw text segments, the system creates task-oriented chunks aligned with how agents will use the information. Chunks are retrieved depending on the role of the agent or the step in the workflow. This approach is most useful in systems where multiple agents collaborate, each requiring different types of context.

Example: In a troubleshooting guide:

• Chunk 1: Problem description (retrieved by a diagnosis agent)
• Chunk 2: Step-by-step solution (retrieved by an execution agent)
• Chunk 3: Warnings and constraints (retrieved by a validation agent)

Works well for: autonomous agent systems, workflow orchestration, multi-step planning, and enterprise copilots.

Limitations: complex to design and maintain, requires clear task modeling, and needs tight coordination between agents and retrieval logic.

Agentic chunking is emerging as an advanced approach in modern AI systems where chunks are designed to support reasoning workflows, not just store text.

Chunking Strategy Comparison

How to Choose the Right Chunking Strategy

There is no single chunking strategy that works for every RAG system. The right approach depends on your data structure, query patterns, and model constraints.

Document structure is the first factor to consider. Structured documents like manuals and legal contracts work well with recursive chunking. Multi-topic documents like research papers benefit from semantic chunking. Unstructured data like logs and transcripts are better suited to fixed-size or token-based chunking.

Query type matters too. Short factual questions need smaller chunks for better precision. Analytical queries benefit from larger chunks that preserve surrounding context.

Model context limits should always inform chunk size. Token-based chunking is often applied as a safety layer to ensure chunks fit within model constraints regardless of the primary strategy used.

Precision vs. context is the core tradeoff. Smaller chunks improve retrieval precision. Larger chunks improve contextual understanding. Most production RAG systems combine multiple methods to balance both.

Best Practices for Production RAG Chunking

Start simple. Begin with fixed-size or recursive chunking before moving to more advanced methods. This makes it easier to measure what is actually improving retrieval.

Tune chunk size and overlap carefully. Chunks that are too small break context. Chunks that are too large reduce precision. An overlap of 10 to 20 percent helps preserve continuity between chunks. Most systems perform well with chunk sizes between 200 and 500 tokens.

Preserve natural boundaries. Avoid splitting in the middle of sentences, paragraphs, section headers, or code blocks. Keeping boundaries intact improves coherence and retrieval relevance.

Add metadata to every chunk. Include document title, section name, chunk index, and page number where possible. This helps with filtering, ranking, and traceability during retrieval.

Validate with real queries. The best chunking strategy is not the one that sounds best in theory. Test retrieval quality, answer relevance, and token usage using actual user queries. Inspect retrieved chunks manually to see whether the strategy is helping or introducing noise.

Tools for RAG Chunking

1. LangChain

Provides built-in text splitters supporting fixed-size, recursive, token-based, and sentence-based chunking. It integrates chunking directly with embeddings, vector databases, and LLM pipelines. Best for rapid prototyping and end-to-end RAG pipelines.

2. LlamaIndex

Focuses on document ingestion and indexing. It splits documents into nodes, attaches metadata, and builds structured indexes optimized for retrieval. Best for knowledge-base indexing and advanced RAG architectures.

3. Haystack

Offers complete pipelines for document ingestion, preprocessing, chunking, and retrieval. It includes configurable text splitters and integrates with multiple vector databases. Best for enterprise search and production RAG deployments.

4. Vector databases

Like Pinecone, Milvus, and FAISS store embeddings generated from chunks and enable fast similarity search. They are a core component of any RAG pipeline. Best for large-scale retrieval across millions of chunks.

5. NLP libraries

Like spaCy and NLTK handle sentence detection and linguistic preprocessing, which supports sentence-based and semantic chunking strategies. Best for custom chunking pipelines.

How to Evaluate Chunking Quality

Retrieval relevance is the first check. Are the retrieved chunks directly related to the query? Are important passages missing from the top results? Low retrieval relevance usually means the chunking strategy is splitting information too aggressively or grouping unrelated content together.

Context coverage tells you whether all the required information is present in the retrieved chunks. If supporting facts are scattered across too many chunks, the segmentation strategy needs adjustment.

Semantic coherence measures whether each chunk makes sense on its own. Chunks that begin or end abruptly, or that require neighboring text to be understood, indicate poor segmentation.

Answer quality is the ultimate test. If generated answers frequently contain unsupported claims or miss key facts, the chunks are not delivering the right context to the model.

System efficiency matters in production. Track the number of chunks stored, tokens passed to the model per query, and retrieval latency. Well-designed chunking improves both accuracy and operational performance.

Conclusion

Chunking is one of the most important decisions in building a reliable RAG system. The right strategy depends on your document structure, query patterns, and model constraints.

Fixed-size chunking offers simplicity, recursive chunking preserves structure, semantic chunking improves conceptual grouping, and agentic chunking supports advanced AI workflows.

Most production systems combine multiple methods and refine based on real query performance. Thoughtful chunking is one of the simplest ways to improve retrieval accuracy, reduce hallucinations, and build RAG pipelines that actually work.

Frequently Asked Questions

What is chunking in RAG?

Chunking in RAG is the process of splitting large documents into smaller segments before embedding and indexing them for retrieval. These chunks become the units the system searches when answering user queries.

Why is chunking important in RAG systems?

Chunking ensures the retriever returns focused, relevant context instead of large blocks of text. Well-designed chunking improves retrieval accuracy, reduces hallucinations, and makes better use of the model's context window.

What is the best chunking strategy for RAG?

There is no single best strategy. Fixed-size, recursive, semantic, and sentence-based chunking all perform better for different types of documents and query patterns. The right choice depends on data structure and the type of questions users ask.

What is a good chunk size for RAG?

Most RAG systems perform well with chunks between 200 and 500 tokens, with an overlap of 10 to 20 percent to preserve context between chunks.

Can poor chunking reduce RAG performance?

Yes. Poor chunking splits important information across multiple segments or retrieves incomplete context, which leads to lower retrieval accuracy, higher token usage, and increased hallucination in generated answers.