Chunk by meaning while preserving document structure

Retrieval-augmented generation conditions an answer on retrieved material rather than relying solely on model parameters. The original RAG paper combines retrieval and generation in a defined model setting. In a software application, the same broad pattern creates an engineering pipeline: ingest documents, construct retrievable units, find candidates, select evidence, and produce an answer whose claims can be checked against that evidence.

Chunking determines the units available to later stages. If a policy exception is separated from the rule it qualifies, a retriever can return a misleadingly complete-looking passage. If every document is one huge chunk, relevant details may be diluted and context budgets exhausted. The aim is to preserve answer-bearing relationships while making candidate selection efficient. There is no universal chunk size that solves every document type.

Parse headings, paragraphs, lists, tables, code blocks, and document metadata before applying a generic window. A table row often needs its column headers to be interpretable. A code example may need its language and function context. A subsection can inherit a document title and version without copying the entire document into every chunk. Store parent links so retrieval can expand context when a small hit lacks necessary qualifiers.

Suppose a policy has a 90-word rule followed by a 40-word exception. A fixed 100-word boundary splits the exception midway. A structure-aware chunk can preserve the 130-word rule-and-exception pair if its budget allows. If it does not, preserve the relation through parent and sibling metadata so the answer builder can retrieve the exception alongside the rule. Splitting text is easy; preserving its semantics requires explicit decisions.

A semantic chunker can compare representations of adjacent sentences or sentence windows and place a boundary when topic similarity drops. Sentence embeddings provide one possible representation. The mechanism requires a threshold, minimum size, maximum size, and fallback for long passages. Similarity measures topic relatedness imperfectly; a crucial exception can use different vocabulary while belonging with the preceding rule.

In the code, four hand-supplied two-dimensional vectors stand in for sentence representations. Neighbor similarities are high, low, then high. A threshold of 0.5 creates two groups. These vectors are synthetic and do not perform natural-language understanding. The example isolates boundary logic so a learner can inspect what a real embedding model would influence. Do not call a word-count splitter semantic merely because it appears in a RAG application.

A 1,200-token document split into 300-token windows with a 50-token overlap advances by 250 tokens each step. This produces overlapping evidence and increases indexed token volume. Overlap can preserve context near boundaries, but duplicate passages may occupy several top retrieval positions. Deduplicate or diversify candidate selection so one repeated paragraph does not crowd out a necessary second source.

Use stable chunk IDs tied to document identity, revision, and span. A citation should resolve to the source that was actually indexed, not whatever happens to live at a mutable URL later. Keep text offsets or section anchors and a content hash. If a parser changes whitespace or table layout, preserve enough mapping to find the original evidence. Citation stability is an ingestion responsibility as much as a generation feature.

Compare chunking policies on answer-bearing spans, retrieval recall, context cost, and final grounded correctness. Include questions that require an exception, a table header, a cross-reference, and two adjacent paragraphs. A visually tidy set of equal-sized chunks can still lose the evidence needed for the question. Conversely, a semantic splitter may add computation without improving retrieval on a strongly structured corpus.

Start with a structural baseline, then add semantic boundaries where failures justify them. Record parser and chunker versions so changes can be reproduced. When a question fails, inspect whether the evidence was ingested, whether it survived chunking, whether it was retrieved, and whether it was used correctly. This stage-by-stage diagnosis prevents repeated prompt tuning from trying to repair information that never reached the model.

Hand-authored vectors simulate two topic clusters. Adjacent cosine similarity triggers boundaries. The example does not create embeddings, parse real documents, enforce a token limit, or establish that semantic splitting improves a corpus.