Chunking notes for retrieval: what actually mattered
Retrieval only works if the thing you retrieve is a sensible unit. My first version split every note into 512-token chunks with 64 tokens of overlap, because that's what every tutorial does. Results were mediocre in a very specific way: the right note came back, but the chunk started in the middle of one thought and ended in the middle of the next.
What I compared
I built a small test set (more on that in a later post): 60 questions I've genuinely asked myself, each with the note that answers it. Then I measured how often the correct note showed up in the top five results.
| Strategy | Top-5 hit rate |
|---|---|
| Fixed 512 tokens, 64 overlap | 63% |
| Fixed 256 tokens, 32 overlap | 68% |
| Sentence windows (5 sentences) | 70% |
| Split on Markdown headings | 83% |
| Headings + note title prepended | 88% |
Why headings work
Because I already chunked my notes when I wrote them. A heading is me saying “this is one topic”. Fixed-size splitting throws that information away and then tries to recover it with overlap.
Prepending the note title to every chunk helped with short sections like “## Decision”, which mean nothing without knowing which note they belong to.
def chunks(note):
title = note.title
for section in split_on_headings(note.body, max_level=3):
text = f"{title} > {section.heading}\n\n{section.text}"
if len(text) > 2000:
yield from split_paragraphs(text, limit=2000)
else:
yield textSections longer than about 2000 characters still get split on paragraph boundaries. That affects maybe 3% of my notes, mostly long logs.