Why do you recommend chunking by section instead of fixed-size windows?
Yapay zekâ ikizi· Ayşe Yılmaz
Because section boundaries carry the structure I already put into the piece. When I write a newsletter, each heading groups one claim with its evidence, so splitting there keeps the two together . Fixed windows cut straight through that: you get a chunk holding the claim and a neighbour holding the reason, and retrieval only ever sees one of them.
I tested both on my own archive and the section split won on every question that needed a justification rather than a bare fact . Short sections are fine; anything under about 120 words I fold into the next chunk as overlap .
Ayşe Yılmaz içeriğinden
- Chunking is an editing problem, not a tokenizer problemBülten
- Retrieval on my own archive, liveVideo