v1 · naive and rerank measured September 2026 · the other five are still predictions
Retrieval-augmented generation is not one technique any more. It is seven designs that solve seven different problems, and most teams pick one by accident. This page runs all seven on the same small, real knowledge base and the same nine questions, so the only thing that changes between columns is the architecture.
The knowledge base is my own bookshelf: a Goodreads export, my Kindle highlights, a list of who recommended what, and a few photographed pages. I chose it because I can tell a right answer from a plausible one in a second, which is the whole job when you evaluate retrieval, and because "it invented a book I never read" is the most legible failure a RAG system can have. Two of the seven have now been run against the real shelf — eighty answers in September 2026, and the traces below are what they actually did. The other five are still predictions, and every cell says which it is. How it's built says what gets measured and when. The service is live: ask the ledger ↓.
Teams upgrade the embedding model, switch to a stronger LLM, add more chunks to the prompt, and still get shallow or wrong answers. The reason is usually upstream of the model: the retrieval design cannot reach the information the question needs. Similarity search has no idea what "2024" or "under 300 pages" means. A text index cannot look at a photograph. A single retriever cannot see that two books argue with each other, because that fact lives in neither book. And no retrieval at all can compose four constraints into a reading list.
The real skill isn't building RAG. It's choosing the right pattern for the problem, and being able to show why.
Comparing patterns on different demos measures the demos. To compare the patterns you need to hold everything else still: same corpus, same questions, same generation model, and answers you can grade. That is what this page does.
Three files and a handful of photos. The Goodreads export is the ledger: every book, when I read it, what I rated it, how long it is, what everyone else rated it, and the review I wrote. The Kindle clippings are the quotes: one short highlight per block, hundreds of them. A hand-written list says who recommended what. Nothing is fabricated, private notes are never ingested, and the export is curated before it goes anywhere near a public page.
| source | what it holds | what it contributes |
|---|---|---|
| Goodreads export | one row per book: title, author, ISBN, pages, date read, my rating, the average rating, shelves, my review | the ledger. Queried, never embedded. |
| Kindle highlights | one short quotation per block, with book, location and date | the semantic index, and the misattribution trap: a highlight in one book quoting another author |
| Recommendations | title, who (initials), where, when | the people edges in the graph |
| Photos | covers, a few photographed pages, one shelf | answers with no text form |
| Extracted graph | who wrote what, who recommended what, which books mention or argue with which, themes | built once by a model with a fixed schema, published as JSON so a missed edge is visible |
Six were planned, one per failure mode plus one that everything should get right. Three more came from asking my own shelf what I actually wanted to know: my favourites in a theme, my top themes, and my top themes by year. Each has a reference answer written before any pipeline runs, computed from the ledger where it can be. Q1, Q2 and Q3 name real books: the Kindle file never arrived, but five of my Notion pages turned out to hold pasted clippings, so the control question has a real target. The results are in the explorer below: measured for naive and rerank, still predicted for the other five.
Each trace shows what gets retrieved, in what order, what the model is given, and the answer it is likely to produce, graded against the reference. The "why" line under each answer is the point of the page.
| step | what the pipeline is expected to do |
|---|
Two rows are measured, five are not. Naive RAG and retrieve-and-rerank were run against the real shelf in September 2026 — eight questions, five runs each, eighty answers; every trace, cost and refusal is in the repo, including the runs that disagree with what this page predicted. The other five patterns, and Q5 for every pattern, are still predictions written from how the pipeline is designed: each cell says which it is, and each is replaced by a real trace as the version that measures it ships. The schedule is in How it's built.
The built part of the system, running as a service, through the same answer() interface the eval calls. Three routes, and the panel says which one answered. The ledger runs its query against the shelf now, live, for nothing. Naive and rerank replay their measured run: a model call on a public page would spend money on every click and could disagree with the trace above it, so what you get is the measurement, with its date, its cost and the chunks it actually retrieved. The patterns that are still predictions are not offered here at all.
Pick a question. Nothing here is a prediction: the service runs the query and returns the rows.
Three things to read off the grid. Every pattern gets the easy question right, so the easy question tells you nothing about which pattern to use. The Q2 column splits the table cleanly into patterns with a tool and patterns without one, and no amount of embedding quality moves a pattern across that line — that one is now measured, not argued: naive and rerank both refuse Q2, and the ledger route answers it in eleven milliseconds with no model call at all. And reranking is still the first upgrade I would make, but for a narrower reason than I wrote here in v0; see below.
The prediction I most expected to be wrong held. I expected naive to misattribute Q1, because Kindle highlights are short and the nearest chunk is sometimes a different book quoting the same author. It did not: the right highlight led by 0.13, in all five runs. Two predictions broke instead, and both broke in the same direction.
Nothing invented a book. Not once in eighty answers. This page predicted that naive on Q2 would produce "a confident list of five or six titles with dates", at least one of them not in the library. What it does is say plainly that the context holds no ratings, dates or page counts, and stop. I removed the sentence in the shared prompt that named those missing fields, in case the prompt was doing the refusing, and re-ran it: it still refuses. The failure this page was built to dramatise is real — retrieval finds nothing useful for a ledger question — but the invention I predicted on top of it did not happen with this model. That is worth more than being right.
Read the two measured rows against the five predicted ones carefully. They are not graded the same way, and I would rather say so than let the grid flatter them. A measured refusal scores half a mark here, because refusing honestly is not the answer but is not a lie either; the five predicted rows were written assuming the pipeline would invent, and invention scores nothing. Naive and rerank therefore sit higher on the shelf than they might once the other five are actually run — and they now sit at exactly the same height as each other, which is itself the finding: on these nine questions, at v1, one extra call bought margin and better chunks, and not one extra correct answer.
Reranking cannot fix what was never retrieved. I predicted Q3 was the cell where the cross-encoder earns its call: it would promote my review over my own admiring highlights. In fact The Fellowship of the Ring never reaches the context at all, for either pattern, in any run — only passage text is embedded, so naming a book in the question does not fetch that book. Reranking fixes the wrong four out of thirty. It has nothing to say about a book that was not in the thirty. What it did earn, measured: it turned Q1's 0.13 margin into 0.57 for 361 ms, and on Q4 it surfaced the one note where I wrote down a quarrel between two books — an argument found without a graph.
Don't start complex. Start naive, add reranking, give the model a tool for the numbers, and evolve only when a fixed question set shows you the failure the next pattern fixes. A design space is only useful if you can say which point in it you are standing on, and why.