Retrieval quality
This site has had hybrid retrieval since P16 — a vector index, a lexical index, and reciprocal rank fusion over both. What it never had was any way to tell whether it works. The table below runs the answerable half of 12 labelled questions against the live index when this page loads, and scores what comes back.
Recall asks whether the right page made the window at all — the ceiling on everything downstream, since no answer can cite a passage retrieval never returned. MRR asks how high, because a prompt with a token budget does not treat rank 1 and rank 5 alike. nDCG weights position and grade together. A change that raises recall while lowering MRR has moved the right answer into the window and pushed it down inside it; one number would have hidden that.
| Query | First hit | Recall | nDCG | Returned |
|---|---|---|---|---|
| How does the distributed rate limiter work? | #1 | 50% | 0.83 | 5 |
| token bucket algorithm redis | #1 | 50% | 0.83 | 5 |
| What did you do to make LLM inference faster? | #2 | 100% | 0.63 | 5 |
| agent orchestration tool calling | #1 | 100% | 1.00 | 5 |
| Where has Shivam worked? | #2 | 67% | 0.52 | 5 |
| What technologies does he know? | #1 | 67% | 0.83 | 5 |
| How do I get in touch? | #1 | 100% | 0.88 | 5 |
| Can I hire him for consulting? | #1 | 33% | 0.73 | 5 |
| how does search on this site work | #2 | 50% | 0.52 | 5 |
| WebAssembly and WebGPU benchmarks | #1 | 100% | 1.00 | 5 |
| quantum blockchain synergy | not found | 100% | 1.00 | 5 |
| distrbuted rate limitter | #3 | 100% | 0.50 | 5 |
12 queries is a small set, and that is the honest limitation. Enough to catch a retriever that has broken outright; nowhere near enough to separate two good rankings, where the confidence interval on twelve queries is wider than any difference worth shipping. Growing it means writing judgements by hand, which is the real cost of evaluation and the reason so much retrieval ships unmeasured. Judgements are keyed on URL rather than chunk id, so editing a paragraph does not invalidate the set.
Recall came back at 42% with seven misses, which read as a broken retriever. It was not. Six of the seven judged pages the corpus does not index at all — it covers projects, blog posts, skills, service offerings, experience, achievements and certifications, and nothing else. No ranking could have returned /contact or /compute, so those queries were measuring content coverage while reporting a retrieval number. A seventh “miss” was the deliberately-unanswerable query, whose reciprocal rank is zero by definition.
The judgements were not edited to match. “How do I get in touch” is a reasonable thing to ask a portfolio, and deleting the query because the index cannot answer it would have raised the score by hiding the gap. The two are reported separately instead, because they have completely different fixes: one is a retrieval problem, the other is four lines in buildCorpus.
How does the distributed rate limiter work?Direct lookup by topic. If this fails, retrieval is broken outright.token bucket algorithm redisKeyword-shaped rather than a question — the lexical half should carry this one.What did you do to make LLM inference faster?Natural language with no shared vocabulary beyond 'inference'.agent orchestration tool callingThree terms, all of which appear on exactly one page.Where has Shivam worked?Several pages are legitimately relevant — tests that grading is used at all.What technologies does he know?Broad question with a clear best answer and several acceptable ones.How do I get in touch?Two pages equally correct. Either at rank 1 should score the same.Can I hire him for consulting?Commercial intent phrased without any word on the target page's title.how does search on this site workSelf-referential. The corpus describes its own retrieval, so this should be easy — and it is a good canary for the index being stale.WebAssembly and WebGPU benchmarksTerms that appear on one lab page and nowhere else.quantum blockchain synergyDeliberately unanswerable. An empty judgement set means recall and nDCG are 1 by definition; what this actually checks is that the retriever does not crash and that /ask degrades honestly rather than confidently returning the nearest thing it has.distrbuted rate limitterTwo misspellings. Lexical matching fails here by construction, so this measures whether the vector half is contributing anything at all.The comparison inverts, which is why it is a slider rather than a table. At one question a day the download dominates completely and per-token pricing is noise; at a hundred thousand the download amortises to nothing and per-token pricing is the entire decision.
463 input tokens per question at this setting, plus 220 for the answer.
| Option | Runs | Download | Per question | Per year | Fits |
|---|---|---|---|---|---|
| Extractive (what /ask does) | In the browser | — | $0 | $0 | 989 / ∞ |
| Llama 3.2 1B, 4-bit on device | In the browser | 0.9 GB | $0 | $0 | 989 / 4,096 |
| Llama 3.2 3B, 4-bit on device | In the browser | 2.1 GB | $0 | $0 | 989 / 4,096 |
| A small hosted model | Hosted | — | $0.00020 | $4 | 989 / 128,000 |
The embedding is a hashed bag of terms with IDF weighting, and IDF is fitted to the corpus. So nothing here has weights that decay — what goes stale is the relationship between the index and the corpus it was built from. Every project added moves the term distribution, and the IDF baked in at build time drifts from the one the live corpus would produce.
Retrieval keeps working, gradually worse, weighting terms by a document frequency that stopped being true months ago. Nothing errors and no test fails. The only way to notice is to measure the distance, which is what the PSI implementation in lib/mlops/drift.ts is for.
Under 0.1 stable, 0.1–0.25 moderate, above 0.25 significant — thresholds that come from credit-risk modelling in the 1990s. They are used because a shared convention everyone reads the same way beats a bespoke threshold nobody can calibrate, not because they are derived from anything.
PSI is sensitive to a large relative change in a rare bucket; cosine distance is dominated by the common ones and barely moves. When they disagree, the tail moved — which is usually new content rather than a problem. Reporting only one would make that indistinguishable from a real shift.
Quantization stores weights at 4 bits instead of 16, which is what makes a 1B-parameter model an 880MB download instead of 2.5GB and lets it run in a browser at all. It also measurably degrades the model — most visibly on exactly the thing a portfolio assistant needs, which is not paraphrasing fluently but declining to invent a fact.
880MB before the first token, on a connection you do not control, for a visitor who asked one question. On a phone that is not a loading state, it is an exit. The 3B model is noticeably better and 2.1GB, which makes it worse.
It selects sentences from retrieved passages and scores them against the query. It cannot paraphrase and it cannot be wrong about a fact it did not retrieve — and on a page whose purpose is telling the truth about someone’s work, the second property is worth more than the first.
With no generation step there is nothing downstream to paper over a bad ranking. That is the argument for everything above: an extractive answerer is exactly as good as its retrieval, so retrieval is the only thing worth measuring.