Building ClauseLens: what 90.7% retrieval actually took

ClauseLens is a contract-intelligence RAG pipeline: you point it at a repository of agreements, ask "does the MSA auto-renew?", and get an answer with clause-level citations that are validated in code before the answer is allowed out. The reference implementation runs entirely locally — no cloud account, no API key required — and scores 90.7% hit-rate@8 on a 43-question golden set. You can watch recorded runs of it on the systems page.

This post is the honest version of how it got there, including the three bugs the benchmark caught that I would not have found by eyeballing outputs.

Architecture in one paragraph

Documents go through a structural chunker that respects section boundaries instead of splitting every N characters. Retrieval is hybrid: BM25 and dense vectors run in parallel, and their rankings merge via Reciprocal Rank Fusion. Queries get rewritten first (legal synonyms, clause aliases). Synthesis is deliberately boring — Claude Haiku when a key is present, an extractive fallback when not — because the interesting guarantee lives after synthesis: a citation validator that verifies every cited span actually exists in a retrieved chunk, and rejects the answer if it doesn't. Trust in RAG is a code problem, not a prompt problem.

Bug #1: the chunker was eating preambles

Early hit-rate was stuck in the low 80s. The golden set showed misses clustering on questions like "when was this agreement entered into?" — anything answered by a document's opening paragraph. The structural chunker keyed on numbered section headings, and text before the first heading — the preamble with the parties and effective date — was silently dropped. Fix: emit the preamble as its own chunk. Two lines, several points of hit-rate.

Bug #2: persistence that only worked in one process

Everything passed in-process; a fresh process returned garbage. The vector store persisted lazily, and the TF-IDF+SVD embedder refit on the query-time corpus — so vectors written by ingestion and vectors computed at query time lived in different spaces. Fix: persist eagerly at ingestion and load the fitted transformer, never refit. The lesson generalizes: test RAG persistence across process boundaries, because that's how production actually runs it.

Bug #3: SVD dimension drift

Rebuild the index on a bigger corpus and SVD happily picks a different effective dimensionality — then query vectors no longer match stored ones. The fix pins dimensions and handles drift on rebuild explicitly. If you swap in a hosted embedding model this class of bug disappears, which is exactly why the blueprint specifies one for production.

What the benchmark bought

None of these three would have been caught by demoing "a few queries that look right." A 43-question golden set with must_contain assertions, run in CI, is what turns "seems to work" into a number that moves when you break something. The whole benchmark runs in seconds because retrieval is 1.5ms mean — there is no excuse not to have one.

corpus:     3 documents → 24 chunks (40ms ingest)
retrieval:  hit-rate@8 = 0.907 (39/43) · mean 1.5ms · p95 1.65ms
synthesis:  citation-validated, extractive fallback without a key