The replay-demo pattern: show your RAG working without an API key

My RAG demo costs $0 to run and can't be abused — not because I locked it down, but because it doesn't call a model at all. The systems page for ClauseLens replays real recorded pipeline runs: the same queries that live in the benchmark golden set, the same retrieved chunks, the same latency numbers from benchmark_results.json. The difference between that and a fake demo is worth explaining, because they look identical from the outside.

The two bad alternatives

If you want to demo a RAG pipeline on a public website, you have two obvious options. You can wire up a live model and call it on every visitor's question, which means your API bill scales with traffic and your input field is an open prompt-injection surface. Or you can fake it: hardcode a few canned responses that look like real output but aren't actually produced by your pipeline. The first option is expensive and unsafe. The second is dishonest.

Neither felt right for ClauseLens. The pipeline exists as a real, runnable open-source implementation, and the whole point is that you can inspect what it does. A fake demo undermines that. But a live demo that burns API quota on every curious visitor means the demo itself becomes a liability the moment any real traffic shows up.

What the replay pattern is

The replay pattern sits between those two options. Before the demo page goes live, you run a representative set of questions through your actual pipeline and capture the full output — the retrieved chunks, the latencies, the citations, the final answer. You serialize that as JSON. Then the demo page is just JavaScript that reads that JSON and plays it back, animated to look like a live run.

What the systems page shows is exactly this. The honesty box on that page says it plainly: "These are recorded runs of the real clauselens-rag pipeline against its sample corpus — real queries from the benchmark golden set, real retrieved chunks, real latencies from benchmark_results.json. Nothing here calls a model at page-load; that's the point." The latency numbers are real because they were measured when the benchmark was actually run, not manufactured for the demo. The retrieved chunks are real because they came from the pipeline's retrieval step. The citations are real because the citation validator cleared them.

Why it's still honest

A replay demo is honest if and only if the data in it came from your actual system. The runs backing the ClauseLens demo came from the same benchmark that produced the 90.7% hit-rate figure — 43 questions against the bundled sample contracts, run with no model in the loop (the benchmark uses an extractive fallback so the retrieval numbers are reproducible regardless of what API key you have). If I had cherry-picked questions that happened to retrieve cleanly or trimmed latency numbers after the fact, the demo would be dishonest. I didn't, and you can verify that by running the benchmark yourself: make bench writes a fresh benchmark_results.json and you can compare it to what the demo shows.

That verifiability is what separates replay from fabrication. Anyone who clones the repo and runs it on the same sample documents should get numbers within rounding error of what's on the page.

The pipeline runs without an API key anyway

There's a second layer to this for ClauseLens specifically: the pipeline itself doesn't require an API key. The synthesis step has two paths — Claude Haiku when ANTHROPIC_API_KEY is set, and an extractive fallback that returns the top retrieved excerpts verbatim with citations when it isn't. The citation validator runs identically in both cases, so the retrieval guarantee is the same. This means anyone who runs the demo locally isn't dependent on an API account to see the pipeline work end-to-end.

That design choice wasn't primarily about the demo — it was about making the reference implementation genuinely usable as a starting point. But it's a useful consequence: the "no API key" demo isn't a crippled version of something more powerful. Retrieval and citation validation are the hard parts, and they run completely.

The zero-abuse-surface benefit

An underrated advantage of the replay pattern is that there's nothing to abuse. A live inference endpoint takes user input, passes it to a model, and returns output — which means it can be used to extract system prompts, run jailbreak attempts, or just generate large volumes of text on your API bill. A replay demo takes user input, looks it up in a JSON map, and returns the pre-recorded result. There's no inference path to exploit.

This matters especially early in a project, when you want people to be able to explore what you've built without you having to monitor it. The systems page has been up since ClauseLens launched. It has had zero API spend from demo traffic and no abuse incidents, because there is no inference surface to abuse.

When the pattern doesn't work

The replay pattern assumes your demo questions are fixed in advance, which is true for a benchmark-backed demo but wouldn't work for a product that lets users ask arbitrary questions. If the value proposition of your system is that it handles any question — not just a curated set — then a replay demo can't actually show that. For ClauseLens the pipeline is demonstrably general (the golden set covers paraphrases, cross-document questions, and edge cases, not just easy lookups), but the demo page only shows the questions that are in the set. That's a real limitation and I wouldn't pretend otherwise.

For showcasing that a retrieval system works and is reproducible, though, it's the most honest option I've found.