Case study

Synapse — an agentic RAG document assistant

Upload a document, ask it questions, and get answers built only from its pages — with a citation on every claim, a critic that checks the draft before you see it, and a trace that shows the whole pipeline's work.

Python · FastAPI · FAISS + BM25 · Ollama / Groq · 31 tests · live

The problem

Ask an LLM about a document it hasn't read and it will answer anyway — fluently, confidently, and sometimes from thin air. Retrieval-augmented generation is the standard fix: find the relevant passages first, then tell the model to answer from those and nothing else. But most RAG demos treat retrieval as a solved step — one vector search, top few results, straight into the prompt — and that's exactly where they quietly fail. Vector search whiffs on exact terms like invoice numbers and names; keyword search misses paraphrases; and whatever slips through, the model happily narrates over the gaps.

Synapse was my attempt to take the retrieval problem seriously and to make every stage of the answer auditable. Retrieval quality, not the model, is where RAG lives or dies — which is why three of the seven backend modules are retrieval stages.

How a question moves through it

Ingestion first: loaders.py reads PDFs page-by-page (keeping page numbers for citations), chunker.py slices the text into ~1600-character chunks along paragraph boundaries with a 200-character overlap so facts straddling a boundary survive intact, and each chunk lands in three places — its text in sqlite, its embedding in a FAISS vector index, and its words in a BM25 keyword index.

When a question arrives at /api/chat, the orchestrator (orchestrator.py, about 140 lines) runs four LLM roles in sequence, streaming progress to the browser as server-sent events:

• Planner — breaks the question into up to 3 sub-questions, so a compound question gets researched in parts.
• Retrieval — per sub-question, BM25 and FAISS search in parallel, get fused, get reranked (next section).
• Synthesizer — drafts an answer using only the retrieved excerpts, with a [1]-style citation on each claim. The prompt forbids outside knowledge outright.
• Critic — a separate call at temperature 0 whose only job is to re-read the draft against the sources and list unsupported claims. A final pass fixes them, and that corrected answer is what streams to the user.

Every stage is visible in the UI: click "Show reasoning trace" and you get the planner's sub-questions, the exact chunks retrieved for each, the pre-critique draft, and the critic's actual review text. It is not one prompt — it's a plan–retrieve–draft–critique pipeline, and you can watch it think.

The retrieval funnel

Each stage exists because the one before it has a known blind spot. Embeddings catch paraphrases ("firing" matches "termination") but fumble exact tokens; BM25 nails exact terms but misses rewordings — so run both. Their scores live on incomparable scales (a cosine of 0.83 versus a BM25 of 14.2 means nothing), so RRF ignores raw scores entirely and scores each chunk by its rank position in each list — a chunk ranked well by both retrievers floats to the top. And rank fusion is still just similarity math: it can rank a lexically-similar but off-topic passage highly. So the last word goes to an LLM that sees all 12 candidates at once and outputs an ordering — one cheap call over short previews, and it can spot what similarity scores can't.

Retrieval quality, not the model, is where RAG lives or dies. If the right passage isn't in the evidence, no prompt can save the answer.

Decisions I'd defend

• A critic pass instead of one big prompt. Hallucination is the failure mode of RAG. A dedicated temperature-0 call that checks each draft claim against the excerpts catches unsupported claims before the user sees them. It costs an extra LLM call per answer; I decided grounded-with-receipts was worth more than fast.
• Parse-and-fallback everywhere LLM output is structured. Models don't reliably emit clean JSON. If the planner's output won't parse, the original query becomes the single sub-question; if the reranker's ordering won't parse, the fused order stands — and any candidate the model forgot to rank gets re-appended so nothing silently disappears. The system degrades to a plain RAG pipeline instead of crashing, and those fallback paths are what most of my orchestrator and reranker tests actually cover.
• Rebuild BM25 from sqlite on every upload. At personal-document scale the rebuild takes milliseconds, and rebuilding from the chunks table means the keyword index can never drift out of sync with the source of truth. Persisting it would add an invalidation problem for zero real gain.
• Pluggable model backends. I develop against Ollama — free, local, no key — but free hosts can't run an Ollama daemon. One LLM-client interface and env-var factories mean the deployed demo swaps to Groq for generation and Hugging Face for embeddings with zero code changes.
• Memory shapes the query, never the evidence. Sessions keep the last 6 turns, but the planner may use history only to resolve references — turning "why is that better?" into a standalone question. Every factual claim still has to come from the retrieved excerpts.

Honest numbers

All 31 tests run instantly with no model installed, because every LLM call is faked — the fallback paths are the thing under test. CI is a single job on Python 3.12 running pytest on every push. And the eval harness deserves a plain sentence: it's real and runnable — an LLM judge scores each answer 1–5 on faithfulness — but the shipped dataset is 2 stub cases, so I don't quote a hit-rate or a faithfulness score, because I haven't earned one yet.

Limitations, stated plainly

• Sessions and rate limits are in-process dicts. A restart wipes conversations, and a multi-instance deploy would break both. Deliberate for a single-instance free-tier demo; the fix is Redis, and the comment in rate_limit.py says exactly that.
• It doesn't scale. FAISS runs a flat brute-force index and BM25 rebuilds fully on every upload — right-sized for personal documents, wrong past tens of thousands of chunks.
• The judge grades its own author. The eval judge is the same model that wrote the answer — a self-grading bias. The harness architecture is what I wanted to learn; a real eval needs a bigger dataset and a different judge.
• It's slow on purpose. Every answer is four-plus sequential LLM calls — plan, rerank, draft, critique, final. I traded latency for verifiability, and the streamed status lines keep the wait legible.
• No auth, open CORS, plaintext sqlite. Fine for a demo, not multi-user production. The rate limiter is the only abuse protection, because the real asset at risk was my free API quota.

What's next

A real eval dataset, so faithfulness and hit-rate become numbers instead of claims — the harness is already built and waiting. Then latency: skip the planner for obviously simple questions and parallelize sub-question retrieval. Then the scaling list in order — Redis for sessions, an HNSW index, an incremental keyword index. Each is a known fix for a limitation I've already written down above.

One note before you click: the live demo runs on Render's free tier, which sleeps when idle — the first load can take a minute to wake — and uploaded documents reset whenever the instance restarts. Bring a small PDF and re-upload if it's gone.

David Jeremie Anand · New Delhi · every number on this page comes from config.py, the test suite, or the prompts in the repo — nothing measured is claimed that wasn't run.