TruthBase: verifiable RAG with citation-to-highlight receipts

The problem

Every RAG demo can produce an answer with a citation. Almost none can prove the citation. The usual pattern (“[1] report.pdf, p. 12”) asks the reader to open the document and go hunting, so in practice nobody verifies anything and the citation is decoration. My thesis for TruthBase is that trust needs receipts: clicking a citation should scroll the PDF viewer to the exact page and paint a highlight over the exact passage that supports the sentence you just read.

That one interaction is the whole project. It forces most of the interesting engineering: positions have to be captured at ingestion time (retrofitting them means re-parsing the entire corpus), stored in a scale-independent form, and reconstructed accurately across zoom levels. The retrieval pipeline behind it also has to be good enough that the highlighted passage genuinely supports the answer.

Architecture

TruthBase runs live at truthbase.elbaymalik.com: one Nuxt 4 app whose secret-bearing and compute-heavy work (embedding, retrieval, generation) runs in Nitro server routes, deployed as a single Node 24 Vercel Function in dub1, next to the Supabase database in AWS eu-west-1. Nothing secret reaches the client. The demo corpus is four public PDFs (NIST CSF 2.0, NIST AI RMF 1.0, and the RAPTOR and Self-RAG papers), ingested from an operator CLI rather than the browser.

Ingestion extracts each page’s text items with unpdf and PDF.js, then replays the page’s operator list with PDF.js’s own text-advance math to get every glyph’s x-extent. Chunking is page-bounded sentence packing: up to 800 characters with 15% overlap, the smallest size that keeps all 60 golden evidence spans inside one chunk, because the highlight a reader sees is the chunk. Every chunk stores one rect per rendered line in PDF space (chunks.highlight jsonb) next to a generated tsvector and a 768-dimension gemini-embedding-001 vector, L2-normalized by hand because Google only normalizes the full 3072-dim output. The viewer draws the stored rects as they are, converted with PDF.js’s viewport.convertToViewportPoint on every zoom. It never re-derives them from the browser text layer, which lays text out in a substitute font; if the rendered text no longer matches the chunk, it finds the passage by text and shows an “approximate” badge.

A question first passes consume_quotas, one Postgres call that checks the client’s per-minute and daily counters and the demo’s global daily budget, all or nothing, and answers 429 before any model call. The question is embedded while a self-gated rewriter on Groq’s gpt-oss-20b decides whether it is ambiguous enough to split into up to three queries. Retrieval is hybrid and stays inside Postgres: hybrid_search, one SQL RPC, fuses a full-text arm and an exact cosine arm with weighted Reciprocal Rank Fusion. A listwise LLM reranker on gpt-oss-20b orders the fused top 15; its first four picks lead and the fused order fills the rest, up to eight sources. Rewriting and reranking are optional at run time: an error or a timeout falls back to the fused order. The server streams the sources with their stored rects first, then the answer from Groq’s gpt-oss-120b (Gemini Flash-Lite if Groq fails) through the Vercel AI SDK v7, which cites [n] markers or replies NO_ANSWER. Every chat request is traced to Langfuse. The decisions behind it are logged below, each with the alternative it beat.

Evaluation design

The ground truth is a golden set of 50 questions over the four PDFs: 45 answerable, each with its exact supporting quote and page number, and 5 unanswerable (TruthBase docs/golden/golden.json). At n = 45, one question moves a metric by 2.2 points. The same records drive the Playwright highlight tests, because “did the eval cite the right chunk?” and “did the highlight land on the right rect?” are the same question asked at two layers.

Evaluation runs in two layers. A deterministic retrieval gate is set up in CI for every pull request (its last CI run was on 2026-10-21): the real hybrid_search on a committed corpus fixture, failing on any drop from TruthBase docs/evals/retrieval-baseline.json. The model-dependent evals spend free quota, so they run on demand: one script per retrieval technique, and a promptfoo LLM eval with a separate judge model, scheduled weekly.

The plan named its own falsifier: if hybrid search failed to beat dense-only on golden recall, chunking would be revisited before any further retrieval tricks. At Phase 4 the two tied at 73% recall@8 (TruthBase docs/evals/phase-4-baseline.json). An audit found that the keyword arm ANDed every term and scored 27% on its own. Switching it to OR over the question’s own lexemes, with weights and k chosen from a 72-configuration grid under cross-validation, took hybrid recall@8 to 87% against 76% for dense-only, on 45 answerable questions over 564 chunks. At the shipped 15 candidates, the reranker lifted MRR from 0.515 to 0.735 and nDCG@5 from 0.638 to 0.811, and recall@5 from 76% to 84% (TruthBase docs/evals/phase-7-groq-20b-15.json). Query rewriting with the shipped gpt-oss-20b raised recall@5 on the four ambiguous questions from 75% to 100% and overall recall@5 from 76% to 80%, and left recall@8 at 87% (TruthBase docs/evals/phase-8-groq-20b-gated.json).

The latest answer eval runs the shipped pipeline end to end on all 50 questions: rewriting and reranking on gpt-oss-20b, answers on gpt-oss-120b (TruthBase docs/evals/phase-8-groq.json). It refused all 5 unanswerable questions, answered 93% (42 of 45) of the answerable ones and none without a citation. At least one citation was correct for 86% of answered questions, 80% of answerable ones, against 69% in the Phase 5 run. Citation precision was 56%: a lower bound, since a citation to a correct passage that is not the golden evidence counts as wrong. False refusals fell from 7 in the Phase 5 run to 3, still above the target of at most 1: two questions whose evidence retrieval misses, and one table that extraction flattened into running text. The first full promptfoo run then judged faithfulness at 95.2% and citation precision at 92.5% (TruthBase docs/evals/phase-9-llm.json, 2026-10-23), using a qwen model as judge against targets of 0.90. A hand check of 5 answers found the judge lenient in how it splits answers into claims and assigns their citations. Corrected for those answers, the pooled figures are about 92–94% and 88–90%. The run still fails its gate on 4 false refusals, so the repo records no LLM baseline yet.

The highlight itself is checked at three levels. pnpm check:viewer drives the real viewer in headless Chromium over all 60 golden spans at zoom 50, 100 and 200%: since 2026-10-25 all 180 checks pass against the hosted corpus, with a placement error of at most 0.03 px. Vitest pins span edges within ±0.05 pt of MuPDF, an independent PDF engine. Playwright repeats the geometry and ink checks in CI, last run on 2026-10-21. Before that, two cases caused four misses: a big Σ operator that hung below the em box the vertical model covers, and a vector-drawn frame that is page graphics, not text. The operator is now boxed from its glyph outlines, and the ink check counts text ink only.

Cost and latency as design inputs

The original budget was a hard ceiling of 5 EUR/month. The live demo costs 0 EUR/month: every service runs on a free tier, and the free-tier limits were treated as inputs to the design, not surprises (TruthBase README.md, Cost and free-tier limits). Groq allows each model 8K tokens a minute, 1K requests and 200K tokens a day, so answers run on gpt-oss-120b and rewriting and reranking on gpt-oss-20b, each against its own quota. Postgres enforces 10 chats a minute and 30 a day per client and a budget of 80 chats a day for the whole demo; past about 60 enhanced chats a day, gpt-oss-20b runs out and chat answers from the fused order. Supabase’s free 500 MB holds the 564 chunks in about 2 MB of vectors, and a keep-alive workflow is scheduled twice a week against its 7-day inactivity pause; its last successful run was on 2026-10-19. Vercel Hobby allows 4 Active CPU-hours a month, which holds because waiting on a model does not count as active CPU.

On a local production build, Lighthouse scores the landing page 98 on mobile and 100 on desktop, with 100 for accessibility, a mobile LCP of 2.40 s and a CLS of 0 (Lighthouse 13.5.0, median of 3 runs, 2026-10-20). A spot check on the live deployment on 2026-10-21 measured 1.7 s for a search and 1.5 s to the first token of a chat answer: one check, not a percentile. Dev-server traces put time to first token at 1.35–1.6 s without rewrites and 2.2–2.9 s when the question is rewritten (TruthBase docs/TECH_NOTES.md §15), so the p50 under real traffic depends on how many questions get rewritten, and it stays pending until Langfuse has that traffic. Token spend per query is not published either: Langfuse records tokens per model call, and the rewriter alone costs about 650 tokens per question, but no whole-query figure has been taken yet.

What’s built, what’s next

TruthBase has been live at truthbase.elbaymalik.com since 2026-10-21, with no login: ask a question, click a citation, and the PDF opens at the right page with the passage highlighted. Roadmap phases 0 to 15 are built, from the golden set, ingestion and the highlight viewer through hybrid search, reranking, query rewriting, tracing, abuse protection, accessibility, the Vitest and Playwright suites and the deployment, except one: the eval harness (phase 9) stays unticked until its first full promptfoo run lands. Reranking and query rewriting were re-measured at their shipped settings on 2026-10-21, and the whole pipeline end to end.

The known gaps are written down in the repo. Six golden questions are never retrieved in the top 15 (three multi-page or multi-hop, three numeric or factoid details), and query rewriting does not reach them. Extraction flattens tables into running text, which causes refusals even when the evidence is retrieved. False refusals are down to 3 in the end-to-end answer eval, still above the target of 1. Prompt injection through document text is reduced but not prevented. The next phase, 11b, is bring-your-own uploads behind Supabase Auth, with per-user quotas and a prompt-injection review before uploads open; it is deferred.