How I build LLM systems.

This is the method behind the LLM systems I build in my own projects. Their case studies show it in practice; here is the short version.

Evals before features

A retrieval pipeline without a golden dataset is a demo, not a system. I write the eval spec (what will be measured, on what data, with what floor) before the feature it measures, and I wire the eval into CI so quality regressions fail pull requests. This site practices what it preaches: its own search widget ships with a committed golden query set and published recall/MRR numbers.

Numbers only when measured

Design targets are labeled as targets. Metrics tables render an explicit “pending” state until a real measurement exists, with the measurement config alongside the number once it does. No benchmark theater.

Safety as architecture, not garnish

Human-in-the-loop gates on write actions with server-side re-validation, typed tool contracts, prompt-injection assumptions baked into data flow design, per-IP rate limits, and provider fallback chains. If a failure mode isn't in the design doc, it's in production.

Cost and latency as design inputs

Free-tier caps, token budgets, and p95 targets are architectural constraints from day one; they decide chunk sizes, fusion strategies, and fallback chains, not the other way around.