Selected work
Four views of the work: how I structured the system, measured its decisions, tested a judge, and managed the program. Deeper investigations remain linked throughout the record.
The system
Three services wired into one loop: an API, an LLM classifier, and a RAG agent.
Notes API BackgroundTask Classifier Tags Knowledge base KB Agent Grounded answer
The decision log
Reversed. BM25 retiredEVAL · BM25
I added retrieval grounding, measured it, and cut it
I added BM25 lexical retrieval to ground each call, expecting a clear win. The first measurement said it barely helped. Then I found the measurement itself was unfair.
The result: I had been scoring the grounded arm against a stale baseline built on an older prompt. Rebuilt so both arms ran the same prompt, grounding fixed a domain call zero times and broke four across 162 classifications. It was retired outright, not left on as an option.
An unfair baseline flatters the change you hoped for. The retrieval code stays in the repo so the measurement can be rerun. Full eval →
Reversed. Model gated on evidenceSYS-002 · ADR-013
Gate model choice on evidence
Sonnet by default across the system. Escalate to Opus only where an eval shows the quality gain pays for itself. I built the upgrade, measured it, and declined it.
The result: routing the uncertain cases to the premium model moved zero rows on both axes at about 1.97× the cost per article. Sending everything to the premium model scored identically, so there was no headroom for any router to capture.
A targeted prompt fix had already taken the ground routing was aimed at. The cheapest fix won. The harness stays in the repo as the record. Full verdict →
Eval rebuiltEVAL · v1 → v3
I rebuilt the eval on human-labeled real text
v1 graded the model on 300 snippets it generated itself. That measures consistency, not correctness. v2 replaced that with 54 hand-labeled real snippets, cross-checked by an Opus judge.
The result: domain accuracy fell from 97.3% to 88.9% on real text, and that lower number is the trustworthy one.
Category accuracy actually rose, because real vocabulary separates more cleanly; domain fell because the synthetic set was trivially easy to grade. Full eval →
Eight more, in one line each. The full telling of every one is on the page it links to.
- Decoupled the repos with frozen HTTP contracts — then audited the seam and found it broken with both repos' CI green, because each side asserted against its own copy of the shape. Two unit tests that happen to agree are not a contract test. The System →
- Audited my own gates and found six of them reporting success for work that had not happened — a review lane that went green in 36 seconds having posted nothing, and the counter built to catch that reading a key that does not exist. False Green →
- Reversed. Built hybrid lexical retrieval to rescue one specific miss, measured it, and kept the dense-only default — MRR identical to four decimals, and the miss it was built for turned out not to be a lexical failure at all. Retrieval →
- Nearly published a dramatic premium-vs-cheap model gap that was my own bug — a
max_tokenscap truncated the cheap tier before its verdict on 20% of claims, and my scoring counted each as a disagreement. Judge → - Called it a ceiling, then went looking anyway — the misses clustered on one boundary, and two rubric clauses moved category 90.7% → 94.4% with nothing regressing. A prompt problem wearing a ceiling's clothes. Classifier →
- Reversed. A prompt that read better to a human and measured worse — category 79.0% → 76.7%, industry recall 0.217 → 0.100. Reverted. Classifier →
- Reversed. Built the enrichment loop on Kafka to learn it properly, then cut it — a single-user idempotent workload never needed a distributed log. Now an in-process background task. The System →
- Forced output to a tool schema instead of parsing prose, so an invalid label is rejected at the API layer — one out-of-enum response in 300, handled by a re-sample. Classifier →
Classifier evidence
Where the classifier stands now, on 54 hand-labeled real snippets.
- 94.4%Category
- 98.1%Domain
- 94.4%Region
Category and region are scored separately. They happen to land on the same figure here.
Those numbers are from the harder gold set. Full eval →
The classical baseline that lost the case for an LLM is vendored here and runs in the browser, on the same rows. Run it yourself →
Field notes
Notes I keep while learning, in plain language, on the techniques behind all of this. Read them →
- Structured output via tool-use
- Eval-driven development
- Reading the numbers
- Tool use, the general idea
- The agentic tool-use loop
- RAG: answering from your own docs
- Embeddings and vector stores
- Synthetic data and circular eval
- Tiered model routing
- Secrets and env
- Reproducible environments with uv
- Checkpoint, resume, and retries
A selection; the full, growing set is on the notes site.
About
For the short version of how I came to this work and how I approach it, read About →
It is a solo build, so the program layer is simulated — I lay out what that means on the Product & Program page.
A second record sits off this system: a physical lab that makes a factory-blank router provision itself. Zero-touch provisioning →
Outside the work I hike, take pictures, and live with a Scottish Fold named Sango. The rest is on About and in the gallery.
