Backline

An agent platform for music label ops: 3 agents over 468K royalty lines, structured-first RAG, human-gated writes, and a 133-question eval suite with an exact answer key that gates CI.

Comment

Retrieval over near-identical documents doesn't degrade. It fails outright — MRR 0.000.

385 contracts generated from the same templates are indistinguishable to an embedding model. The fix wasn't a better embedder. It was resolving which documents govern an artist as of a date, in SQL, before any vector math ran.

Inspiration

I wanted to answer, with measurements instead of opinions, what actually makes LLM systems reliable on structured business data — and to do it on a domain where the failure modes are expensive and specific rather than generic.

Music label operations is that domain. Contracts amend each other and supersede at clause granularity. Royalty statements arrive from six distributors in six shapes. Getting an answer subtly wrong means paying the wrong artist the wrong amount, and nobody notices for a quarter.

Backline is the platform layer under that. In music, the backline is the gear behind the band — the amps and drums that make the show possible without ever being the show.

What it does

Three agents on one shared runtime:

  • Counsel — contracts and deal-terms Q&A, clause-cited.
  • Analyst — read-only SQL analytics.
  • Reconciler — the workflow: ingest a distributor drop → match to catalog → scan anomalies → compute allocations → submit for human review.

All three are configurations of the same AgentRuntime: same loop, same tracing, same guardrails, differing only in prompt, tool set, and model policy. A cheap-model router classifies each incoming message and asks a clarifying question instead of guessing.

11 typed tools behind Pydantic schemas — search_contracts, read_clause, calc_royalties, sql_query, ingest_statement, match_lines, scan_anomalies, compute_allocations, submit_batch, save_note, recall_notes.

Guardrails at the parser, not the prompt. The SQL tool rejects anything but a single read-only SELECT against allowlisted schemas. The truth schema — the answer key — is dead by construction, and a test pins it that way; an eval question that even attempts to reach it fails. Retrieved document text is fenced as data, and a seeded injection canary must be flagged, never obeyed.

All writes gated. Agents propose, humans approve. The only write path any agent has lands in staging and the Review Queue. Approval is a guarded SQL transition performed by a person — that's what promotes lines into label state.

Everything traced. Every run emits a span tree (run → iteration → LLM/tool/guardrail/compression) with tokens, cost, and latency, streamed live to the UI over SSE and persisted to Postgres + JSONL. No silent LLM calls anywhere: every call goes through a Provider — Anthropic, OpenAI-compatible, or a deterministic mock.

How I built it

Python 3.12 + FastAPI, Postgres with pgvector, uv, mypy --strict. Next.js UI with four surfaces: Chat, Trace Inspector, Review Queue, Eval Dashboard.

The foundation is a deterministic synthetic label — Foldback Records, 150 artists, 385 contracts, 468,160 statement lines, built from a seed in under 3 minutes. 14.1M corpus tokens, or 70.5× a 200K context window, which answers "why not just paste it all into the model" empirically instead of rhetorically. Because the world is generated, every eval question has an exact answer key. That single property is what makes everything else in the project measurable.

Retrieval is structured-first: a SQL join resolves governing documents with amendment supersession at clause granularity, and only then does hybrid FTS + pgvector retrieval with RRF fusion and cross-encoder reranking run — over governing clauses only.

Deployed to AWS with Terraform (ECS Fargate + RDS/pgvector behind an ALB), the full 133-question suite re-run on cloud infrastructure as a one-off Fargate task to prove parity, then everything destroyed the same day. The artifact is the Terraform tree, the runbook, and an invoice that matches the estimate.

Cold start is one command, ~4 minutes with Docker pulls included:

git clone https://github.com/sergioavilax/backline && cd backline
make up

No API key required. Keyless, the chat serves a scripted demo through the real stack — real router, real tools, real SQL, real staging writes, every span traced. Add ANTHROPIC_API_KEY and the same chat runs live agents.

Challenges I ran into

Retrieval scored a clean zero and I nearly shipped past it. Unscoped vector retrieval over the contract corpus returned MRR 0.000 — not degraded, not noisy, zero. The instinct is to assume a config bug and move on. Interrogating it instead was the whole project: 385 contracts from shared templates are lexically and semantically near-identical, so cosine similarity has nothing to grip. No embedder fixes that. A structural filter does.

Mode MRR R@10
Unscoped vector retrieval 0.000 —
Governing-scoped + reranked 0.398 0.97

The eval suite's first real job was auditing itself. Its first week of live runs surfaced four bugs — in my own harness, not the model: a cost meter billing 1.5× actual, a concurrency race in the budget enforcement gate, a per-run cap sized below the workflow it was gating, and an output-truncation hazard acting on mid-JSON truncated replies. Model hallucinations found in the same window: zero. The measuring instrument was the least reliable thing in the room, which is worth knowing before you trust any number it produces.

A provider outage froze ten zeros into a benchmark row. The harness recorded them as real scores. Adjudicating that from spans, then writing a regression test so it can't recur silently, took longer than the benchmark did.

Accomplishments that I'm proud of

A 133-question, three-tier eval suite with exact ground truth, gating CI on every PR against committed baselines. A sabotage test proves the gate can actually fail. Nightly runs the full suite on main. tests/test_docs.py pins the README's own claims to committed artifacts.

A model benchmark that produced a finding people usually argue about by vibes:

Model Score $/query
claude-opus-5 91.7 $0.2432
claude-sonnet-5 91.6 $0.0591
claude-haiku-4-5 82.7 $0.0154

A dead heat at roughly 4× the price — a 0.09 gap against a same-model run-to-run noise floor of 3.2 points. 73 of 133 questions scored identically across the entire sweep. The haiku→sonnet step buys 8.9 points for +$0.044/query; the sonnet→opus step buys 0.09 for +$0.184, about 400× steeper per point. Instruction-following under adversarial pressure turned out not to be price-tiered at all.

The whole thing built in roughly 26 hours, public and MIT from the first commit, with every judgment call logged in a decision record.

What I learned

Interrogate your zeros. A metric that reads exactly 0.000 is not a broken config to route around — it's the most information-dense result you'll get all week. Every good finding in this project came from refusing to explain one away.

Guardrails belong in the parser, not the prompt. A prompt instruction is a request. A SQL parser that rejects anything but a single read-only SELECT is a property, and you can write a test that proves it.

Structure beats similarity when documents are near-duplicates. If your corpus has relational truth in it — dates, supersession, ownership — resolve that first and let vector search work on the remainder. Most RAG advice assumes documents are independent. Business documents aren't.

Benchmark your own instrument before you benchmark anything with it. Four harness bugs, zero model bugs.

What's next for Backline

Nothing further, on purpose.

Backline is a finished, MIT-licensed artifact — a platform built to produce measurements, and it produced them. It isn't becoming a product and won't be onboarding users. What's left is durable: a public repo you can clone and run keyless in four minutes, a generator that builds a 468K-line world from a seed, and an eval methodology worth stealing.

The retrieval finding is the transferable part. Point it at your own near-duplicate corpus and see what your MRR actually is.

Built With

Share this project:

Updates