Blazewatch

Quality per dollar for generative pipelines — know which provider and model are actually worth what you're paying

Comment

Inspiration

Generation pipelines built on genblaze can point at any number of providers and models, and every one of them will happily produce output. What nothing tells you is whether the expensive provider was actually worth it, or whether a provider you already trust has quietly gotten worse. Two systems hold half of that answer each: a Backblaze B2 bucket knows what was produced, and a Parquet warehouse knows what it cost. Neither alone can answer quality per dollar, by provider and model — only a join across both can. That question, plus a second one that follows from it — "did this provider silently swap the model behind a floating alias like gpt-5.1 and change my results without telling me?" — is what BlazeWatch exists to answer.

What it does

BlazeWatch is an observability layer that sits over a genblaze generation pipeline and does three things:

  • Reconciles, rather than reacts. It sweeps the B2 bucket, diffs what's been generated against what's already been scored or already failed, and scores only the difference. A repeated sweep costs almost nothing; a crashed cycle loses nothing; the first sweep backfills a bucket's entire history for free.
  • Scores quality with a vision-model judge. Two metrics — vision-quality (a defect score) and vision-adherence (does the image match its prompt) — come out of one shared API call per asset, judged by a real vision model rather than a fixed algorithm, so it judges execution within the style attempted instead of penalizing illustration for not being a photograph.
  • Joins quality against cost in a DuckDB warehouse built over the pipeline's Parquet output, and serves the result — a quality-per-dollar leaderboard, per-run cost and latency, and per-metric distributions — through a Next.js dashboard that never lets an unmeasured asset render as a perfect score.
  • Watches for drift. A scheduled job generates a fixed, versioned "golden canary" prompt set against every configured provider, embeds the judge's description of each result, and runs four independent statistical tests — PSI and KS on the scalar quality/adherence values, MMD on the embeddings, and a Bayesian online changepoint detector over the MMD series — against a rolling reference window. The goal is catching a silent model swap before the provider announces it, not building another benchmark leaderboard.

How we built it

The core of the system is a three-listing set difference (assets - scored - failed = todo), where a score's identity is sha256(key + etag) — so the "already done" set arrives as one listing per prefix instead of one HEAD request per object. That single design choice is what makes the whole system idempotent and backfill-free.

Around that core:

  • Python + FastAPI for the reconciliation poller and the read-only dashboard API, with the poll loop hosted as a lifespan task so the worker and the HTTP surface can run as one process or two.
  • DuckDB over Parquet for the warehouse — the asset_quality join and the quality-per-dollar query are plain SQL views the dashboard, the CLI, and a drift job all read the same way.
  • OpenAI's vision and embedding APIs for judging. The judge is asked for a compact text description as part of the same call that scores quality/adherence, and that description is what gets embedded for drift detection — one extra API call reuses work the judge already did instead of computing a second, true visual embedding.
  • Pure-numpy statistics for drift — PSI, KS, MMD, and BOCPD are all implemented from scratch with no scipy dependency, so the drift feature adds exactly one optional package.
  • Next.js 15 + React 19 + Recharts for the dashboard, rendered server-side so the browser never talks to the API directly and B2 credentials never leave the server; asset previews are proxied through the Next.js server instead.
  • Backblaze B2 (via the S3-compatible backend) as the object store, and a Render blueprint that deploys the API as a private service and the dashboard as the only public one.

Challenges we ran into

  • The original design was a webhook, and it worked — it just couldn't ship. B2 fired an event, an endpoint verified an HMAC and scored the object; both structural invariants were verified under 47 checks plus 26 edge cases. Then it turned out the Backblaze account can't create event-notification rules at all, so no delivery would ever arrive. That forced the move to polling — which turned out to be the better design anyway (nothing to retry, nothing to lose on a crash), but only because the account limitation forced the question.
  • The image-quality metrics worked on a workstation and nowhere else. BRISQUE, NIQE, and CLIP adherence ran fine locally over torch, but the Dockerfile only ever installed requirements.txt — never the ~2GB torch stack — so the deployed poller could never actually run an image metric. That was only discovered once the container shipped. Worse, once measured against this project's own data, BRISQUE/NIQE turned out to be undefined outside real photographs (5–10x out of range on the synthetic gradient demo images), and CLIP adherence isn't comparable across different prompts at all. All three got retired in favor of a vision-model judge that needs no new install and no new credential.
  • A NULL column with one exception broke the warehouse read. steps.error is NULL on every successful run, so DuckDB infers it as a null type from whichever Parquet file it reads first — then throws a conversion error the moment it hits the one run that actually failed. Fixing the schema mismatch (union_by_name) wasn't enough; every typed read downstream still has to cast explicitly, because the column's type, not just its name, differs file to file.
  • Alerting on three independent statistical tests without a false-positive flood. The obvious answer, Benjamini-Hochberg FDR correction, needs a well-defined test family, and there wasn't one obviously correct choice — across cells tested the same day, across the three tests in one cell, or across a cell's own history. We deliberately chose something a reviewer can check by eye instead: alert only after a cell breaches on multiple consecutive cycles in a row.
  • Validating drift math with no live traffic to test it against. PSI, KS, MMD, and BOCPD all needed correctness checks before ever touching real data, so demo/check_drift_stats.py asserts textbook-known behavior directly — PSI near zero (in median, over repeated trials) on identical distributions, MMD significant on a visibly different one, BOCPD dating a synthetic mean-shift to the right run length.

Accomplishments that we're proud of

  • A sweep that costs ceil(n/1000) requests per prefix, not n — the entire history of a bucket backfills for free, on the first sweep, with no special case for it.
  • A dashboard that refuses to lie in three specific ways: an unmeasured asset never renders as a number, two differently-configured metric sets are never silently pooled into one leaderboard row, and quality bands are read off the stored record rather than hard-coded from documentation that can drift out of date.
  • A drift-detection pipeline that piggybacks entirely on infrastructure that already existed — the canary generator writes into the same asset prefix the ordinary poller already sweeps, so scoring a canary image costs zero extra configuration and the same reconcile guard that stops the poller from scoring its own output applies automatically.
  • Two independent cost levers — image detail and reasoning effort — tuned separately for two different judging tasks (spotting a defect vs. verifying prompt adherence) that still share one API call instead of two.

What we learned

  • Reconciliation beats delivery for this kind of problem. A missed webhook is a permanently lost event; a missed poll cycle just gets recomputed by the next one. Once identity is deterministic, "did I already do this" becomes a listing instead of a promise you have to trust.
  • A metric that can't run where the code ships isn't a metric. Verifying something on a workstation says nothing about a container that never installs the dependency it needs — that gap should be checked against the actual Dockerfile, not assumed.
  • Worst-of catches what mean-of hides. A pristine, prompt-ignoring image averages out to "unremarkable" quality but is exactly the case worth flagging — the two metrics measure unrelated failure modes, so the aggregate has to ask "is anything wrong," not "how wrong on average."
  • A floating model alias is a versioning hazard, not a convenience. gpt-5.1 can silently resolve to a new dated snapshot with no code change visible in a diff, which is why every score now records the snapshot that actually answered, not the alias that was requested.
  • "Not enough data yet" needs to be a visible state, not an absent one. A drift cell with an insufficient baseline still writes a row, with every test column NULL — so a fresh deployment reads as "nothing to compare yet," not as a system that silently isn't working.

What's next for Blazewatch

  • Run the vision metrics over a live bucket at scale. They're verified against single images and dry-runs today; the reconciler path with these metrics attached has never actually swept a real bucket.
  • Price the judge itself into the warehouse. Every vision call's token usage is recorded per score, but nothing joins it to a dollar figure yet — so quality-per-dollar currently accounts for generation spend only, and is blind to its own judging cost.
  • Calibrate the vision-quality/vision-adherence thresholds against real data. The current bands are the rubric's own defaults, not a measured calibration the way the retired torch metrics' thresholds were.
  • Exercise the WER/speech metric against a real audio asset. It's written against the documented Whisper API and its gating logic is verified, but no audio asset has ever existed in this project to run it against.
  • Route drift alerts somewhere other than a console line. Persistence-based alerting decides when to alert; there's no notification channel wired up yet for where that alert should go.

Built With

Share this project:

Updates