Newsroom Fleet

A fleet of editorial agents gives a small newsroom the checks of a national masthead claim verification, standards review, corrections. The model never publishes. Your editor does.

Comment

The inspiration

A local paper and a national masthead carry identical legal exposure. Only one of them has a data desk, a standards editor, and a fact-checker.

That asymmetry is the whole problem. When a five-person newsroom publishes a wrong unemployment figure, a quote the transcript doesn't support, or the word "guilty" where the record says "charged," there is no institutional layer between the reporter and the consequence. The paper that story belongs to can be gone. Not because anyone was careless because the checks that catch those mistakes are staff positions, and the staff isn't there.

The obvious move is to point an LLM at the draft and ask "is this true?" We deliberately did not build that. A confident model that occasionally verifies something false is worse than no system at all, because it manufactures the appearance of a check. So we inverted the design: the agents produce evidence, and a deterministic policy decides what publishes.

What we built

Newsroom Fleet reconstructs the missing desks as a fleet of bounded specialist agents claim extraction, source verification, data checking, standards review Google ADK agents on Gemini 3.7 Flash, running on Cloud Run.

A draft moves through the pipeline like this:

  1. Screening at intake. The body and every attached source are screened before orchestration. A quarantined source's content is dropped at the router reviewers receive only its screening metadata, so an injected instruction never enters a reviewer's context.
  2. Decomposition. The draft becomes atomic claims, each independently checkable.
  3. Policy routing. Each claim is routed by code, not by a model, to the desks whose evidence boundary covers it, with a defined minimum evidence per desk.
  4. Bounded review. Each desk returns a signed structured verdict. A live desk may only cite a locator it was actually handed; an unrecognised locator rejects the citation and downgrades the verdict.
  5. The verdict matrix. Aggregated per claim, with disagreement preserved rather than averaged away.
  6. The Editor Gate. A deterministic policy evaluation over persisted verdict state. It denies, and it never mutates a verdict.
  7. Publication as a human act, recorded with identity and timestamp. Numeric claims are snapshotted, because numbers change.
  8. The corrections watcher. A scheduled job compares published snapshots to live authoritative data and drafts a correction candidate in house style. It never auto-corrects, and an accepted correction is appended the story is never rewritten.

One invariant governs all of it, and it's enforced in code rather than promised in a prompt:

Missing evidence, disagreement, quarantine, low confidence, or worker failure can never become VERIFIED.

How we built it

Every cloud-bound capability sits behind an interface with a local implementation, and each one is an independent switch. That single decision shaped the whole project.

Interface Local (default) Google Cloud Switch
Desks fixture (deterministic) ADK + Gemini 3.7 Flash NRF_MODE=live
Repository SQLite Firestore NRF_REPOSITORY=firestore
ReviewQueue asyncio Pub/Sub + push worker NRF_QUEUE=pubsub
Screener heuristic detector Model Armor NRF_SCREENER=model_armor
MemoryStore JSON file Vertex AI Memory Bank NRF_MEMORY=memory_bank
Tracing off Cloud Trace NRF_TRACING=cloud

The payoff is that the entire system runs with no API keys, no network, and no cloud account, so the safety properties are testable in CI rather than only observable in a demo. uv run python -m newsroom_fleet.evaluation prints a scored report; the number that matters is unsafe false verifications: 0. And because a fallback is a real possibility, GET /api/runtime reports what was actually constructed a component that quietly fell back to its local implementation is never advertised as running on Google Cloud.

The stack: Cloud Run (asia-south1) for the API and the editor desk, Firestore for claims, verdicts, snapshots and an append-only audit trail, Pub/Sub with a dead-letter queue for review fan-out, Cloud Scheduler for the corrections watcher, and Cloud Trace for a span waterfall per claim. /api/internal/* requires either a shared-secret header or a verified OIDC token from an allowlisted service account and neither endpoint can approve a publication. The backend service account is deliberately not a Pub/Sub subscriber.

Cost discipline was part of the build, not an afterthought: every action taken against Google Cloud including read-only checks is recorded with its cost in deploy/CLOUD_LEDGER.md, and both deploy scripts are dry-run by default. Total cloud spend for the entire project came to roughly ₹2 against a ₹10,000 budget.

Challenges we ran into

Almost everything we learned came from a failure that the fail-safe caught. That pattern turned out to be the most interesting result of the project.

The model's own claim labels were unsafe to route on. Our first live run had Gemini label "the council voted 6-1" as numeric and "unemployment fell to 4.2 percent" as attribution. Both are defensible readings of the prose. Both are wrong for routing: the figure went to the Source Verifier with no source to check, and the vote went to the Data Checker with no adapter coverage. Every claim still blocked, so nothing unsafe escaped but the contradiction the entire case rests on was never found. Routing now lives entirely in domain/routing.classify, reading signals out of the claim text; the model's label is a fallback used only when no signal fires. This is the clearest lesson of the build: failing closed is not the same as working. A system can be perfectly safe and completely useless, and only an evaluation harness with ground truth tells you which one you have.

A user-submitted article found a real cross-article bug. Claim ids (clm_01…) are unique per article, but the verdict slot SQLite primary key, Firestore document id, the runner's idempotency pre-check, and the queue's idempotency key was keyed (claim_id, desk) only. A second article minting the same claim ids silently inherited the first article's verdicts: those desks never ran, and its aggregates overwrote the first article's rows. The gate still failed closed claims blocked as "no verdict on record" which is exactly why it surfaced as missing reviews rather than as a safety hole. The slot is now (article_id, claim_id, desk) everywhere, pinned by a test.

Gemma 4 is a thinking model, and that quietly cost the PII pass its entire budget. At max_output_tokens: 200, every call returned thought parts only response.text was None, finish_reason: MAX_TOKENS so the classifier abstained on everything. A PII hit also reasons harder than a clean pass: 1024 tokens still truncated mid-thought. The escalate-only fail-safe held, which made this a dead feature rather than a safety hole, but a dead feature that reports abstention looks identical to a working one on a clean document.

A 12-claim submission saturated the fleet. review_all fired every claim's Gemini and search agents at once; the event loop crawled a trivial list endpoint took 4.7s and the review stalled behind the storm. Claim review is now bounded by a semaphore, one claim at a time by default, with desks inside a claim still running concurrently. The accidental benefit: sequential pacing streams beautifully in the editor UI, and you can watch the fleet work down the claim list.

Opting out of Gemini request storage took real digging. The standard Gemini API stores GenerateContent requests by default to help with debugging. Opting out is a top-level store: false body field that google-genai 2.18.1 does not yet model we verified live that the API parses and type-checks it while the SDK rejects it on GenerateContentConfig. It travels instead through the SDK's documented HttpOptions.extra_body and ADK's Gemini(client_kwargs=...), so every desk, the search researcher, and the PII pass carry it. A newsroom's unpublished drafts and confidential sources are not something to leave on a default.

There were smaller ones too: live desks were timing out for purely operational reasons because a 5s default is shorter than one Gemini round trip (NRF_MODE=live now defaults to 60s), and the evaluation harness caught an extractor gap on its very first run where comma-grouped figures like "812,000 trips" were typed general and never reached the Data Checker.

What we learned

Fail-safe defaults hide bugs as effectively as they prevent disasters. Four of the five problems above were invisible in normal operation precisely because the system correctly refused to publish. Safety gave us permission to ship; only ground-truth evaluation told us whether the thing worked.

The boundary is the product. The valuable engineering wasn't in the agents it was in deciding what an agent is never allowed to do: never choose its own routing, never cite a locator it wasn't handed, never see a quarantined source's content, never approve a publication. Bound the agents tightly enough and you can afford to trust them for what they're good at.

A single flag between deterministic and live mode is the best debugging tool we built. Every live-mode bug was diagnosed by asking "does fixture mode do this too?" and that same seam is what makes the safety claims reproducible on a laptop with no API key.

What's next

The honest gap: identity is a request field, not an authenticated session. The reporter/editor toggle is a demo affordance; the authorisation rule is genuinely server-side and evaluated before anything changes, but a deployable newsroom tool needs a signed token. That's the first thing we'd build.

Beyond it: broadening authoritative adapter coverage (anything outside the configured adapters is abstained on, never estimated), running the evaluation harness in live mode rather than only in fixture mode, and letting the corrections ledger accumulate a newsroom's own approved standards and precedents as institutional memory.

And what will not change: no autonomous publishing, ever. The model never publishes. Your editor does.

Built With

Share this project:

Updates