Sourcetrace

An event-driven release gate that verifies factual claims before technical prose ships.

Comment

What it does

Sourcetrace is an autonomous release gate for factual claims in technical prose. Instead of asking an author to paste a document into a chatbot and manually interpret the answer, Sourcetrace handles the complete review workflow: detect a changed document, extract atomic claims, choose the appropriate verifier, retain the evidence, decide whether the change can ship, and report the decision where the work happened.

A signed GitHub pull-request webhook creates a durable, idempotent Firestore job. Pub/Sub invokes a Cloud Run worker asynchronously with a service-account OIDC identity. Gemini 3.5 Flash extracts atomic claims; Gemma 4 supplies an independent claim-type label; deterministic SI and symbolic verifiers handle applicable STEM claims; Gemini Search grounding handles source-backed factual claims; and unsupported claims abstain instead of guessing. Google ADK provides the research → analysis → verification → review agent workflow.

Why it is agentic

Sourcetrace is not a chat interface or a one-shot LLM call. It runs a multi-stage background workflow and takes an operational action:

  1. receives and authenticates a pull-request event;
  2. creates exactly one durable job for the delivery;
  3. dispatches asynchronous processing;
  4. extracts and routes each claim;
  5. executes deterministic or grounded verification tools;
  6. persists evidence and a tamper-evident application audit chain;
  7. calculates a release decision; and
  8. optionally posts the result back to the pull request.

Release policy

  • Any rejected claim → BLOCK
  • No rejection but at least one abstention → REVIEW
  • Otherwise → PASS

Production architecture

  • Gemini 3.5 Flash and the Google GenAI SDK
  • Gemma 4 (gemma-4-31b-it) as a second-opinion classifier
  • Google Agent Development Kit (ADK)
  • Cloud Run for the API and asynchronous worker
  • Firestore for durable job state and sequenced audit events
  • Pub/Sub for at-least-once asynchronous delivery
  • Secret Manager for runtime credentials
  • Dedicated Cloud Run runtime and Pub/Sub push service accounts
  • HMAC-SHA256 GitHub webhook verification
  • OIDC-authenticated Pub/Sub push
  • Transactional processing leases and idempotent job creation

Firestore stores both durable job state and a sequenced, hash-linked audit trail. Duplicate intake resolves to one deterministic job, concurrent redelivery is controlled by a transactional lease, and an upstream extraction failure produces an explicit failed job rather than an empty success.

Data and evidence sources

Sourcetrace operates on the prose changed in a GitHub pull request or submitted through its API. Verification evidence can come from a deterministic known-value registry, symbolic comparison, and URLs returned by Gemini Search grounding. The project does not train on or silently retain submitted documents; persisted audit detail uses fingerprints and redacted summaries rather than presenting raw model output as proof.

What we learned

The central challenge was not adding another model call—it was making failure semantics trustworthy. At-least-once delivery requires idempotency and leases. A missing source must become an abstention, not a confident verdict. A model outage must become a failed job, not an empty PASS. A second model can improve routing diversity, but it must not be allowed to override deterministic evidence. Finally, an audit trail is only useful when it is tied to the exact job, sequence, deployment revision, and previous event hash.

Reproducibility and proof

The README contains local smoke-test and Google Cloud deployment instructions. The repository includes network-free tests, a deploy script, a threat-aware architecture document, the required architecture diagram, and a four-minute live-demo run sheet.

Honest boundary

A PASS means that the claims Sourcetrace extracted cleared the configured verifiers. It is not proof that the document is complete or universally true. Claim extraction can miss content, source quality still matters, and high-stakes publication requires human review. canClaimAGI: false.

Built With

  • cloud-run
  • fastapi
  • firestore
  • gemini-3.5-flash
  • gemma-4
  • github-webhooks
  • google-adk
  • google-genai
  • pubsub
  • secret-manager
Share this project:

Updates