Proofline

An execution and verification layer for AI agents — so "done" comes with evidence, not just a claim.

Comment

Inspiration

What it does

How we built it

Proofline

Proofline gives an AI agent an execution trail and a verification layer, so a completed task can be inspected rather than merely trusted.

Problem

AI agents are getting good at saying they did something. That is not the same as it having actually happened, happened correctly, or being checkable after the fact. Most agent demos are a chat transcript: the model says "done," and you take its word for it.

Solution

Proofline sits between an AI agent's proposal and a real outcome. A bounded Gemini agent reads real, typed evidence about a submitted result and proposes what's wrong with it but it never gets to decide the outcome. A separate, deterministic verifier checks the claim against actual recorded artifact identities and versions. If evidence is missing or inconsistent, a durable correction task is created and the result is held open. Only an explicit human reviewer can approve closure. Every step is recorded in an audit trail.

The shape is:

TASK → EXECUTION → EVIDENCE → VERIFICATION → HUMAN APPROVAL

not:

CHAT → MESSAGE → MESSAGE → MESSAGE

Why Proofline is different

Typical agent Proofline
"I completed the task." "I completed the task. Here is what I executed, here is the evidence, and here is how the result was independently verified."
Model decides when it's done A deterministic, non-AI check decides; the model only proposes
No durable record Every state transition and event is idempotent and auditable
Trust the summary Inspect the evidence

The model (Gemini) never has authority to close a result. It proposes a classification and, when evidence is genuinely missing, a correction task. A separate VerificationService — plain Python, no model call — checks the claim against the actual artifact registry. Closure requires an explicit reviewer action. This boundary is enforced in code (proofline/workflow.py, proofline/verification.py), not by convention.

Demo: a real case, not a script

The featured workflow ingests a real, independently-checkable case: a Seattle weather dataset analysis pinned to an actual public GitHub commit (vega-datasets@4ddef64). The dataset's SHA-256 in the manifest was verified against a fresh download of that exact commit during this project's own testing (see PROVENANCE_REPORT.md) — it is not an invented checksum.

Running it through Proofline: the agent inspects the evidence, two obligations (dataset integrity, parameter record) are genuinely provable and pass; two (Gemini execution provenance, source repository link) are genuinely missing because that run never actually executed Gemini or had a repository present — and Proofline says so, honestly, rather than closing anyway. That gap is real, not staged; see test-cases/clean_seattle_weather_case.json and PROVENANCE_REPORT.md.

A second, minimal case (res-infra-check-cloudrun-vertex-001) was run all the way through OPEN → UNDERSTANDING → VERIFYING → READY_FOR_REVIEW → CLOSED against the live Cloud Run deployment today, proving the full approval-gated happy path works end-to-end in production, not just locally.

Architecture

Proofline architecture diagram

USER
  |
  v
FRONTEND (React, served by the same Cloud Run container)
  |
  v
CLOUD RUN  ──────────────────────────►  FIRESTORE (durable state: results,
  |                                       obligations, tasks, audit events)
  v
PROOFLINE WORKFLOW ENGINE
  |
  +--> GOOGLE ADK AGENT (bounded proposal agent, one tool-free turn) ──►
  |     VERTEX AI / GEMINI (gemini-3.5-flash, served from the "global"
  |     Vertex AI location, with automatic routing to the same model
  |     for multi-obligation cases and gemini-3.5-flash-lite as fallback)
  |
  +--> TYPED TOOLS (read_evidence_bundle, propose_correction_task —
  |     allowlisted, validated, no free-form execution)
  |
  +--> DETERMINISTIC VERIFIER (no model call; checks artifact identity,
  |     project scope, availability, and version against Firestore)
  |
  +--> CLOUD STORAGE (large artifact bytes, referenced not embedded)

Pub/Sub (proofline-events topic, push subscription to /events) exists for asynchronous result-submission ingestion but is not required for — or exercised by — the primary UI-driven demo path. See ARCHITECTURE.md for the full diagram and what's deployed vs. deferred.

Google Cloud usage

  • Google ADK — the reconciliation proposal step runs through a real google.adk.agents.Agent executed via ADK's Runner (proofline/adk_reasoning.py), not just Vertex AI called directly. ADK's authority is scoped to that one tool-free turn; it never touches workflow state or closure.
  • Vertex AI / Gemini — real model calls via the google-genai SDK, Application Default Credentials (no API key), from both local development and the deployed Cloud Run service. Verified working end-to-end today; see CLOUD_DEPLOYMENT.md.
  • Cloud Run — hosts both the API and the built React frontend from one container (Dockerfile builds the frontend during the image build itself).
  • Firestore — durable state: results, obligations, tasks, audit timeline. Native mode, (default) database.
  • Cloud Storage — artifact bytes (referenced by URI, not embedded in Firestore).
  • Pub/Sub — optional async ingestion path, OIDC-authenticated push subscription.
  • Secret Manager — the direct-ingress event token; no secrets in source, environment, or logs.
  • Artifact Registry — the built container image.

Verification and provenance

proofline/verification.py checks, deterministically:

  • Does the linked artifact actually exist in this project?
  • Is it available?
  • Does its recorded version match what the obligation claims?

A controlled tamper test (documented in PROVENANCE_REPORT.md) shows that changing a single value in the dataset produces a completely different SHA-256 — proving the recorded checksum is a real, sensitive fingerprint of the file's content, not decoration.

Local setup

python run_server.py

Open http://127.0.0.1:8000. Uses SQLite and a deterministic offline model provider by default (no cloud credentials required). To build and serve the React frontend:

cd frontend
npm install
npm run build
cd ..
python run_server.py

Cloud deployment

gcloud run deploy proofline-service --source . --region us-central1

The multi-stage Dockerfile builds the frontend during the image build — no separate local build step is required before deploying. See CLOUD_DEPLOYMENT.md for the exact deployed resources, and the one action item still required from a human (making the service publicly reachable — see below).

Test case manifests

test-cases/ holds real, ingestible case manifests for exercising the workflow end-to-end:

  • clean_seattle_weather_case.json — the featured real case (see above), where two obligations are genuinely missing and Proofline says so.
  • correction_flow_demo_case.json — a case designed so a human correction submission can genuinely, honestly succeed.

Paste either into the ingestion screen's "Paste JSON" tab, or use the guided form to build a new one from scratch.

Tests

python -m pytest tests/ -q

26 tests covering the vertical slice, cloud contracts, event idempotency, model routing, restart durability, optional-dependency handling, and today's two regression fixes (Vertex model config per model family, and correction-task backfill for obligations the agent itself is barred from targeting). See TEST_REPORT.md.

Limitations

  • The live Cloud Run service is now publicly reachable. Making it public required a deliberate human decision — the coding agent's own safety controls blocked granting allUsers invoker access, since that's an internet-exposure change an automated agent shouldn't make silently. The project owner ran the command directly; confirmed live.
  • No real user authentication. Actor identifiers like reviewer:x are self-declared strings from the client, not verified identities. Acceptable for this submission; the top item for real production hardening.
  • Verification checks identity/version, not a live re-hash of remote content. The tamper test proves checksums are meaningful; automatically re-fetching and re-hashing an artifact's URI on every check is future work, not wired into _check_artifact today.
  • See TEST_REPORT.md and PROVENANCE_REPORT.md for the complete, honest list.

Challenges we ran into

Accomplishments that we're proud of

What we learned

What's next for Next Gen Agent

Built With

Share this project:

Updates

Submission history