Verity

Verity runs the evidence before you build on it - autonomously reproducing public AI claims and filing honest, execution-backed verdicts.

Comment

Inspiration## Inspiration

AI teams make expensive decisions based on performance claims from papers, GitHub repositories, and vendor pages. But reproducing those claims is usually slow and manual: someone must identify the exact metric, find the code, reconstruct the environment, run the evaluation, debug failures, compare the result, and document what happened.

The dangerous shortcut is to treat a convincing explanation—or even a successful process exit—as verification.

I built Verity to automate the entire workflow while preserving a stricter principle:

Missing evidence must remain missing. The agent must never manufacture a successful result.

What Verity does

Verity accepts a public arXiv, GitHub, or vendor URL and autonomously:

  1. Extracts a typed numerical claim, including its metric, value, dataset, source quotation, and experimental conditions.
  2. Creates a bounded evaluation plan.
  3. Executes the associated code in an ephemeral, credential-free sandbox.
  4. Examines real errors and performs at most three transparent repair attempts.
  5. Compares the claimed value with the evidence actually observed.
  6. Stores the complete trace and final verdict.
  7. Files a detailed public GitHub Issue automatically.

Verity can return precise outcomes such as:

  • verified
  • contradicted
  • conditions_not_comparable
  • inconclusive
  • could_not_verify
  • no_verifiable_claim_found
  • environment_incompatible

These outcomes deliberately remain separate. For example, an evaluation that exits successfully without producing the claimed metric is not considered verified.

How I built it

Verity is a Taskmaster-style asynchronous workflow built on Google Cloud.

The public FastAPI service runs on Cloud Run. It validates the submitted URL and stores durable jobs, traces, verdicts, and canonical-URL claim memory in Firestore.

New jobs are published through Pub/Sub, which invokes the private pipeline using an audience-bound Google OIDC token. The private pipeline then launches a fresh Cloud Run Job for each sandbox execution attempt.

The reasoning stages use Gemini 3.5 Flash on Vertex AI through typed Google ADK LlmAgent instances:

  • Parser: reads unstructured source material and extracts a precise typed claim.
  • Debug: analyzes genuine execution failures and proposes bounded repairs.

Two other stages are deterministic Python by design:

  • Environment: constructs the bounded execution specification.
  • Reporter: compares claimed and observed values and produces the final report.

This boundary is intentional. Model reasoning helps interpret research claims and diagnose failures, but command construction and final numerical comparison should not depend on generative output.

The final Reporter files a durable public artifact through the GitHub API. Re-submitting a completed canonical URL returns its Firestore-backed result with cached=true, avoiding duplicate execution and cost.

Security architecture

Untrusted benchmark code runs in a separate Cloud Run Job under a service account with no project roles.

The sandbox receives:

  • no Google Cloud credential;
  • no application API key;
  • no GitHub credential;
  • no Firestore access;
  • no Secret Manager access;
  • no Pub/Sub permission;
  • no permission to launch other Cloud Run jobs.

Before production deployment, I ran a live stolen-identity security probe from inside the sandbox. It attempted six sensitive operations against Firestore, Secret Manager, Pub/Sub, Cloud Run, Vertex AI, and Cloud Storage. All six were explicitly denied.

Only the trusted pipeline can persist evidence and file the final report.

Challenges I faced

Designing a genuinely credential-free sandbox

Separating the sandbox into another container was not enough. I had to test the complete identity boundary: IAM roles, secrets, environment variables, job overrides, control-plane access, logs, and the handoff of evidence back to the trusted pipeline.

Cloud Run health routing

A long investigation produced Ready revisions and correct authentication but Google-front-end 404 responses with no request logs. The root cause was the /healthz path, which Cloud Run can reserve or intercept. Renaming the route and startup probe to /health resolved the issue.

Least-privilege Cloud Run execution

The pipeline needed permission to launch sandbox jobs but should not receive broad Cloud Run read access. Verity now extracts the execution identity from operation metadata and reads one bounded result from Cloud Logging instead of granting unnecessary control-plane permissions.

Representing commands in Firestore

Firestore does not support nested arrays. Execution commands therefore needed an explicit serialization boundary instead of storing Python command structures directly.

Staying honest under time and resource limits

Some heavyweight evaluations cannot finish inside the intentionally bounded sandbox budget. Verity reports that limitation rather than increasing resources until a desirable result appears.

What I learned

I learned that safe agentic execution requires more than a good prompt or isolated container. The model, state machine, queue, identities, secrets, execution environment, retry policy, and evidence channel all need explicit contracts.

I also learned that deterministic components are an important part of an agent architecture. Not every stage benefits from model reasoning. Keeping numerical comparison and security-sensitive execution construction deterministic makes the overall system more trustworthy.

Most importantly, I learned that a useful verification agent must be designed to produce honest negative results. Verity's live cloud runs produced no_verifiable_claim_found, could_not_verify, and inconclusive outcomes. I kept those results because refusing unsupported success is the central feature of the project.

Accomplishments

  • Deployed a complete public and asynchronous Google Cloud workflow.
  • Integrated Gemini 3.5 Flash with typed Google ADK agents.
  • Implemented durable Firestore state and cross-revision deduplication.
  • Built authenticated Pub/Sub delivery with Google OIDC.
  • Proved a no-role Cloud Run sandbox against six sensitive APIs.
  • Added bounded, fully visible debugging attempts.
  • Generated real public GitHub reports autonomously.
  • Created local and cloud profiles behind the same interfaces.
  • Passed the complete CI suite, including type checking, unit tests, Docker tests, container builds, and isolation validation.

What's next

The next improvements would strengthen scientific reproducibility rather than simply increase the number of supported sources:

  • cryptographically pin source, dataset, checkpoint, dependency, and runner artifacts;
  • record richer hardware and environment equivalence;
  • add a transactional delivery outbox for automatic queue recovery;
  • support larger evaluations through explicitly priced execution profiles;
  • add organization-level policy controls and private report destinations;
  • extend claim extraction to more multimodal benchmark formats.

Verity does not promise that every claim can be verified. It promises that every verdict is traceable to what actually ran.

What it does

How we built it

Challenges we ran into

Accomplishments that we're proud of

What we learned

What's next for Verity

Built With

  • artifact-registry
  • cloud-build
  • cloud-logging
  • cloud-run
  • cloud-run-jobs
  • docker
  • fastapi
  • firestore
  • gemini-3.5-flash
  • github-actions
  • github-api
  • google-adk
  • google-cloud
  • opentelemetry
  • pub/sub
  • pydantic
  • python
  • secret-manager
  • uvicorn
  • vertex-ai
Share this project:

Updates

Submission history