Proofline

Governed Gemini agents run your SOC 2 / ISO 27001 audit for $0.31, seal every verdict into a hash chain, and give auditors proof they can verify with 120 lines of Python. No trust required.

Comment

Inspiration

Every SOC 2 / ISO 27001 audit begins with weeks of screenshots, ticket exports, and spreadsheet archaeology. Agentic AI can obviously do that collection — but it collides with the auditor's first and best question: "why should I trust your output?" Vendors answer with accuracy claims. Auditors don't accept accuracy claims; they accept evidence they can independently verify. So we built the fleet around that standard instead: Proofline's answer is "you shouldn't have to trust us — here's the proof, check it yourself."

What it does

Proofline runs a weekly, fully autonomous compliance audit of the HR lifecycle (recruitment, onboarding, exit) against ISO/IEC 27001:2022 and SOC 2, over a fictional 16-person company with deliberately messy data:

  • 8 collector agents — one read-only identity per source system (HRIS, background-check vendor, Drive, LMS, Google Workspace, tickets, assets, access reviews) — pull the raw records through a pluggable Connector SPI.
  • A Verifier re-hashes every artifact at collection and quarantines instruction-like content — including a prompt-injection attempt hidden in a contractor's MSA, caught live on camera, with the attempt itself sealed as evidence.
  • A deterministic rule engine issues every verdict of record (70 checks → 44 compliant, 8 needs-information, 18 non-compliant). Gemini 3.5 judges every check in parallel as a rationale layer — explaining, citing policy clauses, drafting remediation — and is measured against the engine on every run with a tiered gate: zero tolerance for a false compliant.
  • Three states, not two: NEEDS_INFO becomes an evidence request to the control owner (drafted by the copilot, sent by a human), not a false finding — the auditor's real workflow.
  • The finalizer seals the ledger into a hash chain (genesis = sha256(manifest) → per-entry chain → one root hash), anchors the root in a retention-locked GCS bucket under a create-only identity, and exports a self-contained Evidence Package: ledger, snapshots, traces, the exact rulebook that judged the run, and verify_bundle.py — ~120 lines of standard-library Python, no network, no Proofline. Change one verdict byte and the chain breaks visibly.
  • The Auditor Workspace streams it all live over Firestore onSnapshot: a mission-control view of the fleet executing, an engine-vs-LLM confusion matrix with an empty false-compliant column, append-only accept/reject decisions that reference the sealed chain (a forged reference is rejected), and an Audit Copilot that can only cite evidence it actually fetched — citations are validated server-side against its tool calls.

Each full audit costs ~$0.31, measured — token counts are facts and live inside the chained spans; dollars live outside the chain.

How we built it

Google stack, end to end. The fleet is a Google ADK agent tree (SequentialAgent/ParallelAgent): 8 collectors fan out in parallel, then verifier → assembler → mapper dispatch → validator → gap analyst → finalizer, all custom BaseAgents with identities, statuses and spans; the Audit Copilot is an ADK LlmAgent with function tools and a terminal respond(answer, citations, ui_action) contract. The Control Mapper dispatches one stateless google-genai call per check to gemini-3.5-flash on Vertex AI — keyless ADC, no API keys anywhere in the system — with structured output (response_schema), an explicitly cached stable prefix (control matrix + policies + SOPs; 70/70 cache hits), precomputed date facts (the model never does calendar math), and per-run nonce delimiters around untrusted evidence. Cloud Scheduler → Pub/Sub → Cloud Run runs it weekly with ack-then-execute, a Firestore lease lock, message dedup, and a daily run cap; Firestore streams entries/spans/agent states to the workspace live; Cloud Build ships both services.

Discipline first. Day one was a golden gate: the deterministic engine had to reproduce the designed 44/8/18 result exactly before any agent code existed. Every later layer is measured against it.

Challenges we ran into

  1. The LLM kept "failing" checks — until we realized the rulebook was wrong. Our agreement arc was 92.9% → 92.9% → 94.3% → 100% across four runs, and not one fix was prompt-tuning. The last three disagreements traced to the published control matrix understating branches the engine enforced (contractor-expiry, remote own-device). We amended the rulebook itself, with a changelog and an automatic rules_version bump that ships inside every Evidence Package — the governance control ("rule changes are visible, never silent") demonstrated on real disagreements.
  2. Absence is invisible to a model. The hardest false-compliants were absence-of-artifact checks — a model can't see that an exit checklist doesn't exist. Deterministic packet builders now surface null fields, nil returns, and "searched, not found" document coverage explicitly.
  3. Prompt injection through the evidence itself. The dataset ships a poisoned contractor MSA. Defense is layered and structural: Verifier tripwire + quarantine at collection, document bodies sanitized before any model context, nonce-delimited untrusted data, and a regression test asserting the injected text never enters a copilot tool payload.
  4. Making "tamper-evident" survive an adversarial reviewer. Hash-chaining is easy to claim and easy to get wrong: we sealed the chain in a single writer pass after all retries, made Firestore a display copy (the zipped bytes are the truth), anchored the root outside our own writable store, and wrote a test that flips one verdict and proves the chain breaks.

Accomplishments that we're proud of

  • 100% LLM-vs-engine agreement on all 70 checks, four consecutive runs, zero flips on the stability pair — earned through deterministic fixes, with the confusion matrices published.
  • The verify-us-don't-trust-us loop closed over the wire: cloud run → sealed bundle → downloaded from the hosted workspace → verified PASS by stdlib Python on a laptop.
  • A platform, not a demo: new client system = one connector class + one YAML line; zero rule changes downstream.
  • The whole thing runs keyless — Vertex ADC in the cloud, no secrets in the repo, gitleaks-clean.

What we learned

  • Give the LLM judgment, never arithmetic or retrieval. Precomputed facts + published status_logic in the packet turned "plausible" into "correct".
  • Disagreement is signal, not failure — our best fixes came from treating every mismatch as a spec bug until proven otherwise. Twice, the spec was the bug.
  • Trust is an architecture, not a promise: verdict authority, citation validation, append-only decisions, and external anchoring each remove one "just trust us" from the pitch.

What's next

An Interface-Designer agent that drafts canonical mappings from a client's sample export (human-approved before registration — agents designing the adapters other agents operate through); delta runs that re-attest unchanged evidence deterministically (~80% cost reduction, keyed on the full judge-identity tuple); Firebase Auth + per-tenant rules + App Check for multi-tenant production; Go connector workers on the SPI's language-agnostic seams. Pricing follows the category: flat per-employee/per-framework/per-year, unlimited runs — because at $0.31 an audit, metering would be the only dishonest number in the product.

Gallery captions (screenshots, in order)

  1. Mission control, live — eight read-only collector identities light up; the Verifier quarantines a prompt-injection attempt in real time; the verdict ring climbs to 44/8/18.
  2. The run overview — verdict counts, the root hash, its GCS anchor, and the one-click Evidence Package download.
  3. Findings & evidence requests — three-state semantics with append-only auditor decisions that reference the sealed chain.
  4. Engine vs LLM — 100% agreement and an empty false-compliant column, published, not claimed.
  5. verify_bundle.py PASS — an auditor's laptop verifying every hash and the full chain with stdlib Python. No Proofline required.

Built With

Share this project:

Updates