GroundTruth

• 

• 

• 

• 

• 

• 

• 

Inspiration

AI made shipping faster. It did not automatically make organizations wiser.

Software teams can now generate more code, more changes, and more experiments than ever. But when evidence, review, ownership, and institutional memory fail to keep pace, velocity compounds while value does not.

Google Cloud's 2024 DORA research makes that tension concrete: more than 75% of respondents used AI in daily work, yet higher AI adoption was associated with lower delivery throughput and stability at the system level. AI is an amplifier—but an organization still needs a mechanism that turns every failure into better decisions everywhere.

The most expensive failure is often not the first bug. It is the same causal mistake being rediscovered by another team, in another product, six months later.

GroundTruth starts with one conviction:

One team makes a mistake. The whole organization learns. The next recurrence is prevented. Value compounds.

What it does

GroundTruth is an institutional learning control plane for AI-native engineering. It turns a real failure into a closed, auditable value loop:

1. Reconstruct evidence and intent instead of accepting a shallow issue summary.
2. Derive a reusable causal signature from the failure mechanism.
3. Search proactively across structurally related components and products.
4. Design a falsifier and safe control before trusting a conclusion.
5. Let deterministic evaluators—not the model—verify decisive claims.
6. Preserve the verified lesson in an append-only SHA-256 chain.
7. Attach preventive controls to future changes.
8. Build shared human capability without employee ranking or blame.

The result is not another chatbot or code scanner. It is a multi-page organizational workflow spanning:

• consequence-routed changes;
• an adversarial assurance workspace;
• institutional memory;
• incidents and learning;
• people and capability; and
• verified customer and business value.

The flagship proof: a blind Kubernetes replay

We built a blind historical replay around real Kubernetes issue #29297.

• Allowed evidence: the issue and bounded source from pinned pre-fix commit d7150bfaeae642efc08c8ede0ed2ec8ecb340c8e.
• Withheld evidence: the eventual fixing PR, maintainer explanation, and four-file patch scope.
• Agent task: determine the mechanism, prove it, and search for the same fault elsewhere.
• Independent answer key: Kubernetes PR #29641, revealed only after the findings are frozen.

Five Gemini agents independently reconstruct the shared-nested-pointer concurrency mechanism and expand one reported ConfigMap failure to three sibling exposures:

• Secret
• Downward API
• GitRepo

Trusted code then runs two decisive checks:

1. Structural signature scan: finds all four exposed plugin paths.
2. Deterministic interleaving: Mount A writes its wrapper name, Mount B overwrites the shared nested object, and Mount A reads B's name; a fresh-object control passes.

When the withheld Kubernetes patch is finally revealed, GroundTruth's four discovered paths exactly match the four historical fix paths: 100% precision and 100% recall.

This is a retrospective blind benchmark—not a claim that we discovered a previously unknown Kubernetes defect. The later human patch is an independent external answer key.

Architecture

The system deliberately separates reasoning from authority:

• Gemini 3.5 Flash on Vertex AI performs bounded causal reasoning.
• Google ADK SequentialAgent orchestrates five structured LlmAgent specialists: Evidence Investigator, Causal Analyst, Pattern Scout, Adversary, and Learning Architect.
• Google GenAI SDK provides the model integration.
• Google Cloud Functions / Cloud Run host the FastAPI workflow.
• Firestore Native persists assurance runs and evidence-linked results.
• Pub/Sub provides an asynchronous incident/evidence trigger.
• Firebase Hosting provides the stable public entry point.
• Deterministic Python evaluators own structural matching, the aliasing reproducer, answer-key comparison, and ledger verification.

The production evidence run assure_ef7e404586 completed in vertex-adk:gemini-3.5-flash mode with:

• five structured agent outputs;
• three trusted proof checks;
• an exact 4/4 scope match; and
• a valid four-event learning chain.

Challenges

The hardest challenge was preserving a trustworthy boundary between AI reasoning and proof. We:

• separated the allowed evidence packet from the historical answer key in code and tests;
• prevented the model from writing its own verification result;
• assigned every decisive claim to deterministic evaluators; and
• adapted around a Google edge-routing anomaly by serving the same FastAPI application through a verified Google Cloud Function while preserving the containerized Cloud Run deployment.

Accomplishments

• A real five-agent ADK workflow, not a simulated trace.
• An externally verifiable blind benchmark with an exact four-path answer-key match.
• A deterministic vulnerable interleaving plus a passing safe control.
• A real append-only hash chain with no public update/delete operation.
• A persisted production Vertex/ADK run and public multi-page platform.
• 63 passing tests, clean lint, reproducible builds, and zero browser-console errors.
• A human-development model based on capability and shared ownership rather than surveillance.

What we learned

• AI is best treated as an amplifier.
• Local generation speed creates value only when evidence, feedback, memory, verification, and ownership improve with it.
• Agent reasoning becomes credible when uncertainty is visible and independent proof has authority.
• Verified knowledge becomes economically valuable only when it changes the next decision—not when it merely closes the previous incident.

What's next

1. Organization-scale causal retrieval across repositories, incidents, CI, and runtime telemetry.
2. Policy-as-code controls generated from verified lessons.
3. Cross-team learning propagation before recurrence.
4. Correction and supersession events in a durable ledger.
5. Longitudinal measurement connecting prevented failures to customer and business outcomes.

Built With

• artifact-registry
• cloud-build
• cloud-run
• fastapi
• firestore
• gemini-3.5-flash
• google-adk
• google-genai-sdk
• pub-sub
• python
• vertex-ai