ClaimFlow AI — Hackathon Submission Write-up
Inspiration
Many document-heavy businesses still receive critical information through fragmented PDFs, scanned forms, mobile photos, invoices, reports, notes, and other inconsistent documents. Staff must manually interpret these materials, copy information into case-management systems, compare conflicting details, and contact people when important information is missing or contradictory.
We started with a simple question:
Can agentic AI turn messy evidence into a trustworthy, review-ready case without hiding uncertainty or removing human control?
That question became ClaimFlow AI.
Rather than building an AI system that silently makes consequential decisions, we wanted to build one that can understand documents, identify uncertainty, preserve the evidence behind every important field, escalate contradictions, and help a human reviewer resolve them.
The working prototype was built in approximately two days during the hackathon.
What it does
ClaimFlow AI is an evidence-first agentic document workflow for claims, restoration, field-service, and other document-heavy business operations.
A user creates a synthetic case and uploads source documents. ClaimFlow then runs them through a controlled multi-agent workflow that:
ingests PDF, JPEG, and PNG source documents;
performs document intake and quality assessment;
uses Gemini multimodal understanding to extract structured fields;
attaches confidence and page-linked source evidence to extracted information;
validates dates, required fields, formats, and other business rules;
detects missing information, low-confidence values, and contradictions;
routes uncertain results to a human review workspace;
preserves the original source evidence while allowing human corrections;
records agent activity and human actions in an auditable timeline.
A particularly important feature is voice clarification.
When documents contain conflicting information that cannot safely be resolved from the evidence alone, a reviewer can create and approve an exact clarification question. ClaimFlow then uses an ElevenLabs conversational voice agent together with Twilio to place a real outbound clarification call to the configured synthetic test participant.
After the conversation, ElevenLabs sends a signed post-call transcript back to ClaimFlow through a webhook. The transcript becomes additional evidence, but it does not automatically change the case. A human reviewer must still enter the final canonical value and provide a reason.
ClaimFlow does not autonomously approve or deny insurance claims. Consequential decisions remain under human control.
How we built it
ClaimFlow is implemented as a TypeScript-based web application deployed on Google Cloud.
The frontend uses:
React
TypeScript
Vite
Material UI
The backend uses:
Node.js
TypeScript
Fastify
Zod for runtime contracts and validation
The agentic workflow uses the Google Agent Development Kit, or ADK, with six controlled stages:
INTAKE → QUALITY → EXTRACTION → VALIDATION → CASE_PLANNER → REVIEW_ROUTER
For multimodal document understanding, ClaimFlow uses Gemini 3.5 Flash on Vertex AI.
We deliberately did not build an unrestricted swarm of autonomous agents. The workflow is orchestrated through typed stages with deterministic boundaries around validation, persistence, permissions, and human review.
The Google Cloud architecture includes:
Vertex AI for Gemini;
Cloud Run for the deployed application;
Firestore for case, workflow, clarification, and audit state;
Cloud Storage for original uploaded documents;
Secret Manager for protected integration credentials;
GitHub Actions with Google OIDC / Workload Identity Federation for keyless CI/CD deployment.
For external clarification, we integrated:
ElevenLabs Conversational AI
Twilio outbound calling
a signed webhook for returning post-call transcripts.
The voice workflow is intentionally approval-gated:
Draft → Human approval → Call → Transcript → Human correction → Resolved
If optional AI-generated clarification wording fails, ClaimFlow falls back to a deterministic editable question instead of blocking the entire workflow.
Challenges we ran into
Building an end-to-end system in roughly 48 hours exposed several practical challenges beyond simply calling an AI model.
One challenge was structured multimodal extraction. We needed Gemini to return useful structured information while preserving evidence references and contradictions rather than smoothing them into one convenient answer. We combined model extraction with Zod validation and deterministic TypeScript rules to enforce the boundaries.
Another challenge was conflicting evidence. Our synthetic five-page motor-claim packet intentionally contains contradictory dates and damage estimates. ClaimFlow needed to preserve both source values, identify the contradiction, and avoid silently choosing one.
We also encountered a real production issue in the clarification workflow. The first Gemini-based clarification-drafting request had a timeout that was too short for the live Vertex AI response and produced a 502. We redesigned that step so it uses the configured AI timeout and falls back safely to a deterministic question if the optional AI wording is unavailable.
The voice integration introduced additional problems:
safely initiating only human-approved calls;
restricting calls to a configured test participant;
handling Twilio trial behaviour;
verifying ElevenLabs webhook signatures;
associating returned transcripts with the correct persisted conversation;
preventing duplicate or ambiguous retries;
keeping transcripts as evidence rather than automatic truth.
Deployment also required care. We needed Cloud Run deployments to preserve Secret Manager references while keeping CI/CD keyless through GitHub OIDC.
These challenges reinforced an important lesson: reliable agentic systems require much more than prompts. They need explicit state machines, schemas, deterministic validation, permission boundaries, observability, and failure handling.
Accomplishments that we're proud of
In approximately two days, we moved ClaimFlow from an architecture and business problem to a working, deployed, end-to-end agentic AI system.
We are particularly proud that we:
deployed a working application on Google Cloud Run;
built a six-stage Google ADK multi-agent workflow;
integrated Gemini 3.5 Flash on Vertex AI for multimodal extraction;
preserved page-linked source evidence for extracted fields;
implemented deterministic validation alongside AI reasoning;
detected deliberate contradictions in a multi-page synthetic claim;
built a modern human-review workspace;
persisted case state and audit history in Firestore;
stored original documents in Cloud Storage;
created a safe clarification workflow that requires human approval;
integrated a real ElevenLabs conversational AI agent;
connected ElevenLabs to Twilio for an actual outbound phone call;
received the post-call transcript back through a signed webhook;
treated that transcript as evidence rather than an automatic decision;
required a human reviewer to save the final canonical correction;
verified the complete flow in production from document conflict to RESOLVED;
implemented automated tests, container smoke tests, Firestore integration tests, and automated Cloud Run deployment.
Our live acceptance test demonstrated this complete sequence:
Document evidence → contradiction detected → human-approved clarification → real voice call → transcript returned → human canonical correction → resolved field → audit trail
That end-to-end loop is the part of ClaimFlow we are most proud of.
What we learned
The biggest lesson was that agentic AI does not mean making every component autonomous.
Models are powerful when the problem involves interpretation, ambiguity, multimodal evidence, or natural-language interaction. But they should not necessarily control every state transition.
In ClaimFlow:
Gemini interprets evidence;
ADK coordinates specialized stages;
ElevenLabs conducts a controlled conversation;
deterministic code validates important constraints;
schemas protect data boundaries;
persisted state makes workflows recoverable;
signed webhooks protect integration boundaries;
and humans retain authority over consequential corrections.
We also learned that preserving uncertainty can be more valuable than maximizing apparent model confidence.
When two source documents disagree, a trustworthy system should be able to say:
“These sources conflict, and I need help resolving them.”
That behaviour is more useful for real operational workflows than an AI system that confidently invents certainty.
Finally, the two-day build showed us that combining multiple AI and cloud services is achievable quickly, but making them reliable, auditable, and safe together is where most of the engineering work happens.
What's next for ClaimFlow AI
The hackathon prototype focuses on one complete, production-deployed vertical slice:
Documents → multimodal understanding → agent workflow → evidence-linked extraction → validation → human review → voice clarification → human resolution
The next step is to expand ClaimFlow from this vertical slice into a broader document-operations platform.
Planned improvements include:
direct email and attachment ingestion;
additional document formats and higher-quality OCR/handwriting evaluation;
SMS and additional communication channels;
configurable schemas for different industries;
role-based authentication and permissions;
organisation-specific validation and escalation policies;
CRM and existing case-management integrations;
multilingual document and voice workflows;
richer analytics and operational dashboards;
larger synthetic and real-world evaluation datasets;
stronger privacy, retention, and compliance controls;
improved observability and cost monitoring;
configurable agent workflows for different business processes.
We also want to explore how ClaimFlow can generalize beyond insurance-related workflows to restoration, field services, healthcare administration, logistics, property management, legal operations, and other businesses where critical decisions depend on fragmented evidence.
Our longer-term goal is not to replace the people responsible for these decisions.
It is to give them better evidence, less manual reconciliation, faster clarification, and a clearer audit trail.
Log in or sign up for Devpost to join the conversation.