Claimaroo

Voice-first AI claims employee for Australian motor insurance: captures accidents by voice, retrieves policies, analyses photos, checks coverage, triages claims, and delivers decision-ready cases.

Comment

Inspiration

Insurance claims start as a messy conversation. Someone has just been in an accident. They do not have policy numbers ready, they repeat the same story across a form, a phone queue, and a follow-up email, and a claims officer later reconstructs the case from a transcript and a pile of photos.

We did not want another chatbot that collects fields. We wanted an AI employee: something that listens, retrieves the policy, persists a real claim, reads the photos, runs the checks an intake team would run, and hands a person a decision-ready case.

Voice is the right interface after a crash. The customer should speak naturally. The system should do the structure.

Australia is already moving this way — Allianz’s agentic spoilage work, IAG’s voice experiments — but motor liability is harder than a $500 fridge claim. That is why Claimaroo stops at a prepared case. It never settles or pays.

What it does

Claimaroo is a voice-first motor claims agent for a synthetic Australian book.

A policyholder opens /claim, adds damage photos, and talks. The agent:

  1. Identifies itself as AI and checks safety first
  2. Looks up the customer and policy — and refuses to invent either
  3. Creates a real claims row in Postgres, not a transcript dump
  4. Analyses photos (local damage model first, then cloud vision to locate findings on the vehicle, then a labelled filename heuristic if both miss)
  5. Runs deterministic coverage and triage rules
  6. Hands an officer a case at /claims/[id]

Three seeded paths show the operating model, not just the happy path:

  • Maya Chen (CLM-DEMO-A) — clear rear-end, complete photos → decision_ready
  • Liam O'Brien (CLM-DEMO-B) — conflicting accounts stay visible → human_review
  • Priya Nair (CLM-DEMO-C) — injury or immediate danger → urgent, always

The officer sees incident facts, policy coverage, evidence, a damage map, risk flags, a recommended next step, and a full audit. Only an officer click on approve_next_stage can move the claim forward. The conversation agent has no tool that can do that. Every dollar figure is labelled preliminary.

How we built it

One Next.js 16 app. One Postgres 16 database. No extra workers.

Layer What we used
Voice ElevenLabs Conversational AI in the browser (@elevenlabs/react). Conversation LLM: Gemini 2.0 Flash. TTS: eleven_flash_v2.
Tools 11 client tools. ElevenLabs tells the tab to call a tool; the page POSTs /api/tools/{name}. That is why a live demo works on localhost with no public webhook.
Data Drizzle + postgres.js. Customers, policies, claims, evidence (bytea — Render’s disk is ephemeral), assessments, audit events.
Vision Local detector (LOCAL_VISION_URL, YOLOv8 trained offline) first. Cloud vision only locates those boxes on the car. If local misses every file, OpenAI-compatible or Anthropic vision runs. Otherwise a heuristic. Failures never throw.
Rules TypeScript, not the LLM: coverage.ts, triage.ts, estimate.ts. Injury or danger hard-codes urgent. Unclear or not-covered goes to human_review — there is no auto-decline.
Officer UI Server components at /claims and /claims/[id].
Tests Vitest for rules and APIs, Playwright for pages, GitHub Actions on Node 22.

Guardrails live in code, not only in the prompt: no invented customers, no invented policies, a full audit row on every tool call, and a human gate on approval.

Challenges we ran into

Keeping the LLM out of decisions that matter. The conversation model is good at asking the next question. It is a bad place to decide coverage, urgency, or approval. We spent more time putting those rules in TypeScript than making the voice sound clever.

Vision that is honest about missing. A local YOLO pass on 822 validation images sits around 0.50 mAP50 and 53% recall. A miss is not proof that the panel is fine. We never treat a missed detection as “no damage,” and we surface confidence and limitations on the case.

Ephemeral hosting. Photos cannot live on the container disk. They go into Postgres as bytea. That is inelegant and it is correct for Render/Vercel.

Microphone reality. getUserMedia needs localhost or HTTPS. A LAN IP demo dies silently. Client tools solved the webhook problem; they did not solve phones.

Demo reliability vs live APIs. Voice and vision keys are optional. Without them, tools, seed data, and the officer workspace still run. The live call is the showpiece; the product cannot depend on it to exist.

Accomplishments that we're proud of

  • A working loop from spoken accident → persisted claim → analysed photos → routed case → officer workspace, not a slide deck with a chatbot overlay.
  • Bounded autonomy: the agent does the repetitive work; a human still owns the consequential click.
  • Uncertainty is a first-class outcome. Conflicting accounts stay on the file instead of being smoothed into a confident summary.
  • Safety is a hard rule. Priya’s injury path cannot be talked out of urgent.
  • A local damage model in the live workflow, with cloud vision used as a locator rather than an oracle.
  • Tests that lock the boring-but-important contracts: unknown customer → found: false; injury → urgent; only approve_next_stage sets that status.

What we learned

Voice is the interface. The product is the state change.

If the conversation does not write a real claim, retrieve a real policy, and leave an audit trail, you have a demo of speech-to-text — not an AI employee.

We also learned where not to use a model. Coverage, triage, and approval are cheaper and safer as functions. Models are for language, tool choice, and looking at a photo. Mixing those jobs is how prototypes become confidently wrong.

Finally: label everything preliminary, use synthetic people, and say the system does not pay claims. That is not caution theatre. It is how you keep a 48-hour insurance demo from making a claim it cannot stand behind.

What's next for Claimaroo

Near term: more claim types, configurable rules, stronger document intake, and a real identity/policy integration instead of a seeded book.

Next: repairer estimates that are still non-binding, multilingual voice, and claim-status follow-up across channels.

Later, and only with monitoring and a carrier’s thresholds: a narrow auto-path for clean, low-severity, fully evidenced collisions — still with a human gate on anything adverse.

Claimaroo is a prototype. The direction is an operating layer between a stressed conversation and a claims professional’s decision, not a machine that settles the file.

Built With

Share this project:

Updates

Submission history