ClaimSherlock - Enterprise Fraud Detection

ClaimSherlock slashes investigative effort by 90%. Gemini 3 Vision finds physics gaps in seconds; Live API interrogates instantly. We turn weeks of manual review into minutes of AI automation.

Comment

Inspiration

Insurance fraud costs the global economy over $300 billion annually, but the most hidden cost is time. Specialized forensic investigators currently spend weeks manually cross-referencing metadata, analyzing impact physics, and conducting interviews. We were inspired to turn this "weeks-long" manual process into a "minutes-long" AI-orchestrated workflow. We wanted to build the "God View" for insurance investigators—a tool that doesn't just look at photos, but interrogates the reality behind them to find the truth.

What it does

ClaimSherlock is an end-to-end Forensic Intelligence Platform that automates the three pillars of investigation:

  1. Visual Forensics: Uses Gemini 3 Pro to analyze evidence photos for metadata integrity (EXIF/GPS), pre-existing damage (rust/oxidation detection), and photo manipulation.
  2. Logic Reconstruction: Uses Gemini 3 Flash Preview to build a 3-way Evidence Chain (Timeline, Causation, and Physical patterns) to identify logical gaps in a claim.
  3. Live Interrogation: Conducts a real-time, low-latency audio interview using the Gemini Live API. The AI acts as a firm investigator, using the forensic findings as "traps" to catch contradictions in the claimant's story.
  4. Verdict Generation: Produces a legally-formatted Forensic Report with a final Integrity Score and specific recommendations for approval or SIU (Special Investigations Unit) referral.

How we built it

  • The Orchestrator: A custom React 19 frontend that manages a complex state machine transitioning from data ingestion to forensic analysis to live interaction.
  • The Vision Engine: Integrated Gemini 3 Pro to process up to 10 high-resolution images simultaneously, utilizing spatial reasoning for bounding-box anomaly detection.
  • The Reasoning Layer: Leveraged Gemini 3’s superior logic to synthesize visual data into structured hypotheses and "interrogation briefs."
  • The Live Interface: Implemented a high-performance audio pipeline using AudioWorklets to stream raw 16-bit PCM audio to Gemini 2.5 Flash Native Audio. We utilized Function Calling to allow the AI to "lock in" confirmed details and capture visual evidence from the camera feed during the call.
  • Aesthetics: A "Cyber-Forensic HUD" designed with Tailwind CSS, featuring a real-time system log and an interactive evidence chain visualizer.

Challenges we ran into

  • The Sync Problem: Synchronizing real-time video frames with high-speed audio streaming in the Live API required a custom capture loop that carefully balanced frame rate with tool-processing latency.
  • Context Persistence: Moving data from a "Vision analysis" to a "Voice conversation" without losing the nuance of the evidence. We solved this by generating a "Forensic Brief" that acts as the model's memory.
  • Audio Interruption Handling: In a real interrogation, claimants talk over the investigator. We had to implement sophisticated "Silence Detection" and "Fade-out" logic to ensure the AI responds naturally to being interrupted.

Accomplishments that we're proud of

  • 90% Efficiency Gain: We successfully automated a workflow that typically takes an investigator 15+ hours of manual work.
  • The "Evidence Chain" Logic: Building a system that can actually reason about physics—detecting that a front-end impact shouldn't cause rust or that the sun angle in a photo contradicts the claimed time of day.
  • Zero-Latency Interaction: Achieving a human-like conversational speed with the Live API that makes the "Interrogation" feel truly immersive.
  • Biometric Stress Analysis: Using the video feed to monitor subtle facial micro-expressions or vocal tremors as secondary fraud indicators.

What we learned

  • Multimodal Orchestration: The true power of Gemini isn't just "chatting"; it’s the ability to pass the baton between Vision, Logic, and Voice models.
  • PCM Forensics: We gained deep knowledge of the Web Audio API and how to handle raw PCM streams for high-stakes enterprise applications.
  • Interrogation Psychology: We learned how to prompt AI to be "skeptical" and "firm" while remaining professional, a key requirement for fraud detection.

What's next for ClaimSherlock - Enterprise Fraud Detection

  • Deep-History Integration: Connecting to DMV and Police databases via Function Calling to verify vehicle accident history in real-time.
  • Multi-Vehicle Modeling: Expanding the forensic physics engine to analyze collisions involving 3 or more vehicles simultaneously.

Built With

Share this project:

Updates