Fraud Check

• 

  Onboarding screen: Explains users what this app is and its functions

• 

  Main Home screen : users can click on quick cards or talk by pressing the microphone. Live session will open camera.

• 

  Violation cards citations : On conversing with users, the AI will provide every violation that users can report about on his experience

• 

  Live Session : Showing violations and problem real time with real live camera capture and conversation.

• 

  Dual Language Support : Users can talk in english and nepali or both to have conversation. The ai will recognize and talk naturally.

• 

  Send Email action : The app lets users send formal compain to official channel of the government who see such violations.

• 

  System Architecure

Inspiration

In Nepal, consumer fraud is common to hear. Overpriced goods, expired products, fake ads, undelivered orders but most people don't know their legal rights and do not take any action. The Consumer Protection Act 2075 exists, but it sits in documents nobody reads. Fraud Check makes people aware and provided explains actual lawful rights as easy as talking to a friend.

What it does

Fraud Check is a real-time voice agent that lets you simply talk about your consumer problem and get instant help:

• Voice conversation: Speak in Nepali, English, or both. The agent listens, asks clarifying questions, and identifies which sections of Nepal's Consumer Protection Act 2075 or E-Commerce Directive 2082 are violated.
• Live camera analysis : Point your phone at a defective product, expired item, or receipt. Gemini analyzes continuous camera frames at ~1 in real-time
• Violation detection: Structured cards surface the exact law, section, and recommended next steps as the agent speaks
• One-tap complaint emails: The agent drafts a formal complaint and sends it directly via Gmail API with photo evidence attached
• The entire experience is conversational and interruptible — like a knowledgeable friend who happens to know consumer law.

How we built it

Gemini Live API (core): The agent runs on Gemini 2.5 Flash Native Audio via the Live API with bidirectional WebSocket streaming. It speaks back with the Zephyr voice, transcribes both input and output in real-time, and processes live camera frames in a single persistent session.

Transcript-based structured extraction: Since native audio models don't support function calling, we built a server-side layer that parses the agent's spoken transcripts to detect legal citations (violation cards) and email drafts (structured "DRAFTING EMAIL...END OF DRAFT" phrases) , turning unstructured speech into actionable UI elements.

Stack: React 18 + Vite PWA frontend, Node.js + Hono + TypeScript backend, Gmail API for email sending, deployed on Google Cloud Run.

Key technical details: *AudioWorklet-based mic streaming at 16kHz PCM *Live camera frames sent as JPEG at ~1 fps to Gemini

• Anti-hallucination guardrails: temperature 0.3, exhaustive legal section allowlist, strict system instructions preventing fabricated citations *Session persistence via localStorage for transcripts, violations, and evidence

Challenges we ran into

*No function calling with native audio: Gemini's native audio model doesn't support tool use, so we engineered transcript parsing for violation detection and email extraction *Bilingual voice agent : Getting natural Nepali/English mirroring while keeping legal terms accurate required careful prompt design *Anti-hallucination for legal AI : One fabricated law section could mislead someone, so we built an exhaustive allowlist of valid citations and strict "say I don't know" fallback behavior *Mobile audio edge cases : AudioWorklet mic input, PCM playback, interruption handling, and WebSocket reconnection across browsers

Accomplishments that we're proud of

• A consumer can go from "I got scammed" to a formal complaint email sent with evidence just by talking in under 2 minutes
• The agent never hallucates legal citations , only references validated sections
• Live camera + voice working simultaneously in a single Gemini session
• Works as a PWA on any smartphone, no app install needed

What we learned

• Voice-first UX is fundamentally different : Everything must be interruptible and forgiving of noise.
• Legal AI needs extreme guardrails: Gemini is capable but must be constrained to prevent harmful misinformation The Live API's native audio + real-time transcription enables experiences that feel genuinely conversational, not transactional

What's next for Fraud Check

• Expand to other laws and rights, expand to other countries consumer laws.
• Community-powered fraud map from anonymized violation data
• Offline legal reference caching for low-connectivity areas

Built With

• gemini-2.5-flash-native-audio
• gemini-live-api
• gmail-api
• google-cloud-run
• hono
• node.js
• react
• typescript
• vite
• websockets