BillSenseAI

BillSense is an AI RAG based multi agent system that detects invoice anomalies and automatically resolves them. It deploys Billy, an AI voice agent, to call vendors and negotiate discrepancies.

Comment

BillSense 🧾🤖

Autonomous Invoice Intelligence & Resolution

BillSense is an AI powered agentic system that detects quote vs. invoice anomalies and automatically resolves them. Powered by a RAG-based multi-agent architecture, BillSense doesn’t just flag billing errors — it deploys Billy, an AI voice agent, to call vendors and negotiate discrepancies in real time.

Inspiration

We were inspired by a simple but costly reality: most billing errors never get caught. Organizations rely on manual invoice reviews, so only the biggest, most obvious mistakes get flagged — everything else “survives” and gets paid. That survivorship bias in revenue auditing leads to silent financial leakage across logistics, moving, construction, and SaaS. We wanted to build something that doesn’t just detect quote vs. invoice anomalies, but takes action: an AI that can actually call vendors, reference the right documents, and negotiate corrections. BillSense (and Billy, the voice agent) is our answer — automation that goes from “something’s wrong” to “we’re fixing it” in one flow.

🚨 The Problem

Addressing the Survivorship bias in Revenue Auditing

Invoice overbilling, hidden fees, and scope creep are widespread across industries such as logistics, moving services, construction, and SaaS procurement.

Most organizations rely on manual invoice reviews, which leads to:

  • Missed discrepancies due to time and attention limits
  • Focus only on large, obvious errors
  • Silent financial leakage from small but recurring overcharges

This creates a survivorship bias in invoice auditing:

Only the most obvious billing errors are caught — the rest survive and get paid.

Over time, these unnoticed discrepancies compound into significant revenue loss.

💡 The Solution

BillSense is an end-to-end invoice intelligence and resolution system. Users connect their Google workspace to pull quotes and invoices from their Gmail. The system:

  • Indexes and understands documents — RAG over a vector store (FAISS) with structured extraction (totals, line items, vendor, dates) so every decision is grounded in your actual paperwork.
  • Chats with you — A multi-agent graph (LangGraph) coordinates an info/summary agent, chat agent, call agent, action agent, and suggestion agent. You can ask questions, request comparisons, or ask for a call.
  • Calls vendors as Billy — When you ask to call a vendor, BillSense generates a call script from the documents and places a real phone call via Twilio. Billy (powered by Gemini Live) speaks to the vendor, references specific amounts and line items, negotiates or clarifies, and can end the call when done.
  • Summarizes and acts — After the call, a summary is produced and optional follow-up actions (e.g. “draft a letter”) are extracted and passed to the action agent.

BillSense systematically reviews 100% of invoices, identifies both obvious and subtle anomalies, and takes autonomous action to resolve them.

🧠 System Architecture

1. Knowledge Indexing & Vector Database

All company documents are embedded and stored in a vector database:

  • Quotes and invoices
  • Contracts and pricing policies
  • Historical vendor communications
  • Past dispute resolutions

This enables fast semantic retrieval across unstructured data.

2. RAG (Retrieval-Augmented Generation) Pipeline

Before any reasoning or action:

  • Relevant documents are retrieved from the vector database
  • Provides agents with complete historical and contractual context
  • Ensures decisions are grounded and explainable

3. Anomaly Detection Engine

  • LLM-based semantic matching between invoices and quotes
  • Detects:
    • Unauthorized charges
    • Price inflation beyond agreed thresholds
    • Quantity or scope mismatches
    • Duplicate or hidden fees
  • Every anomaly is linked to source documents

4. Multi-Agent Resolution System

BillSense uses a coordinated multi-agent architecture (LangGraph):

  • Info Summary Agent – Extracts and summarizes document content (invoices, quotes) into structured context
  • Chat Agent – Handles conversation, answers from documents, and triggers voice calls or follow-up actions when the user asks
  • Billy (Call Agent) – Orchestrates outbound calls via Twilio, runs Billy’s voice (Gemini Live), and summarizes the call transcript
  • Action Agent – Executes follow-up tasks (e.g. draft a letter) derived from the call or user request
  • Suggestion Agent – Suggests next steps and responses based on current state

Agents communicate via a shared state graph with conditional edges for iterative reasoning and escalation.

📈 Business Impact

💰 Revenue Recovery

  • Recovers overcharges that typically go unnoticed
  • Converts previously unrecoverable losses into direct savings

🧠 Eliminates Survivorship Bias

  • Reviews every invoice consistently
  • Catches subtle discrepancies humans overlook

🕒 Operational Efficiency

  • Eliminates manual invoice audits and follow-ups
  • Operates 24/7 across vendors and time zones

🔍 Transparency & Trust

  • Every decision is explainable and auditable
  • Builds stronger vendor accountability

Even small discrepancies across high-volume invoices can translate into hundreds of thousands to millions of dollars annually.

How we built it

  • Backend (Python)

    • FastAPI for REST and WebSockets (agent state streaming, voice media).
    • LangGraph for the multi-agent workflow: state graph with conditional edges (e.g. chat → call when user requests a call, call → action when there’s an action request).
    • LangChain for prompts, tool binding (voice_call, action), and LLM calls (Gemini).
    • RAG pipeline: document_rag with a DocumentParser (LLM-based extraction into a canonical schema), VectorStoreManager with FAISS and Gemini embeddings, and optional Gmail integration (OAuth, fetch emails/attachments, run extraction on attachments).
    • Voice: Twilio for outbound calls and Media Streams (WebSocket). We built a voice_ai layer that bridges Twilio’s audio to Gemini Live API (GeminiVoice), with support for real-time streaming, tool use (e.g. end call), and barge-in handling.
    • Agents: InfoSummaryAgent (document extraction/summary), ChatAgent (conversation + voice_call/action tools), CallAgent (orchestrates VoiceCall, summarizes transcript, derives action_request), ActionAgent (follow-up tasks), SuggestionAgent (suggestions).

  • Frontend

    • React Native with Expo for cross-platform (iOS, Android, web).
    • Firebase for client hosting.
    • React Query and React Navigation for data and screens.

  • Infra / APIs

    • Environment config for Twilio (account, auth, from number), Gemini (API key, model), and a public WS domain for Twilio callbacks and media.

Challenges we ran into

  • Voice pipeline: Twilio sends/receives μ-law 8 kHz; Gemini Live expects different formats and sample rates. We had to implement conversion (e.g. PCM ↔ μ-law, sample rate handling) and ensure frames stayed in sync so Billy’s speech and vendor audio didn’t drift or glitch.
  • When to call vs. suggest: The chat agent had to be carefully prompted and tool-bound so it only triggers a voice_call when the user clearly asks to call someone, and to ask for the phone number (and confirm it’s their number in a demo). Avoiding accidental or premature outbound calls was important.
  • Multi-agent state: Deciding which node runs next (chat → call vs. action vs. suggestion) based on tool calls, call_script, phone, and action_request required clear state shapes and conditional edge logic in LangGraph so the graph didn’t get stuck or skip steps.
  • Document extraction consistency: Invoices and quotes use different wording (e.g. “total amount” vs “total”). We introduced canonical key aliases and normalization so RAG and agents see a single schema (e.g. total, line_items) across all docs.
  • Real-time behavior: Handling user barge-in (caller talks over Billy) and cleaning playback on Twilio’s side so the conversation feels natural required mark/clear handling and coordination with Gemini’s streaming.

Accomplishments that we're proud of

  • End-to-end flow: From Gmail ingestion → RAG and structured extraction → chat → one-click outbound call with Billy on a real phone number, then call summary and action extraction. No handoff to a separate tool; it’s one product.
  • Real voice AI in production: Twilio + Gemini Live working together with our own voice_ai bridge, tool use (e.g. end call), and streaming so Billy can have a natural, document-aware conversation.
  • Multi-agent design: A single LangGraph application with multiple specialized agents (info, chat, call, action, suggestion) and conditional routing so we can add more behaviors (e.g. verification agent) without rewriting the whole pipeline.
  • Document-grounded negotiation: Billy’s call script and context come from the same RAG and extraction pipeline that powers chat, so the agent really “knows” the invoices and quotes it’s discussing.
  • Cross-platform frontend: One React Native/Expo app for web and mobile with auth, dashboard, and voice call UX so we can demo the full experience on any device.

What we learned

  • LangGraph is powerful for multi-agent apps: state graphs and conditional edges make it clear when control moves from chat to call to action, and the checkpointer gives us a natural place to persist and resume sessions.
  • Real-time voice AI is more than “LLM + TTS/STT”: telephony codecs, sample rates, and Twilio’s Media Streams required a dedicated bridge layer; Gemini Live’s native audio API simplified our stack once we got the format right.
  • Survivorship bias in auditing is a strong product angle — framing the product around “the errors that survive” resonated and kept our scope focused on end-to-end resolution, not just dashboards.
  • RAG and structured extraction together are essential for negotiation: retrieval finds the right docs, and a consistent schema (with key normalization) lets the model and Billy reference the same numbers and terms reliably.
  • Prompting and tool design matter a lot for safety and UX: making the chat agent conservative about when it calls (and always confirming the number in a demo) prevented misuse and made the demo predictable and professional.

Built With

Share this project:

Updates

Submission history