Smart AI (Gemini) Expenses Analyzer

Analyze prices and offers with Gemini AI. The perfect expense tracker for students and families to save money and manage shared costs."

Comment

Project Story: Smart Expenses Tracker

Inspiration

Smart Expenses Tracker was born from a personal challenge: becoming a new parent.

Daily purchases—formula, diapers, groceries—started fluctuating in price, and receipts quickly became unmanageable. Existing expense apps relied on manual entry or opaque automation, neither of which inspired trust.

At its core, the problem was simple but hidden:

$$ \text{Savings} = \text{Income} - \sum (\text{Verified Spending}) $$

The difficulty wasn’t math—it was extracting verified spending from unstructured, inconsistent receipts. I wanted a system that could understand receipts, learn their structure, and keep humans in control.

About the Project

Smart Expenses Tracker is a production-ready Flutter + Firebase mobile application that converts real-world receipts into structured, verifiable financial data and long-term price intelligence.

The system is built around a human-in-the-loop AI model:

  • AI proposes extracted data
  • Users confirm or correct it
  • Only verified values are used for analytics

This makes the system accurate, explainable, and safe for personal finance.

How I Built It

Architecture Overview

  • Frontend: Flutter (feature-first modular architecture)
  • Backend: Firebase (Authentication, Firestore, Cloud Storage, Cloud Functions)
  • AI Stack:
    • Google Document AI – custom receipt processor
    • Gemini 3 – multimodal reasoning and enrichment

Document AI Training Pipeline (Custom Processor)

Training Objective

Generic OCR performs poorly on retail receipts due to inconsistent layouts, low print quality, mixed languages, and non-standard discount placement.
To address this, I created a custom Google Document AI processor optimized for retail receipts.

Dataset Preparation

Training data was composed of:

  1. Real-world receipts collected from everyday purchases
  2. Public receipt datasets sourced from Kaggle

All documents were anonymized and manually labeled using Document AI tooling.

Training Flow

Receipt Images (PNG / JPG / PDF)
        ↓
Preprocessing & Normalization
        ↓
Manual Annotation
        ↓
Custom Document AI Training
        ↓
Evaluation & Iteration
        ↓
Production Deployment

The processor outputs structured entities with confidence signals and bounding-box traceability.

Gemini Usage (Reasoning, Not Training)

Gemini was not trained or fine-tuned.

Gemini is used strictly as a post-OCR reasoning layer to:

  • Validate consistency (items ↔ totals ↔ discounts)
  • Resolve multi-image receipts
  • Flag uncertainty instead of guessing
  • Generate user-facing explanations

Gemini never overwrites user-confirmed data.

Gemini Personas (Task Isolation)

Receipt Analyst Persona

SYSTEM:
You extract receipt data conservatively.
Prefer null over guessing.
Never invent values.
Return JSON only.

Validation Persona

SYSTEM:
Verify numeric consistency.
Report mismatches clearly.
Do not auto-correct totals.

Explanation Persona

SYSTEM:
Explain results in simple language.
Do not mention AI or confidence scores.

Prompt Versioning Strategy

Prompts are treated as versioned code.

{
  "promptVersion": "receipt_analyzer_v1.3.0",
  "schemaVersion": "expense_v2",
  "breakingChange": false
}
  • Old receipts retain original prompt versions
  • New receipts use the latest stable prompt
  • Reprocessing is explicit and user-controlled

This ensures traceability and prevents silent regressions.

Data Model & Intelligence Layer

Firestore stores:

  • Field values
  • Source (user / OCR / AI)
  • Confidence metadata

Merchant normalization, fallback logic, and product fingerprints enable reliable long-term price analysis:

$$ \text{Price Trend}_{product} = \frac{\Delta \text{Price}}{\Delta \text{Time}} $$

What I Learned

  • Domain-specific OCR beats generic models
  • AI must assist, not decide
  • Explainability builds trust
  • Data modeling is foundational

Challenges

  • Receipt variability → solved via custom OCR + multi-image reasoning
  • Merchant inconsistency → solved via normalization
  • Automation vs trust → solved with human confirmation

Outcome

Smart Expenses Tracker is a verifiable financial intelligence system.

It demonstrates responsible AI usage:

  • Custom Document AI for extraction
  • Gemini for reasoning
  • Humans for final authority

This project reflects production-grade AI system design: transparent, safe, and accountable.

Built With

Share this project:

Updates