Milo

Milo - Your AI-powered financial sidekick that tracks expenses, manages budgets, and coaches your spending through natural conversation, voice notes, receipt scanning, and WhatsApp guided by squirrel.

Comment

Inspiration

Managing personal finances shouldn't feel like a chore. Most budgeting apps are either too complex or too boring to stick with. We wanted to build something that feels like texting a friend who happens to be great with money. The idea was simple: what if you could just tell someone "spent 15 bucks on lunch" and it would all be tracked for you? That's how Milo was born — a friendly squirrel who lives in your browser (and your WhatsApp) and actually makes you want to manage your money.

What it does

Milo is a full-stack personal finance app with an AI-powered squirrel assistant at its core.

  • Natural language expense tracking — Tell Milo "paid 50 for groceries yesterday" and it logs the transaction with the right category, amount, and date automatically.
  • Receipt scanning — Snap a photo of a receipt and Milo extracts the transaction details using Gemini's vision capabilities.
  • Voice notes — Record a quick audio clip describing your purchase and Milo transcribes and logs it.
  • Budget management — Set monthly limits per category with visual progress bars. Milo alerts you over WhatsApp when you hit 80% of any budget.
  • Debt & subscription tracking — Monitor what you owe and what's auto-billing, with reminders 2 days before charges hit.
  • WhatsApp integration — Chat with Milo over WhatsApp for on-the-go expense logging and receive automated budget alerts and subscription reminders.
  • 3D animated mascot — A mood-reactive squirrel built with Three.js that reacts as you chat — happy when you save, thinking when processing, surprised when you overspend.
  • Multi-currency support — Works with 12+ currencies (USD, EUR, GBP, AED, SAR, and more) with proper locale formatting.

How we built it

The frontend is React 19 with Vite and Tailwind CSS v4, using shadcn/ui for a polished component library. The 3D mascot is built from scratch with React Three Fiber and Three.js — no pre-made models, just procedural geometry for the squirrel's body, eyes, whiskers, and tail, all with mood-driven animations.

The backend runs entirely on Convex, which gives us serverless functions, a real-time database, scheduled jobs, and file storage in one platform. Authentication uses Convex Auth with Google, GitHub, Apple, and password providers.

The AI layer is powered by Google Gemini, using function calling to let Milo execute actions like adding transactions, querying budgets, and looking up spending patterns — all within a natural conversation. Gemini's multimodal capabilities handle receipt image analysis and audio transcription.

WhatsApp integration uses Twilio's API with a webhook routed through Convex HTTP actions, enabling two-way messaging. Cron jobs run every 6 hours for budget alerts and daily for subscription reminders.

Challenges we ran into

  • AI reliability — Getting Gemini to consistently return structured JSON for transactions required careful prompt engineering. We built fallback heuristics that parse natural language when the AI's function calls fail, and added validation layers to catch hallucinated categories or impossible dates.
  • Smart date parsing — "Last Friday", "yesterday", "the 15th" all mean different things depending on context. We built a custom normalizer that handles relative dates, payday-based cycles, and timezone edge cases.
  • 3D performance — Rendering a Three.js scene alongside a chat interface required lazy-loading the entire 3D pipeline. The mascot only loads on the Milo page, keeping the rest of the app fast.
  • Budget cycle logic — Not everyone's month starts on the 1st. Supporting custom payday-based budget cycles (e.g., "my month starts on the 25th") required rethinking how we calculate spending windows and remaining days.
  • WhatsApp authentication — Linking a WhatsApp number securely required building a full OTP verification flow with expiration, attempt limiting, and phone number validation through Twilio.

Accomplishments that we're proud of

  • Milo can handle a full expense logging flow in a single natural language message — parsing amount, category, date, and description — with ~95% accuracy.
  • The 3D squirrel mascot is entirely procedural (no external 3D models) and runs smoothly with mood-reactive animations.
  • The WhatsApp integration creates a genuinely useful mobile experience without needing a native app — you can log expenses, get alerts, and chat with Milo all from your messaging app.
  • Session management with pinning, archiving, and auto-generated titles makes conversation history actually usable.
  • The entire backend — auth, database, real-time sync, cron jobs, file storage, and webhooks — runs on Convex with zero infrastructure management.

What we learned

  • Gemini function calling is powerful but requires defensive programming — always validate outputs and have fallbacks for when the model doesn't follow instructions perfectly.
  • Convex as a backend-as-a-service dramatically speeds up development. Real-time queries, scheduled functions, and HTTP webhooks all in one platform meant we could focus on features instead of infrastructure.
  • Prompt engineering is UX design — the personality, tone, and context window of the AI system prompt directly impact how useful and engaging the assistant feels.
  • Building a multimodal AI experience (text + images + audio) that feels seamless requires careful state management and graceful error handling at every step.

What's next for Milo

  • Bank account linking via Plaid for automatic transaction import
  • Spending insights dashboard with AI-generated weekly/monthly reports
  • Shared budgets for couples and families
  • Recurring expense predictions using historical spending patterns
  • Mobile app with push notifications using React Native
  • More AI tools — investment tracking, savings goal planning, and bill negotiation suggestions

Built with

React, TypeScript, Vite, Tailwind CSS, shadcn/ui, Radix UI, Three.js, React Three Fiber, Recharts, Convex, Convex Auth, Google Gemini API, Twilio, Node.js, Vercel

Built With

Share this project:

Updates