Ghost Accountant: Autonomous UI Navigator & Financial Agent

An autonomous agent that captures financial intent through live chat and executes real-world tasks like booking travel directly in third-party apps via the Ghost Navigator.

Comment

Inspiration: Managing personal and business finances is painful. People spend hours manually entering receipts, updating ledgers, and hopping between booking apps to schedule travel, hotels, or shows. Existing AI assistants often provide suggestions but cannot execute tasks autonomously, leaving users to do the repetitive work themselves.

We envisioned an agent that combines real-time chat, vision, and autonomous navigation to handle financial and everyday tasks seamlessly. Ghost Accountant was born to transform tedious workflows into a single intelligent conversation.

What it does: Ghost Accountant is an autonomous AI agent that: Observes - Captures user intent through live chat (text/voice) or receipt images. Reasons - Parses financial transactions and booking requests using advanced AI reasoning. Acts - Automatically updates ledgers or navigates websites to book flights, hotels, or tickets. Responds - Confirms actions instantly in chat, providing transparency and insight.

Example: User: "Book a hotel in New York for next weekend and add my lunch expense of $45." Ghost Accountant: Adds $45 lunch expense automatically Opens the hotel booking website, pre-fills details, and confirms the reservation Gives live chat confirmation

This solves the real problem of manual bookkeeping and fragmented task management, delivering immediate productivity gains.

How we built it, Frontend: Flutter (Dart) with Riverpod for state management and go_router for navigation AI Integration: Gemini Live API (chat & voice), Gemini Vision API (receipt parsing), Gemini Reasoning Backend: Firebase Auth + Firestore, Cloud Run backend (ADK) Databases: Firestore (real-time), Hive (offline caching) Deployment: Google Cloud (Vertex AI, Cloud Run, Firebase Hosting)

The system follows an observe → reason → act → respond loop, enabling continuous autonomous operation.

Challenges we ran into: Parsing multi-transaction chat inputs in natural language with high accuracy Ensuring Ghost Navigator can safely interact with third-party websites without errors Achieving real-time synchronization between AI reasoning and UI updates Handling edge cases in live voice interactions while maintaining smooth UX We overcame these by iteratively testing commands, building robust error handling, and leveraging Gemini’s reasoning and vision APIs.

Accomplishments that we're proud of, First fully autonomous agent capable of updating expenses and executing real-world booking tasks Live multi-modal interaction — chat, voice, and visual receipt processing Ghost Navigator that navigates apps and websites autonomously Instant multi-transaction parsing, saving users hours of bookkeeping

What we learned: Real-time AI can reliably handle financial and task commands if built with modular observe → reason → act → respond loops Multimodal integration (voice + vision + text) is critical for human-like agent interaction Proper cloud architecture and offline caching are essential for scalability and robustness

What's next for Ghost Accountant: Autonomous UI Navigator & Financial Agent: Scalable Impact: The system can be extended to enterprise bookkeeping, travel agencies, and multi-platform task automation Future Features: Auto tax reports, accounting software integrations (QuickBooks/Xero), fraud detection, smart budgeting, multi-currency support Massive Reach: Millions of users can save hours per week, reduce errors, and improve financial visibility

Why Us: Ghost Accountant is unique because it combines real-time chat, autonomous navigation, and financial intelligence in one agent, something no existing tool does at scale

Ghost Accountant isn’t just a finance app, it’s a productivity multiplier with real-world impact, ready to transform the way individuals and businesses manage tasks and finances.

Built With

  • cloud-run-(backend)
  • firestore-(real-time-database)
  • flutter
  • gemini-reasoning-api-backend-&-cloud-services:-firebase-auth
  • gemini-vision-api-(receipt-extraction)
  • go-router-(navigation)-ai-&-apis:-gemini-live-api-(voice-&-chat)
  • hive-(offline-caching)-other-tools:-google-cloud-platform-(hosting-&-deployment)
  • languages-&-frameworks:-dart-&-flutter
  • mobile
  • riverpod-(state-management)
  • vertex-ai-databases-&-storage:-firestore-(cloud)
  • web
Share this project:

Updates