Shared Household Expenses Agent for Tracking & Settlement

AI-powered family expense tracker with receipt ingestion, semantic search and settlement calculation with future provisions for collaboration with other agentic systems using advanced capabilities.

Comment

## Inspiration

During Covid, unfortunately both my parents fell seriously ill and were hospitalized for several months, requiring round-the-clock ICU nursing care at hospital & home. Four of us siblings have shared the financial burden of his medical care, household expenses, and daily needs. For years, we tracked everything manually on WhatsApp — screenshots of bills, voice messages about payments, informal IOUs. I built this agent to solve a real problem many families face daily.

## What it does

The Shared Household Expenses Agent is an AI-powered family financial management system built on Google ADK 2.2.0, Gemini 3.5 Flash, and MongoDB Atlas.

Core capabilities:

Natural language queries — ask "How much did we spend on nursing staff?" and get an instant answer from 18 real transactions (as of now). Plan to quickly populate both previous transactions and current ones on a real time basis.

Receipt image ingestion — photograph a medicine bill, the agent extracts all fields using Gemini Vision, confirms with you (Y/N), and saves to MongoDB Atlas with a Vertex AI embedding for semantic search

Settlement calculation — calculates exactly who owes whom based on contributions. Future provision to also do TVM (Time Value of Money) calculations via a separate tool.

Semantic search — powered by MongoDB Atlas Vector Search with 768-dimension Vertex AI embeddings — finds relevant expenses even when query wording differs from stored data

MongoDB Atlas Charts — real-time dashboard showing category breakdown, member contributions, and monthly trends — built with zero frontend code using Natural Language Charts. Easy to visualise for all the family members.

A2A Protocol — Agent Card published at /.well-known/agent.json enabling future integration with personal ledger accounting, tax filing, insurance claim, medical prescription, procurement, etc. agents.

## How we built it

Agent Framework: Google ADK 2.2.0 with adk web for local development and adk api_server for Cloud Run production deployment

AI Model: Gemini 3.5 Flash via Vertex AI global endpoint — handles both text queries and receipt image analysis (multi-modal)

Database: MongoDB Atlas M0 (free tier) with the household_expenses database containing 5 collections — transactions, family_members, settlements, agent_memory, time_logs

Vector Search: MongoDB Atlas Vector Search index with 768-dimension embeddings generated by Vertex AI text-embedding-004 model — enables semantic queries like "Papa's medical bills" to find records tagged as nursing staff expenses

MongoDB MCP Server: npx mongodb-mcp-server@latest --readOnly integrated as an ADK MCPToolset — confirmed called at runtime (list-databases visible in ADK trace)

Sequence of activities is as follows :

User provides image path where the expense bills are stored along with the expense owner (Mom,Dad, Household,etc), expense payer (one of the siblings) and proposed sharing ratio across siblings. Gemini Vision extracts multiple data fields — date, amount, vendor, category, bill number, remarks. Agent presents all fields with confidence level. Human confirms with (Y - Yes/ N -No) (HITL) (Human In The Loop). Vertex AI generates embedding. Transaction saved to MongoDB Atlas. Immediately searchable via Vector Search.

Infrastructure: Google Cloud Run (us-central1), Google Secret Manager for credentials, Google Artifact Registry for container images

A2A: Agent Card with 8 skills published at /.well-known/agent.json per Google A2A protocol specification

## Challenges we ran into

  1. Windows subprocess limitation — MongoDB MCP Server runs as a Node.js subprocess via npx. Windows SelectorEventLoop does not support subprocesses, so MCP only works on Cloud Run (Linux). Solved by deploying to Cloud Run for production use.

  2. Special characters in MongoDB password — The Atlas connection string password contained # and * characters which are URL-reserved. Solved using Python urllib.parse.quote_plus() for proper URL encoding.

  3. Bill date parsing — Gemini Vision read 08/06/24 as 2024 instead of 2026 on a pharmacy receipt with a 2-digit year. Solved by adding a date correction utility and updating the Vision prompt to assume current year for 2-digit dates.

  4. A2A SDK dependency — The --a2a flag in adk deploy cloud_run requires a2a-sdk which was not in the base ADK requirements. Resolved by switching to gcloud run deploy --source with a custom Dockerfile giving full control over dependencies.

  5. Cloud Run cold start delays — Initial requests took 15-20 seconds due to MongoDB MCP server startup via npx. Solved by pre-caching the MCP package in the Docker image build step.

  6. Since this was the first time I was working with MongoDB, it took a couple of days to understand the environment.

## Accomplishments that we're proud of

Real family data — not synthetic demo data. 18 actual transactions totalling ₹6,71,552 in real medical care expenses. Family siblings are happy that a global contemporary solution is being created for a pressing family issue.

Complete agentic workflow — receipt image → Gemini Vision → HITL confirmation → MongoDB save → Vector Search retrieval — 8 steps, fully autonomous.

MongoDB MCP confirmed at runtime — list-databases() call visible in ADK trace, satisfying the hackathon partner integration requirement.

Multi-modal — same agent handles text queries, image analysis, and structured data queries.

A2A ready — Agent Card published with 8 skills, enabling future multi-agent financial ecosystem.

Zero frontend code — MongoDB Atlas Charts dashboard built entirely with Natural Language Charts in under 30 minutes.

## What we learned

Google ADK 2.2.0 is powerful but has Windows-specific limitations around subprocess handling that require Linux deployment for full MCP functionality.

MongoDB Atlas Vector Search with semantic embeddings dramatically outperforms keyword search for natural language expense queries.

Gemini Vision is remarkably accurate at extracting structured data from printed receipts — High confidence on 8 fields from a pharmacy bill.

The A2A protocol Agent Card is straightforward to implement and adds significant architectural value even without a full task endpoint.

MongoDB Atlas Charts Natural Language feature is genuinely impressive — production-quality dashboards in minutes with zero code.

## What's next for Shared Household Expenses Agent

MongoDB Atlas Auto Embeddings — replace manual Vertex AI embedding pipeline with MongoDB's native Voyage AI integration (Public Preview May 2026).

Gmail draft ingestion — family members email expense details, agent processes drafts automatically

Time Value of Money (TVM) settlement — adjust settlement amounts for cost of capital (% per annum) from payment date to reimbursement date.

Voice input — Gemini 3.1 Flash Live for hands-free queries while caregiving

Field-level correction tool — agent-assisted correction of individual transaction fields

MongoDB Time Series — daily interest accrual tracking on outstanding balances

MongoDB Client-Side Field Level Encryption — encrypt sensitive financial fields before storage

Full A2A task endpoint — enable agent-to-agent communication with agents in the domain of personal ledger accounting, tax filing, insurance claim, medicine prescription, medicine procurement, etc. in the future.

Multi-family support — extend beyond single household to support multiple families.

Expense Analysis - Advanced Expenses Split & Analysis methods

Generic Document Cluster - Mapping of MongoDB collections to proprietary generic document clusters

##System Capability Map

The agent's capabilities are expressed using the NVLN methodology (Noun→Verb→Link→Noun) across 9 functional layers:

Layer 1: Data Ingestion,

Layer 2: Embedding & Search,

Layer 3: Query & Analytics,

Layer 4: Settlement & TVM,

Layer 5: MongoDB MCP,

Layer 6: Multi-Modal Vision,

Layer 7: A2A Protocol,

Layer 8: Visualization,

Layer 9: Security & Compliance.

Conclusion - A modern agentic system in combination with a contemporary cloud database powered with AI intelligence available at one's fingertips confirms what YCombinator president Gerry Tan says, "that it is indeed possible to boil the ocean" for solving real world challenges.

Built With

  • a2a-protocol
  • atlas-charts
  • docker
  • gemini-3.5-flash
  • google-adk
  • google-cloud-run
  • mongodb-atlas
  • mongodb-mcp-server
  • python
  • secret-manager
  • text-embedding-004
  • vector-search
  • vertex-ai
Share this project:

Updates