RAGs to Riches

Stop asking "Where is my money?" Rags to Riches is your personal AI financial manager that turns messy bank CSVs and receipts into clear insights, instantly generating charts of your spending.

Comment

Inspiration

The idea for RAGs to Riches was sparked by a conversation with a friend who realized they’d been paying for a subscription they hadn't used in months, an invisible, recurring drain on their wallet. It made us wonder: "Where exactly is our money going?"

Driven by personal curiosity and a literal mountain of collected receipts, we decided to audit our own spending habits. Since moving to the US, we had meticulously saved every physical bill and digital invoice. We wanted a tool capable of analyzing this real, lived-in data, but we knew a standard text-based AI chat would fall flat. We needed a solution that could not only answer our financial questions but instantly back them up with rich, visual insights.

We realized a standard AI chat would fall flat because it lacks temporal persistence. While ChatGPT treats every session like a blank slate, we needed an Evergreen brain, one that has years of indexed financial history ready to query at a moment's notice without us having to re-upload files every time we log in

What it does

RAGs to Riches is a personal AI financial manager that turns messy data into actionable wealth insights. You upload your scattered physical receipts and unstructured bank CSVs, and the AI automatically extracts, categorizes, standardizes, and deduplicates overlapping transactions across all your sources.

Because physical receipts capture granular, line-item details that standard bank statements completely miss, RAGs to Riches unlocks a new level of financial insight. Instead of just seeing a generic $80 charge at a restaurant, you can ask deep, hyper-specific questions like: "How much did I actually spend on tips and alcohol last month?"

Instead of building tedious pivot tables, you simply ask the AI these questions in natural language. Powered by RAG, the agent analyzes your exact spending and dynamically generates interactive, real-time Plotly charts right in the chat to visualize your cash flow and expose hidden leaks.

How we built it

We built the application around a Retrieval-Augmented Generation (RAG) architecture, split into distinct read and write pipelines:

  • Data Pipeline (Write): Users upload either unstructured bank CSVs or physical receipt photos via our React Native Web App dashboard. Receipt photos are processed using Gemini's Multimodal OCR. Both data streams then pass through an "LLM Mapper" for schema normalization, followed by a DuckDuckGo Web Enrich step to pull in missing merchant details.
  • Storage Layer: The enriched, clean data is stored relationally in PostgreSQL (via Supabase or Databricks) for exact number crunching, and simultaneously embedded as vectors in a VectorDB (via Qdrant or Actian) for semantic similarity searches.
  • AI Orchestration (Read): When a user asks a query, it is sent to our LangGraph Agent Controller. Powered by an underlying LLM (like Gemini or GPT-4), this agent acts as the core reasoning engine.
  • MCP Tool Server: To answer the query, the LangGraph agent dynamically routes requests to a custom Model Context Protocol (MCP) server containing four specialized tools:
  • SQL Tool: Runs exact queries against PostgreSQL for hard, structured numbers.
  • Vector Tool: Performs semantic searches on the VectorDB to find ambiguous line items.
  • Chart Tool: Generates on-the-fly UI visualizations using Streamlit. Image Tool: Retrieves and renders receipt media back to the user interface.

Challenges we ran into

  • Coming into this project, we were completely inexperienced with full-stack mobile app development, which brought intense challenges.
  • De-duplication Logic: We quickly realized that uploading a bank CSV and the physical receipt for the same meal created duplicate transactions. Building a reliable programmatic hashing system to match identical purchases across different data sources without losing the granular receipt line-item data was incredibly difficult.
  • Untangling Unstructured Data: Bank CSVs use terrible formatting (e.g., UBER *EATS 0932). Building an LLM pipeline to reliably enrich this into a clean taxonomy took heavy prompt engineering.
  • Database Architecture: Designing bulletproof PostgreSQL schemas so that parsed receipt items mapped correctly to parent transactions—surviving asynchronous uploads and upsert conflicts—was a steep learning curve.
  • Taming the Agent: Striking a balance between hallucination and letting the AI autonomously execute complex SQL queries against our database required strict safety layers and Row-Level Security (RLS) policies.
  • Debugging vibe coded code that don't stop making more bugs

Accomplishments that we're proud of

We are incredibly proud to have built a fully functional app that will be very important in managing our personal finances. Along the way, we tackled real technical challenges, solving the infamous long tail problem in our vector database to ensure accurate calculations, designing a reliable architecture where precise computation and RAG work seamlessly together, and integrating web search augmentation to improve vector retrieval and enable true semantic search over transactions. We believe it has genuine product potential. The hard technical problems have been solved, and while time constraints prevented us from delivering the most polished version of the app, the foundation we've built is something we're truly excited about.

What we learned

This project was a trial by fire in modern AI development. We learned how to move beyond basic chatbot wrappers into true agentic workflows where the AI actively queries databases, handles vector embeddings, resolves matching algorithms for duplicates, and writes user interface components natively. We also learned a tremendous amount about PostgreSQL, Python web frameworks, and the intricacies of the Model Context Protocol.

What's next for RAGs to Riches

We plan to transform RAGs to Riches from a reactive tool into a proactive financial guardian. Our roadmap includes:

  • Email Integration: Forward digital receipts directly to the app for automatic parsing.
  • Proactive Alerts: Receive autonomous AI notifications if your spending spikes mid-month.
  • Interactive Voice Mode: Speak to the agent in real-time to log cash expenses or check budgets hands-free.
  • Smart Money Splitting: Upload group receipts to automatically itemize the bill, calculate shared taxes, and figure out exactly who owes what.

Built With

Share this project:

Updates