PocketPilot

Camera Page
Reccomendation Page
SpendMap
Results Page
History Page
Stats Page

Inspiration

When Owen and Tony went to the University of Toronto in their first year, they quickly realized the rising costs of living and inflation in Canada. Furthermore, during the upcoming midterms, they did not have time to budget and keep track of where their money was going.

Expense tracking is not a new idea. Applications such as Expensify and Receipt Bank already allow users to scan and input receipts and digitize expenses. However, these platforms are all designed for businesses and enterprises to report expenses and manage accounting.

When Owen and Tony were searching for budgeting tools for consumers, such as Mint, those apps relied heavily on bank integration. Since much of their income came from multiple sources, including part-time jobs and financial support from their parents, the money flowed from different accounts, and some spending happened directly through cash, debit, or shared purchases with friends.

We wanted to create an application that helps them quickly and effectively gauge a sense of their spending and financial insights.

What it does

PocketPilot is an AI-powered financial wellness app that turns receipt photos into spending statistics.

Users take a picture of a receipt, and the system automatically extracts important information such as merchant name, items purchased, taxes, payment method, date, and store location. This information is then organized into categories such as grocery, miscellaneous, and recurring expenses.

Unlike other budgeting apps, PocketPilot analyzes item-level purchase data, allowing users to understand not only where they spent money, but also what they actually bought.

How we built it

PocketPilot was built using a modern cloud-based architecture combining mobile development and AI.

The frontend was developed with React Native and Expo, which allowed us to build a fast cross-platform mobile experience with camera integration for receipt capture.

The receipt images are uploaded to a a backend built with Node.js and Express, deployed through Firebase Cloud Functions. Before analysis, the images are processed and then stored using Cloudinary, which enhances clarity and sharpness to improve extraction accuracy.

The enhanced image is then analyzed using Google Gemini 2.5 Flash, which performs multimodal receipt understanding and extracts structured data such as merchant names, line items, totals, and payment metadata.

The extracted information is stored in Firebase Firestore, where each user maintains a private receipt history. The merchant addresses are put into Apple Maps geocoding, allowing users to visualize where their spending occurs geographically.

Gemini also generates financial advice and personalized recommendations based on the user’s spending patterns.

Challenges we ran into

One challenge we encountered was running and testing the mobile app using Expo Go. Normally, Expo allows developers to run an app locally and connect to it from a phone over the same WiFi network. However, because we were working on a public network at the hackathon venue, the phone and development machine could not communicate directly. The connection would fail when scanning the QR code in Expo Go. To resolve this, we switched to Expo’s tunneling mode, which routes the connection through the internet instead of relying on the local network. This allowed our phones to successfully connect to the development server and continue testing the app.

Another challenge was the limited time frame of the hackathon. Since we wanted our project to be unique and effective, we spent a significant amount of time early on discussing the project idea, planning the system architecture, and deciding how different components such as AI receipt extraction, the backend pipeline, and the mobile interface would work together. As a result, we had to work quickly and prioritize the most important feature: building a functional prototype that demonstrated the core idea of PocketPilot.

Accomplishments that we're proud of

We are proud that PocketPilot successfully transforms something people normally discard, a receipt, into a form that people can visualize and gain insight from.

Within the time limit, we built a system capable of scanning receipts, extracting structured data, categorizing purchases, and generating AI-driven financial insights, which required significant backend work.

We are also proud of the AI pipeline that uses Cloudinary for image enhancement, multimodal analysis, and financial recommendations within a single workflow.

What we learned

In this project, we gained a lot of insight into how AI can be used to solve real-world problems.

We learned how powerful multimodal AI can be when applied to real-world data. Furthermore, we also learned the importance of building reliable pipelines that handle messy inputs such as low-quality images by using Cloudinary for image enhancement.

In addition, working under hackathon constraints taught us how to rapidly prototype, iterate on ideas, and collaborate effectively across frontend, backend, and AI components.

What's next for PocketPilot

Future development will focus on scaling our app. We will optimize each user's memory usage to be as efficient as possible (for example, deleting images from the cloud after 30 days) and find scaling opportunities.

We also plan to introduce spending trend analysis that compares weekly and monthly spending patterns and detects unusual spikes.

In the long term, if our app gains popularity, we can include a paid option that is more generous with AI usage and offers more features.

We believe that in the long run, PocketPilot could evolve into a full AI-powered financial assistant that helps users like Owen and Tony make smarter financial decisions every day.