-
Pie chart that uses an API endpoint to update
-
Time series graph
-
Adjustable pie chart for goal setting
-
Document upload
-
Gemini LLM output relating to user inputs
-
Sign in page
-
Sign up page with user prompts that are passed into the LLM Agent
-
Supabase database schema
Inspiration
Inspired by CapitalOne's problem statement, we decided to make a project to attempt to address the key issue of a lack of financial literacy among children, the elderly & everyone in between. CapitaLiteracy makes use of computer vision, machine learning and helpful infographics to build a user profile while providing actionable insights into their spending, credit and ways to improve the two.
What it does
We built CapitaLiteracy with an integrated React + Flask architecture:
Frontend: built using Vite, Next.js, and React, styled with TailwindCSS, and deployed on Vercel.
Backend: implemented with Flask, where we handled OCR, classification, and data analysis.
Database: Supabase is used for authentication and storing user profiles securely.
APIs: integrated Tesseract for debit/credit statement recognition and Gemini API for generating user insights.
Hosting: Flask backend deployed through Vercel functions with REST endpoints consumed by the frontend.
The architecture allowed seamless communication between the machine learning pipeline and the frontend visualisations.
Challenges we ran into
OCR accuracy: Ensuring consistent text extraction across various receipt formats.
Dataset limitations: We had to augment limited data to train models for transaction categorisation.
Integration issues: Bridging Flask and Next.js required configuring CORS and proxying requests correctly.
Visualisation clarity: Designing infographics that made sense to non-technical users while retaining data accuracy.
Accomplishments we are proud of
Successfully integrated computer vision and AI-driven insights into one cohesive app.
Built a complete full-stack system within the hackathon time limit.
Designed a clean, intuitive user experience for visualising complex financial data.
Created a solution that could genuinely improve financial literacy and empower better decision-making.
What we learned
Technical: Improved our skills in React hooks, Flask routing, and full-stack integration.
Machine Learning: Learned how to fine-tune OCR and text classification models for structured financial data.
Collaboration: Understood the importance of version control, agile teamwork, and time-boxing features.
Finance: Gained a deeper appreciation for the nuances of spending categorisation and user data privacy.
What's next for CapitaLiteracy
Introduce gamification (financial challenges, rewards, and badges).
Expand our ML model to handle multi-currency and regional financial data.
Add personalised goal tracking and AI financial coaching.
Develop a mobile version with offline receipt scanning.
Log in or sign up for Devpost to join the conversation.