HARINA v3 - AI Receipt Recognition Web Application

An open-source web application that leverages multiple Large Language Models (LLMs) to automatically extract product information from receipt images, streamlining household budget management

Comment

Inspiration

In daily household budget management, manually entering receipt data is time-consuming and tedious. This is particularly challenging for couples sharing household expenses, where accurately tracking who purchased what and when becomes difficult. Inspired by recent advances in LLM technology that have dramatically improved image recognition accuracy, we developed an automated receipt recognition system utilizing these cutting-edge AI technologies.

What it does

HARINA v3 automatically processes receipt images using multiple AI models (Gemini, GPT-4o, Claude) to extract structured data including store information, purchased items with categories, prices, and transaction details. The web application provides:

  • Multi-file receipt processing with real-time progress tracking
  • Support for multiple AI providers with free Gemini API option
  • User-specific tracking (husband/wife expense separation)
  • Interactive dashboard with spending statistics and charts
  • CSV import/export functionality for existing data
  • Japanese-inspired UI design optimized for household budgeting
  • Complete Docker-based deployment solution

How we built it

This project was developed entirely through dialogue-driven development with Kiro. Every component was built incrementally through conversational programming:

Technology Stack:

  • Frontend: Next.js 14 + TypeScript + Tailwind CSS + Chart.js
  • Backend: FastAPI + Python + PIL for image processing
  • Database: PostgreSQL with normalized schema
  • AI Integration: LiteLLM supporting multiple providers
  • Infrastructure: Docker Compose microservices architecture

Kiro Development Process: Each feature was developed through structured conversations covering architecture design, database schema optimization, React component implementation, API endpoint creation, and error handling strategies.

Challenges we ran into

  1. Multi-Provider LLM Integration: Balancing accuracy, cost, and availability across different AI services
  2. Complex UI State Management: Managing real-time updates across multiple React components during file processing
  3. Asynchronous File Processing: Implementing robust progress tracking for multiple concurrent receipt uploads
  4. Database Design: Optimizing for both performance and data integrity with normalized receipt/item relationships
  5. Dialogue-Driven Development: Structuring conversations with Kiro to systematically build complex features

Accomplishments that we're proud of

  • Complete AI Pair Programming Implementation: Successfully developed a full-stack application entirely through Kiro dialogue
  • Multi-LLM Architecture: Created a flexible system supporting three major AI providers with seamless switching
  • Free Operation Model: Implemented using Gemini API free tier, making advanced AI accessible to everyone
  • Open Source Contribution: Provided a complete reference implementation for LLM-powered web applications
  • Real-World Utility: Built a practical solution that genuinely improves daily household management

What we learned

  • AI Integration Patterns: Effective strategies for combining multiple LLM providers in production applications
  • Prompt Engineering: Techniques for structured data extraction using XML templates and category classification
  • Dialogue-Driven Development: How to systematically build complex features through conversational programming with AI
  • Modern Full-Stack Architecture: Integration patterns for Next.js, FastAPI, and PostgreSQL in containerized environments
  • User Experience Design: Creating intuitive interfaces for AI-powered applications with appropriate feedback and error handling

What's next for HARINA v3 - AI Receipt Recognition Web Application

  • Mobile Application: React Native version for on-the-go receipt capture
  • Advanced Analytics: ML-powered spending pattern analysis and budget predictions
  • Multi-Language Support: Expanding beyond Japanese receipts to international formats
  • API Marketplace: Public API for third-party integrations with existing finance apps
  • Community Features: Sharing anonymized spending insights and budgeting tips
  • Enhanced AI Models: Fine-tuning custom models for specific receipt types and improved accuracy

Built With

Share this project:

Updates