Smart Receipts

Inspiration

The inspiration behind Smart Receipts stemmed from a common frustration shared by many: the chaotic and often overwhelming task of managing daily expense receipts. We envisioned a solution that would eliminate the stress of lost receipts, simplify tax filing, and provide clear insights into spending habits. The idea was to leverage cutting-edge technology to transform a mundane and often frustrating chore into an effortless and organized experience.

What it does

Smart Receipts is a comprehensive cloud-based application designed to revolutionize personal expense management. It allows users to easily capture, categorize, and track their expenses. Key functionalities include:

AI-Powered Receipt Scanning: Users can simply snap a photo of a receipt, and the AI automatically extracts all relevant details like vendor, amount, category, and date.

Tax-Ready Organization:All receipts and expense data are stored securely in one place, making them easily accessible and organized for tax filing season.

Monthly Expense Tracking: The application automatically monitors spending patterns, providing users with a clear overview of their expenditures and helping them stay within budget.

Secure & Private Storage: Financial data is encrypted and stored securely.

Manual Expense Management: Users can also manually add, edit, or delete expense details.

Receipt Image Download: Users can download original receipt images using pre-signed URLs.

Weekly Reports: Provides a summary of weekly spending, budget comparison, and spending breakdown by category, with options to download or share the report.

Tax Calendar Feature: Displays expenses in a calendar view and allows exporting filtered tax data as a CSV.

How we built it

Smart Receipts was built on a robust, scalable, and serverless AWS cloud architecture.

Frontend: The user interface is a React TypeScript application hosted on an Amazon S3 bucket, configured for static website hosting. It uses the AWS SDK for JavaScript for integration.

Authentication & Authorization: Amazon Cognito handles user sign-up, sign-in, and access control, utilizing both a User Pool for identity management and an Identity Pool for obtaining temporary AWS credentials.

Backend Logic (AWS Lambda): The core business logic is implemented using several AWS Lambda functions written in Python:

UploadImageLambda: Stores uploaded receipt images in an S3 bucket.

BedrockCategorizationLambda: Retrieves images from S3, invokes the Amazon Bedrock API (specifically the Claude 3 model) to categorize expenses, and extracts details.

SaveExpenseLambda: Saves categorized expense data to DynamoDB.

GetExpensesLambda: Fetches expense data for users from DynamoDB.

UpdateExpenseLambda:Updates existing expense data in DynamoDB.

DeleteExpenseLambda: Deletes specified expense items from DynamoDB.

GetPresignedUrlLambda: Generates temporary, secure URLs for accessing S3 objects (receipt images).

UpdateUserPreferencesLambda, GetUserPreferencesLambda: For user preferences.

Data Storage: Amazon S3 buckets are used for storing receipt images and hosting the frontend application. Amazon DynamoDB stores the categorized expense data in a SmartReceiptsExpenses table.

AI Categorization:Amazon Bedrock, utilizing the Claude 3 model, powers the AI-driven categorization of receipts.

IAM Roles and Policies: Specific IAM roles with least-privilege policies were created for secure interaction between services.

Amazon EventBridge Trigger:This was set up to trigger the SendNotificationLambda function on a daily schedule (e.g., at 10:00 AM UTC) for sending receipt reminders.

Challenges we ran into

One of the primary challenges was accurately extracting and categorizing data from various receipt formats using AI. Receipts come in all shapes and sizes, with different layouts and fonts, which can make OCR and subsequent categorization tricky. We addressed this by fine-tuning our prompts for the Claude 3 model in Amazon Bedrock to ensure robust extraction of vendor, amount, category, description, and date, even for missing values. Ensuring secure and efficient handling of image uploads and generating temporary secure URLs for downloads also required careful implementation of S3 bucket policies and Lambda functions. Managing user authentication and authorization across various AWS services using Cognito also presented a learning curve to ensure seamless and secure user experience.

Initially we used textract in a state machine which invoked textract lambda and pass extracted data to bedrock lambda which finally sends data to save dynamodb lambda, somehow this state machine did not worked as expected so changed the whole architecture to the existing one.

Accomplishments that we're proud of

We are particularly proud of several key accomplishments:

Seamless AI Integration: Successfully integrating Amazon Bedrock with the Claude 3 model to provide accurate and automatic categorization of expenses from receipt images, significantly reducing manual effort for users.

Robust Serverless Architecture: Building a highly scalable and resilient serverless architecture using AWS Lambda, S3, DynamoDB, and Cognito, which ensures cost-effectiveness and high availability.

Intuitive User Experience: Developing a user-friendly React frontend that simplifies the complex process of expense management, from snapping a photo to viewing detailed reports.

Comprehensive Feature Set: Delivering a wide range of features, including tax-ready organization, monthly expense tracking, and weekly reports, all designed to empower users to take control of their finances.

What we learned

Throughout the development of Smart Receipts, we gained valuable insights into building robust serverless applications on AWS. We learned the importance of:

Leveraging Managed Services: How services like AWS Lambda, S3, DynamoDB, and Cognito significantly accelerate development and reduce operational overhead.

AI Prompt Engineering: The nuances of crafting effective prompts for large language models (like Claude 3) to achieve desired data extraction and categorization accuracy.

Security Best Practices: Implementing least-privilege IAM roles and policies to ensure secure interactions between different AWS services and protect user data.

Scalability Design: Designing the architecture with scalability in mind from the ground up, ensuring the application can handle a growing number of users and data.

What's next for Smart Receipts

For the future of Smart Receipts, we plan to enhance the application with several exciting features:

Advanced Reporting and Analytics: Implementing more sophisticated reporting tools, including custom date ranges, multi-category analysis, and predictive spending insights.

Integration with Financial Institutions: Exploring possibilities for secure integration with bank accounts and credit cards to automatically pull transaction data, further reducing manual input.

Multi-currency Support: Adding functionality to handle expenses in multiple currencies, catering to international users and travelers.

Budgeting Tools: Developing more interactive budgeting features, allowing users to set specific budgets for categories and receive real-time alerts.

Receipt Archiving and Export Options: Providing more diverse export formats for tax purposes and long-term archiving.

Machine Learning for Anomaly Detection: Implementing ML models to identify unusual spending patterns or potential fraudulent activities.