SafeScan: Scan Smart, Live Smarter

Sample food product image to try on our application
Sample medicine image to try on our application

Inspiration

In today’s marketplace, consumers face an overwhelming array of choices in beauty products, medications, and packaged foods, many of which may contain toxic ingredients and allergens with potential health risks. Ingredient labels often use scientific terms or obscure language, making it difficult for non-experts to discern whether products contain toxic substances, allergens, or components with known side effects. This lack of transparency can lead to unintended health consequences for individuals seeking to make safe and informed choices.

What it does

Safescan takes an uploaded image and identifies the product using Google Cloud Vision to recognize the exact name of the desired product in the picture. To get the product's information, SafeScan uses AI technologies to gather a list of the product's ingredients, search for each ingredient's allergen information, and detect any components that may pose a health risk. SafeScan is even able to list specific ingredients that are generalized on the label (i.e. generalized "preservatives" and "natural flavors"). Finally, it rationalizes a conclusion message for the user, telling them whether it is safe for consumption or not, and what some potential risks might be. It also lists the sources used so that the user can explore further if desired. There is also a side bar that the user can navigate to view more information about SafeScan's mission and about statement.

How we built it

Our project is a WebApp built with Streamlit that leverages advanced AI to autonomously gather information from the internet. The AI agent, powered by OpenAI and LangGraph, is capable of making independent decisions and browsing the web to retrieve relevant information. With Google Vision, the application can recognize products from images and fetch specific details about them.

Key Technologies Used:

OpenAI & LangGraph: Built an AI Agent to make decisions without any human intervention to gather information on different aspects (ingredients, allegens, etc.) of the product which user has shared.
Google Vision API: Used for image recognition to identify products and gather related data.
Docker: We containerized the app, ensuring a consistent and portable environment for deployment.
Terraform: Automated the deployment of the application to AWS Cloud and workers on the cloudflare, making it scalable and easy to manage.
Cloudflare Worker: Set up to proxy our EC2 instance directly to our .tech domain, enhancing the app's accessibility and performance. To sum up, building an independent, decision-making AI agent required integrating multiple APIs and ensuring smooth communication between them. Additionally, deploying on AWS Cloud with Terraform and setting up Cloudflare to proxy requests to our domain was essential for robust hosting and accessibility.

Challenges we ran into

One of the primary challenges we ran into was building the AI agent using LangGraph. LangGraph is a relatively new tool, and finding comprehensive documentation and resources was difficult. This made the process of developing and fine-tuning the agent complex, as we had to experiment extensively to understand how LangGraph integrates with OpenAI and how to structure workflows for independent decision-making and information retrieval. Another challenge we faced was ensuring that the AI agent could browse the internet efficiently while balancing performance and relevance of results. Additionally, setting up the deployment pipeline on AWS with Terraform and configuring Cloudflare to seamlessly proxy requests required careful planning to ensure secure and smooth redirection to our .tech domain.

Accomplishments that we're proud of

We’re proud of successfully creating a fully autonomous AI agent that can independently make decisions and gather information from the internet—a feature that bridges the gap between static knowledge and dynamic, real-time insights. Integrating LangGraph and OpenAI to achieve this capability, despite the limited documentation, was a significant achievement for our team. We’re also proud of the seamless deployment pipeline that we established. By containerizing our application with Docker and automating the infrastructure setup with Terraform, we achieved a robust, scalable deployment on AWS. Setting up Cloudflare to proxy our EC2 instance to our custom .tech domain was another highlight, ensuring that our app is not only accessible but also optimized for performance and security.

What we learned

This project taught us a lot about working with cutting-edge AI tools and the importance of adaptability when documentation is limited. Building the AI agent with LangGraph required a deeper understanding of AI workflows and the architecture behind autonomous decision-making models. We learned to rely on experimentation and community forums to solve challenges, which strengthened our problem-solving skills. We also gained hands-on experience with cloud infrastructure management and automation. Using Docker for containerization and Terraform for deploying on AWS taught us how to build a scalable and maintainable deployment pipeline. Additionally, setting up Cloudflare for our domain helped us understand the nuances of web security, proxy configurations, and performance optimization for real-world applications. Overall, this project enhanced our technical expertise and prepared us to tackle similar challenges in future endeavors.

What's next for SafeScan

Adding User Interaction and Customization: Introducing user settings for personalization, such as search preferences and customizable alerts, would add value to the app and improve user engagement. Users would be able to create an account and set up a profile, saving information and getting suggestions geared specifically toward their bodies.
Developing a Mobile Application: To make the platform even more accessible, we plan to develop a mobile version of the app. This would allow users to quickly scan products, retrieve information, and access insights on the go. A mobile app would also enable us to incorporate features like push notifications for real-time alerts and recommendations.
Introducing a Product Safety Score: We aim to implement a personalized product safety scoring system that analyzes products based on each user’s profile, preferences, and health concerns. By leveraging the AI agent's decision-making capabilities, this feature would provide users with a customized safety score for each product, offering valuable insights aligned with their lifestyle and needs.