calfax

Inspiration

Me and Collin share a passion for dining out, but we're also committed to maintaining healthy, active lifestyles. We realized that there was a gap in the market for a tool that would allow individuals like us to make informed decisions about their food choices seamlessly. This realization was the spark that ignited our journey to create this project.

What it does

By simply snapping a photo of your meal, the app utilizes advanced AI to identify the food items and instantly provides a detailed nutritional breakdown. Built with a cloud-based backend, 'Calfax' ensures fast, accurate, and seamless access to the nutritional information you need, making informed dining choices easier than ever.

How We Built It

We employed NextJS for the frontend and fullstack development. For the backend, we utilized Convex, leveraging their backend-as-a-service software solution. To access hosted open-source multimodal models, we integrated with the Replicate API. Furthermore, we collaborated with together.ai's API to finetune the LLaMa model. This ensured that our application provided consistent and accurate outputs regarding nutritional values.

Challenges we ran into

Every project comes with its set of hurdles, and ours was no exception. Grappling with Convex was a novel experience. We had to familiarize ourselves with various tools like mutations, actions, stores, and more. Understanding the concept of backend-as-a-service was a challenge in itself. Extracting the food contents from an image was another significant hurdle. We experimented with various methods of extracting data from images before landing on one that perfectly suited our needs. Another major challenge was sourcing a reliable dataset for our nutritional facts. Before we stumbled upon the ideal dataset, we explored the possibility of requesting popular food databases.

Accomplishments that we're proud of

Successfully integrating multiple AI models and backend services was a significant milestone for us. We managed to combine the Replicate API and together.ai's API, which was a challenging yet rewarding experience. The fact that 'Calfax' can identify food items from images and provide a basic nutritional breakdown is an accomplishment we're genuinely proud of, given the technical challenges we faced.

What we learned

Throughout the development of 'Calfax', we gained hands-on experience in finetuning AI models and working with cloud-based backends. We learned the nuances of multimodal models, especially in the context of image data extraction and conversion. The introduction to Convex's backend-as-a-service was a new learning curve, teaching us about modern backend solutions. The process of sourcing and integrating a dataset for nutritional facts also provided insights into the importance of data in such applications.

What's next for calfax

We're keen on refining 'Calfax' based on user feedback and real-world usage. Potential future steps include exploring collaborations with local eateries to integrate specific menu items, adding features to cater to specific dietary needs, and possibly integrating with other health-focused platforms. Our primary goal is to ensure 'Calfax' serves its users effectively and becomes a reliable tool for nutritional insights.