Our app's homepage before scanning
Our app's camera screen
Our app's manual input screen
Our app's homepage after scanning

Inspiration

We got inspiration from people who wanted to know what they were actually consuming. Many people pick up products at the grocery store without understanding what the ingredients actually are. We wanted to provide a solution for that with just a simple scan.

What it does

KnowYourBite allows users to use the camera on their device to take a photo of any food ingredient label. The app utilizes Tesseract to read the text from the photo, break it down into individual ingredients, and display each one on a colorful card. The phi-3 quantized model then gives a brief description of each ingredient and a health score, so that everyone can understand exactly what they are putting into their bodies.

How we built it

We have created KnowYourBite with Python as our primary programming language. To take and read images of ingredient labels, we used OpenCV, Tesseract, and a wrapper called Pytesseract. A custom parser was then used to parse the text and get individual ingredients. For the User Interface (UI), we used Kivy with KivyMD. The AI model used in the application was Phi-3, which runs on LLaMA cpp, and we stored our data in a json file, which contains information from Open Food Facts.

Challenges we ran into

Some of the challenges we ran into were prompting the AI and testing the app. In our app, we use a small model, so that means we had to be very specific and word our prompts in a certain way to get the response we wanted. Otherwise, the AI would respond with the part of the question in the answer, create questions of its own in the answers, or respond with prompt template tags. When testing the app, we found it difficult to include ingredients that would return a variety of different responses to fully test the AI summaries and OCR.

Accomplishments that we're proud of

We built a fully working mobile app as high schoolers. We also learnt multiple new technologies like Tesseract and LLaMA cpp. We made sure the UI looked fun and creative to match the theme which we feel is actually great.

What we learned

We learned a lot about the data sources and regulations that are related to food safety. We also learned about the styling and User Interface (UI) for applications and some prompt engineering.

What's next for KnowYourBite

We want to add a feature where you can take a picture of the product and input the store location, and it will try to recommend healthier versions of the product. We would also like to add something where you could input your allergies and it would modify the risk factors depending upon if it contains the allergen.