Medisense

Web app that lets you scan the bar code on medicine and ask an AI questions about it.

Comment

Inspiration

We discovered an API called OpenFDA, which provided data on various kinds of prescription drugs. We decided to build an app that would help users make safe choices with common over-the-counter medication without needing to add to the workload of their doctor or pharmacist.

What it does

Medisense scans the barcode on a medication’s packaging and uses the info it provides to answer the user’s questions about that medication. It can be easily augmented to work for other industries, such as the grocery industry.

How we built it

We used Flask for backend and FlowByte for frontend. Dynamsoft Barcode Reader scanned the bar codes, and FlowByte provided the speech to text input. We used exa-py to gather the most relevant search data on the medication and used a Llama 3 instance hosted on Groq to answer the user’s questions.

Challenges we ran into

Integrating flask with the JavaScript frontend was very challenging for this project, as was barcode scanning with our initial libraries. Also, the OpenFDA API ended up not providing data that was useful to us, so we had to pivot to exa-py fairly late in development. This pivot ended up greatly improving the product and enabling a new level of scalability and adaptability.

Accomplishments that we're proud of

  • We were able to finish the application after a late pivot away from our initial API
  • Got the barcode scanner working after extensive issues
  • Chained together two LLMs to produce a more advanced AI system

What we learned

  • Flask, FlowByte, and Tailwind
  • Groq API
  • exa-py
  • Barcode Reader Some more good lessons:
  • Read API documentation very closely!
  • Stay on your feet and be ready to adapt to disaster

What's next for Medisense

This proof of concept could eventually become a mobile app, or with some reactive changes, be adapted to work as a mobile website. It can easily be adapted to other industries, and could see a home as part of an app for grocery stores to help customers learn more about their products.

Built With

Share this project:

Updates