Smart Med Assistant

user prescription dashboard
sign-up form
add prescription form
add prescription with OCR

Inspiration

We were inspired to build this project because we wanted to address a common, everyday challenge that was simple to approach but affected a large group of people. Pill management was a natural choice, as we have all struggled to stay on top of prescriptions—especially the many high-allergy individuals in our group. We wanted to make this task easier for everyone through an intuitive and efficient application.

What it does

Smart Med Assistant is a prescription-tracking full-stack application that helps users manage their medications. Users can sign up and track their prescriptions through individual accounts. On their dashboard, they have clear records of each medication's dosage, frequency, and duration.

New prescriptions can be added manually via a form or by scanning a pill bottle, leveraging the Google Vision API and OCR to extract text. To streamline the process, we utilized the Gemini API to categorize and structure the extracted text into the appropriate form fields automatically.

Each time a user adds a new prescription, we call the Gemma 3 LLM to check if the medication is safe to take alongside their existing prescriptions. Additionally, the model provides insights into potential side effects the user should be aware of before starting the medication. This allows users to make informed decisions about their health.

Users can also delete prescriptions as needed. Lastly, we implemented a Google notification system that enables users to register for medication reminders through their Google Calendar. While this feature functioned for most of the time, it requires further refinement and time to be fully-functional.

How we built it

We used Flask for the backend and React and Bootstrap for the frontend. We integrated multiple AI-powered APIs, including Gemma and Gemini LLMs, as well as the Google Vision API for extracting and structuring prescription details from scanned pill bottles. A significant portion of our development time was dedicated to debugging and refining the AI-driven components, so we did not do extensive design.

Challenges we ran into

This was the first hackathon for three of the four team members, making full-stack development a major learning curve. Our biggest challenge was implementing a complex, AI-integrated full-stack application while efficiently dividing tasks among team members with varying levels of experience.

Additionally, the Google Vision API initially struggled with extracting text from pill bottles, requiring extensive refinement and debugging. We also faced challenges integrating the Google Calendar API, as setting up event creation across multiple user accounts proved trickier than expected. Given these obstacles, we had to scale down some of our planned features to focus on ensuring core functionalities worked smoothly.

Accomplishments that we're proud of

We are especially proud of successfully integrating Google Vision API and Gemma LLM into our system. Extracting clear text from pill bottles was a difficult problem that required persistent debugging and testing, but we stuck with it and made it work.

Additionally, the Gemma integration provides valuable insights by checking drug interactions, alerting users to potential risks, and offering information on possible side effects, helping users stay informed and safe when managing their prescriptions.

What we learned

Since most of our team members were beginners, this project was an intensive learning experience. We gained hands-on knowledge of Flask, JavaScript, APIs, and databases, which were new concepts for many of us. This project challenged us to think critically about full-stack development and AI integration in real-world applications.

What's next

Our next goal is to implement a chatbot feature that allows users to ask questions about medications, including side effects, interactions, and over-the-counter alternatives. Many people have general medical questions but lack easy access to a doctor, so we believe an AI-powered chatbot could be a valuable resource for users who need quick and reliable medication advice. Additionally, we plan to refine the Google Calendar integration to ensure notifications function seamlessly across all user accounts.