Smart Health Record System

Managing health records shouldn't be stressful. I built an AI app that digitizes records using Agentic AI, offers a RAG-chatbot for diet, medication reminders, and a pharmacy locator.

Comment

Inspiration

Managing a stack of paper prescriptions and lab reports is a common struggle, especially in regions like Pakistan where physical records are still the norm. I observed patients frequently losing important medical history or struggling to find specific medicines in local pharmacies. This inspired me to create a digital bridge that simplifies healthcare management and ensures critical information is always accessible.

What it does

Smart Health Record is an AI-powered mobile application that digitizes healthcare management. It uses Agentic AI to automatically extract structured data—such as doctor names, medicine names, and dosages—from photos of handwritten or printed prescriptions. Beyond digitization, it features an Integrated Pharmacy Locator to find nearby medicines and is designed to include a RAG-based chatbot for personalized dietary advice and a medication reminder system.

How I built it

The application was built using Flutter for the frontend to ensure a smooth cross-platform experience. The backend is powered by Python and Flask, with Firebase handling secure authentication and cloud storage. We leveraged Google Gemini for multimodal AI processing, utilizing advanced Prompt Engineering to accurately interpret medical documents in both Urdu and English. But at first I try to train models on detection of prescriptions through traditional OCR and NLP but its accuracy was low.

Challenges we ran into

Initially, I attempted to train a custom machine learning model, but I faced significant hurdles due to the lack of access to large-scale, sensitive medical datasets. Early tests also showed that custom models struggled with the high variability of medical handwriting. I had to pivot from model training to using high-level APIs, which required extensive prompt optimization to achieve the necessary accuracy for healthcare data.

Accomplishments that we're proud of

I am particularly proud of successfully implementing a system that can accurately read and structure data from messy, handwritten prescriptions in multiple languages. Successfully integrating the Pharmacy Locator to provide real-world utility and building a secure, automated pipeline for medical record organization are also major milestones for the project.

What we learned

This project provided deep insights into handling real-world data variability and the ethical complexities of working with sensitive medical information. I learned how to implement AI fallback mechanisms to manage API limits and discovered that Prompt Engineering can sometimes be a more effective solution than training a model from scratch when data is scarce.

What's next for Smart Health Record System

The immediate next steps involve finalizing the Medication Reminder System and the Chronic Disease Dietary Suggestion Module. To ensure reliability, I plan to implement more robust fallback mechanisms within the API integration to handle quota limits effectively. While the project is currently a functional real-time application developed as a student, I aim to move beyond the academic scope and focus on marketing the system to evolve it into a professional business solution. I plan to integrate a Medicine Interaction Checker and a pharmacy-specific module to automate the entire prescription-to-delivery pipeline.

I have uploaded some screens here but implemented most and created demo video and only dietary, chatbot and reminders are static frontend screens. Rest is integrated with backend and stored in database.

Share this project:

Updates

Submission history