Inspiration
I was inspired by current health technology, where the doctor's just put their phone on a stand and record the entire appointment. I wanted to expand on that idea and use agents to improve it and reduce doctor burnout.
What it does
It is an AI scribe that records appointments, summarizes it, and organizes it into notes for the doctor and patient. It uses agents to be able to do this. The AssemblyAI agent converts the speech into text, and the Gemini agent uses that transcription and summarizes it into notes for the doctor. Then, the doctor can look over the notes, edit if needed, and approve it to send to the patient. That's all the doctor has to do. Hit record, finish, and approve. This goes to the patient app, where the patient can view the notes, and it automatically sets reminders tailored for the specific patient so they stay on track with their personalized healthcare plan. This reminders can be dietary plans, reminders for workouts, or when to take medicine and how much, all depending on what advice your doctor gave you. You can change the reading level and language, so there will be no language barriers between the patient and doctor, and advanced medical terms that the patient doesn't know will be translated into simpler words. Also, with the reminder and calendar feature, it keeps the patient on track because most times, they would forget everything the doctor told them. This app solves the problems of language barriers, patient forgetfulness, and doctor burnout at the same time.
How I built it
I built it in VS Code with the backend coded in python and the frontend in HTML. I also used flask to build it. I got free API keys from Google AI Studio and Assembly AI, and I implemented them into the code so they would be able to convert the audio into text, and then summarize it into readable notes. I built the first prototype myself, then used Claude to look over, fix anything, and add anything I didn't know how to do such as language translation, adding patient accounts, etc.
Challenges I ran into
I faced a lot of challenges with the functionality of the agents, because I would get a lot of errors from the agent. Also, getting the html to actually work was another hassle, because this is my first time working with html and tailwind-css. Also, sometimes the agent wouldn't respond and I had to tweak the code again and use different models to make it work. However, after debugging, I finally got it to work.
Accomplishments that I'm proud of
I am proud that I was able to learn how AI agents work and how cloud APIs work. I'm more of a hardware guy, this was more of a side quest, so I was surprised I was able to pull it off. I am also proud that I was able to teach myself HTML, because it was a language I had little experience in but still managed to pull it off for this project.
What I learned
I learned how cloud APIs work when implementing my agents, and I also learned how databases work when I was making the patient accounts. I also learned some HTML in order to make the frontend of the project.
What's next for HealthSync
So far, there is only a patient log in screen. However, clinics have lots of doctors too. I will eventually add a doctor's login for the doctor portal, so the whole thing will become a functional system. The patient portal has much more features than the doctor side, so I might add some more features to the doctor portal. I will also consider making this an actual ios app instead of just a web app using a mac and writing in Swift, so it will feel more like an actual product than a prototype. I was also thinking of adding a private patient and doctor chat so the patient can consult to their doctor if they need to talk. I also wanted to add a place where the patient can log things on their own, like sleep or weight, and the agent can help create advice for that and add to your reminders and goals.