Inspiration
Many people leave the hospital holding a paper they can't understand. It is full of short codes like "BID" and "HS." The writing may be hard to read. It may be in a language they don't speak. Some people can't read well at all.
Then they have to take medicine on their own, at the right time, in the right amount. Mistakes here can make people sick. We wanted to fix this. Anyone should be able to understand their medicine, whether or not they read well.
What it does
Pill-o-Talk takes a photo of a medical paper and turns it into simple steps that anyone can follow.
- In plain words, hard medical words become everyday words. "BID" becomes "twice a day."
- Your reading level. Choose very simple (3rd grade), simple (5th grade) or a little more detail (8th grade).
- Your language. The whole app works in 10 languages, including Hindi, Bengali, Spanish, Arabic and Swahili.
- A daily schedule. It shows what to take in the morning, afternoon, evening and night, with small pictures, so you can follow it without reading much.
- It talks to you. Press a button and it reads the instructions out loud in a calm voice. You can also download the audio and send it to family on WhatsApp.
- Ask out loud. Don't want to type? Say your question, like "Can I take this on an empty stomach?", and hear the answer.
- Reminders: Add daily medicine reminders to your phone calendar.
- Safe by design. It checks that every medicine and dose still matches the paper. If something is not clear, it says so and tells you to ask a doctor or pharmacist. It never guesses.
How we built it
- Gemini API reads the photo, finds each medicine, and rewrites the instructions in plain words in the language you choose. A second check compares the new version with the original paper. Gemini also answers spoken questions using only your paper.
- ElevenLabs turns the instructions into natural speech and turns your spoken question into text.
- Streamlit is the app you see, and it runs on Streamlit Community Cloud. The code is on GitHub.
- The calendar reminders are a standard file that works with Google Calendar, Apple Calendar and Outlook.
Challenges we ran into
- Papers often don't say when to take a medicine. We made sure the app never guesses. It asks you or a family member to pick the time from a simple list.
- Names and doses must never change. Translation can go wrong, so we keep the medicine names and numbers exactly right and add a second check.
- Keeping it simple. It was hard to remove medical words without removing safety. We tested many versions to find wording that is easy to follow and still correct.
- Cost and speed. We saved results so the app doesn't repeat work, and audio is only made when you press the button.
Accomplishments that we're proud of
- Someone who can't read the paper can still hear what to do, in their own language.
- The app says "I'm not sure, ask your pharmacist" instead of guessing.
- Even the buttons and labels are translated, not just the instructions.
What we learned
- The simplest answer is often the hardest to build.
- Sound matters. Many people understand better when they hear instructions than when they read them.
- In health care, being honest about what you don't know is better than sounding sure.
What's next for Pill-o-Talk
- More languages and local voices.
- Warnings about medicines that should not be taken together.
- Large-print pages to print out.
- Reminders sent straight to your phone by text or WhatsApp.
- A way for a pharmacist to review the instructions.
*Pill-o-Talk is a helper, not a doctor. Always check your medicine and doses with a doctor or pharmacist. In an emergency, call your local emergency number.
