MedMate

MedMate is a multimodal AI medication companion using voice and vision to help patients identify pills, follow medication schedules, and keep caregivers informed through session summaries and alerts.

Comment

MedMate

Inspiration

MedMate was inspired by a very personal experience.

My grandmother takes several medications every day. Like many older adults, she organizes them in a weekly pill box. But sometimes she pauses with a tablet in her hand and asks a simple question:

“Is this the right one for now?”

For people managing multiple medications, that moment of uncertainty happens more often than we realize. It becomes even harder for people with limited vision, complex medication schedules, or memory challenges.

Medication mistakes lead to hundreds of thousands of preventable emergency visits every year.

We realized that what people often need is not another complicated health app — but a simple assistant they can talk to.

That idea became MedMate.

What it does

MedMate is a multimodal AI medication companion that helps users safely manage their medications using voice and vision.

Users can simply ask:

“What should I take now?”

Or show a pill to the camera and ask:

“Is this the right one?”

MedMate can:

  • Identify pills by reading imprints using the camera
  • Guide users through morning, afternoon, and night medication schedules
  • Ask important safety questions before ending a session
  • Generate automatic session summaries
  • Notify caregivers through email updates
  • Suggest nearby pharmacies if medication runs out

The goal is to make medication management feel simple, conversational, and safe.

How we built it

MedMate is powered by Google's Gemini Live API, enabling real-time multimodal interaction using both voice and vision.

The system allows users to speak naturally with the assistant while also showing medications through the camera.

Core components include:

  • Gemini Live API for real-time voice and vision interaction
  • A lightweight web interface for user registration and medication setup
  • A medication schedule system that tracks morning, afternoon, and night doses
  • Vision-based pill identification using pill imprints
  • Automatic session summaries and caregiver notifications

The assistant combines contextual data such as:

$$ Medication_Recommendation = f(Time, Schedule, Pill_Identification) $$

This allows MedMate to determine whether the pill shown matches the user's medication schedule at that specific moment.

Challenges we ran into

One of the biggest challenges was reliable pill identification.

Many medications look extremely similar, and incorrect identification could be dangerous. To address this, MedMate guides the user to show multiple sides of the pill so the imprint can be clearly read before making a decision.

Another challenge was creating a natural conversational experience. We wanted users to interact with MedMate like a real assistant — asking follow-up questions, interrupting responses, and switching between voice and camera seamlessly.

Balancing simplicity with healthcare safety was also critical. We introduced mandatory safety questions before ending sessions to ensure important information is never missed.

Accomplishments that we're proud of

We're proud that MedMate combines voice conversation, camera vision, and medication schedules into a single simple experience.

Within a short development cycle, we built a working system that can:

  • Identify medications using live camera input
  • Guide users through daily medication routines
  • Automatically generate caregiver updates
  • Suggest nearby pharmacies when medications run out

Most importantly, we built something that feels practical and meaningful for real people.

What we learned

Building MedMate taught us that the best AI solutions start with a human problem, not just a technical one.

On the technical side, we learned how powerful multimodal AI systems can be when voice and vision operate together in real time.

Using Gemini Live, we were able to create an interaction where users can speak naturally, show a pill to the camera, and receive immediate contextual guidance.

We also learned the importance of designing AI systems with safety constraints and verification steps, especially in healthcare use cases. Instead of allowing the system to guess, MedMate guides the user to show multiple sides of a pill to confirm the imprint before responding.

Finally, we learned how to combine real-time AI interaction, structured medication schedules, and automated caregiver notifications into a system that feels simple for users but still provides accountability for families.

What's next for MedMate

In the future, we plan to expand MedMate with additional capabilities such as:

  • Prescription scanning for automatic medication setup
  • Drug interaction warnings
  • Smart refill reminders and ordering
  • Integration with pharmacies and healthcare providers
  • Voice assistant integration for smart home devices

Our vision is simple:

No one should ever have to guess when it comes to their medication.

Built With

Share this project:

Updates

Submission history