Pillow

A medicine log you can talk to. Say "Mom took her 8 AM pill" and the whole family sees it, with a spoken reminder 10 minutes before each dose. Built on MCP for Alexa+

Comment

Inspiration

My grandparents often forgot to take their medicines on time. Now my dad is getting older, and he forgets too. Reminders from phone apps didn't help, because the people who most need them are the ones least comfortable with small screens and menus. We wanted something they could simply talk to, that the whole family could see and trust.

What it does

Pillow is a spoken medicine log for a whole family.

  • A caregiver creates a private household and adds each person's medicines, times and days.
  • Anyone can then say or type things like "Mom took her 8 AM medicine", "What's due now?" or "How did today go?", and everyone sees the same record.
  • Medicines can be added, changed or removed by chat, for example "Change Raj's dopamine to 9 AM on weekdays".
  • Ten minutes before a dose, Pillow shows and speaks a reminder, such as "Mom's metformin is due at 8 AM, in 10 minutes."
  • Overdue doses show a clear alert for the family.
  • It's a household log, not medical advice. It never diagnoses or changes a prescription.

How we built it

  • MCP server: the core is a self-hosted MCP server (protocol 2025-11-25, Streamable HTTP) with 11 tools: due, log, status, summary, register, add, change, remove, list, clear, and remove person. The web app calls those same tools, the way Alexa+ would.
  • Backend: Python (Starlette) with SQLite, private households, and per-day schedules.
  • Frontend: a React app (Vite) with a calm green design, modals for people and medicines, and a simple voice view for older adults.
  • Voice: the microphone feeds a chain of transcribers: Gemini 3.5 Transcribe, then ElevenLabs Scribe v2, then local Whisper. If one is out of quota or offline, the next takes over.
  • Choosing the action: a Gemini model picks the closest tool and fills in the arguments. Its reply is checked against the tool definitions, and any bad reply falls back to built-in rules.
  • AWS, in the code today (off by default): Amazon Bedrock for choosing actions and Amazon Cognito for sign-in. One switch, AWS_ENABLED=1, connects them.

Challenges we ran into

  • Hearing medicine names. Speech recognition turned "dopamine" into "lo for me" and a new name into "Rajester". We added vocabulary hints, mishearing repairs and an LLM step to choose the action, and we kept the person-adding path so names are known before voice is used for medicines.
  • Quotas and credits. Our Gemini live-transcription service rejected us because the project's credits ran out. We built the fallback chain so a failed or out-of-credit call never blocks the user, and Gemini pauses itself for 10 minutes after a quota error.
  • Honest feedback on the screen. Our first version said "Listening…" before the microphone was ready. We changed it to show "Starting microphone…" first, then added a keep-warm microphone so Speak starts at once.
  • A calmer interface. The first design was crowded. We rebuilt it with a dark sidebar, one big chat card and a plain Today column, using shades of green.
  • Not overclaiming. Real Alexa delivery and the AWS deployment aren't built yet, and we labelled them that way throughout.

Accomplishments that we're proud of

  • A complete, working loop: speak, transcribe, choose the action, change the shared record, and see it update for the family.
  • Reminders that look ahead, not only react to a missed dose.
  • A voice setup that keeps working when a cloud service fails.
  • A design simple enough that we'd be comfortable showing it to our own parents.

What we learned

  • Voice products live or die on small details: how fast the microphone starts, what the screen says while it listens, and how names are spelled.
  • An LLM works well as a router for a fixed set of actions, but only if its output is checked strictly and there's a safe fallback.
  • Testing with real voices, accents and noisy rooms matters more than testing with clean sentences.

What's next for Pillow

  • Photo to medicines: snap a prescription or a pill box and have the medicines and times added automatically (Textract and S3 on AWS).
  • Family accounts: let more family members join the same household with their own sign-in, so anyone's dose is logged for everyone.
  • Real Alexa reminders: proactive spoken reminders on every Echo, ten minutes before each dose, scheduled with EventBridge Scheduler and delivered through SNS and Alexa+.

Built With

Share this project:

Updates

Submission history