PillSense

Inspiration

An estimated 125,000 Americans die each year due to medication non-adherence, according to the American Heart Association. For elderly patients and people living with dementia, remembering medication can be a daily struggle.

Traditional pill dispensers remind patients to take medication, but they cannot reliably determine whether someone actually took it.

We built PillSense, an AI-powered smart medication dispenser that combines robotics, computer vision, and autonomous agents to help patients stay on track and give caregivers peace of mind.

What it does

PillSense combines five core features:

Smart Dispenser: A custom 3D-printed, Arduino-powered rotating pill dispenser with Bluetooth control and LED status indicators.
Computer Vision: Uses OpenCV and MediaPipe to recognize hand-to-mouth gestures and provide additional context about medication activity.
Agentic AI: Claude investigates missed doses using medication schedules, device activity, and patient reports, then recommends caregiver follow-up.
Voice Companion: Cartesia-powered voice interaction helps patients manage reminders and report medication activity.
Caregiver Dashboard: Real-time medication tracking, missed-dose alerts, supply forecasting, and caregiver-reviewed refill workflows.

How we built it

Hardware: Arduino UNO R4 WiFi, 28BYJ-48 stepper motor, ULN2003 driver, custom 3D-printed carousel, and three status LEDs.
Computer Vision: Python, OpenCV, and MediaPipe for real-time gesture detection.
Frontend: Next.js, React, TypeScript, and Tailwind CSS.
Backend: Convex for real-time synchronization and Clerk for authentication.
AI Agents: Anthropic Claude, Fetch.ai uAgents, Agentverse, and ASI:One for intelligent medication investigations.
Voice AI: Cartesia for speech recognition and text-to-speech.

We connected our physical dispenser to the web application using Bluetooth Low Energy (BLE).

Challenges we ran into

Designing and 3D-printing a functional rotating dispenser under hackathon time constraints.
Debugging stepper motor positioning, Arduino wiring, and BLE communication.
Integrating multiple AI services with real-time hardware and backend events.
Distinguishing observed medication activity from confirmed ingestion.
Ensuring AI agents could investigate missed doses without making unsafe medical decisions.

Accomplishments that we're proud of

Built and tested a custom physical medication dispenser with working motor control and LED feedback.
Connected Arduino hardware to our web application through BLE.
Developed a real-time computer vision observation system.
Successfully demonstrated a Fetch.ai and Claude agent workflow for missed-dose investigation.
Created an integrated patient and caregiver experience with voice assistance, reminders, and refill intelligence.

What we learned

We learned how to integrate embedded hardware, computer vision, real-time databases, and autonomous AI agents into one system.

More importantly, we discovered that building responsible healthcare technology means understanding uncertainty. A motor completing its movement doesn't prove a patient swallowed their medication, and AI should recognize that distinction.