-
-
3D model of HealthBox
-
Front close up
-
Final HealthBox
-
-
-
Inspiration
A medication reminder can tell someone it is time to take a pill, but it leaves important questions unanswered: Is this the intended person? Does the dose contain the expected pills? Was it collected, or is it still sitting in the tray?
We wanted to connect reminders with practical support and accountability. Forgetting a dose, dismissing a notification, and becoming unsure whether you already took something are different problems. We wanted Healthbox to help people understand what is happening and what to do next.
Our goal was to make the routine more approachable through a physical pillbox, camera-assisted checks, and a web app that guides the person one step at a time.
What it does
Healthbox is a smart pillbox prototype that combines ongoing reminders, guided dispensing, visual checks, and a record of dose activity.
Its main features include:
- Scheduled reminders and follow-ups: Audible reminders, snooze controls, and follow-up prompts help users respond when a dose is due and revisit an unresolved dose.
- Camera-assisted recipient checks: The application compares the person in view with a locally enrolled reference before progressing through the dispensing workflow.
- Visible pill checks: The software checks the empty tray, compares visible pill classes and counts with the confirmed plan, and checks for collection afterward.
- Dose history and visual evidence: Users can review recorded events and annotated tray images to understand what the camera checked.
- As-needed medication support: Users can confirm minimum intervals and daily limits, which the application checks before accepting a request.
- AI assistance: Gemini helps read label information and answer questions about the confirmed schedule, device status, and next steps. Extracted information requires user review.
- Cartridge tracking: Guided refill checks and estimated remaining counts help users notice when a cartridge may be running low.
- Accessible guidance: Large controls, adjustable text, high contrast, a simplified view, multilingual support, and spoken instructions make the workflow easier to follow.
How we built it
We divided the work between two teammates: one focused on software and the other on hardware.
The application uses Python, FastAPI, SQLite, HTML, CSS, and JavaScript. SQLite stores confirmed plans, reminders, commands, observations, and dose history so that refreshing the browser does not reset the routine.
For computer vision, we used pretrained OpenCV face detection and recognition models, marker-based tray detection, and color-and-shape processing. We added calibration tools because lighting, pill appearance, and camera placement affect what the system can reliably distinguish. The camera architecture supports a laptop camera for the person and an ESP32-CAM above the tray.
Our hardware design uses an ESP-WROOM-32 controller, two 28BYJ-48 stepper motors with ULN2003 drivers, a shared servo-operated tray gate, and three microswitches. Two dispensing channels support two calibrated demo-token classes.
We integrated Gemini for label-reading assistance and contextual explanations. The assistant cannot change a confirmed plan, dispense a dose, or unlock the tray.
Challenges we ran into
Dispensing different pill shapes one at a time was a major mechanical challenge. Different shapes and sizes can cause jams, empty feeds, or multiple pills being released together. Rotor pockets, clearances, and the separation mechanism require careful adjustment. We also needed an independent camera check because completing a motor rotation does not guarantee that exactly one pill reached the tray. Broader pill compatibility remains something we need to validate through physical testing.
Making the camera feel responsive was another challenge. Repeated image requests made the preview choppy, so we introduced continuous streaming and separated lightweight positioning feedback from identity checks. We also mirrored the face preview to make positioning feel more natural.
Handling imperfect images required more than a working recognition model. Blur, shadows, touching objects, similar colors, and stale frames can produce misleading results. We added calibration, freshness checks, and understandable retry messages instead of treating every image as a confident answer.
Accomplishments that we're proud of
We brought reminders, visual checks, device-control logic, dose history, and contextual help into one guided experience.
We are especially proud of:
- Making accessibility part of the main workflow through spoken guidance, readable controls, and multilingual support.
- Giving users annotated tray evidence rather than only a pass-or-fail message.
- Keeping reminders and core device logic independent of the AI assistant.
- Building simulation tools that exercise wrong recipients, missing or extra tokens, interrupted connections, and repeated commands.
- Separating what the system observed from what the person reported, without claiming that a camera can prove ingestion.
What we learned
We learned that medication accountability requires more than sending notifications. People need a clear next step, useful feedback when something goes wrong, and a record that accurately reflects what happened.
We also learned how closely mechanical design, computer vision, and interface design depend on one another. A model cannot compensate for every poorly lit image, and software cannot make an unsuitable rotor reliably separate every pill shape.
What's next for Healthbox
Our next priorities are to expand physical dispensing tests, measure consistency for both cartridges, and refine the mechanism for a wider range of pill shapes and sizes.
We also want to test the interface with intended users, improve recognition under different lighting conditions, and refine reminder timing and follow-up messages based on their feedback.
Longer term, we want to explore standalone operation so Healthbox can work without an always-on laptop, while preserving clear guidance, local processing, and an honest record of each dose.
