ThuocTracker

Inspiration

"Thuốc" is Vietnamese for "medicine."

Many of us have grandparents who take several pills a day. They forget a dose, or they forget they already took one and take it twice. Their doctor writes the prescription and never finds out whether it was taken. Their family usually hears about a problem only once it has become serious.

We wanted to close that loop for elderly and Alzheimer's patients without asking them to learn a smartphone app.

What It Does

ThuocTracker is a smart medicine cabinet linked to a doctor dashboard and a mobile app for patients, families, and doctors.

Pill time: The cabinet's touchscreen flashes, the right door opens with a servo, and a voice says, "Time for your blue pill." The patient's phone and the family's phones both get a push notification, because many older patients don't use a phone. The reminder repeats every 10 minutes.
Checking whether a pill was taken: The camera photographs the slot before and after. If there is exactly one pill fewer, the dose counts as taken. The patient can also tap "I took it."
Missed doses: If no pill is taken within 30 minutes, the family and doctor are alerted. The patient gets a message that only informs them, never one saying "take it now," because ThuocTracker never changes a dose.
Double doses: If the patient opens the same dose twice, the screen warns, "You already took this one."
Voice questions: The patient can ask, "Did I take my pill today?" The answer comes only from the real dose log. If the answer isn't there, it says, "Ask your doctor."
Label scanning: A doctor or family member photographs a medicine bottle. AI fills in the name, dose, times, quantity, and expiration date. A person checks every field before it is saved.
Alerts: Low pills (fewer than 5 left) and medicine expiring within 7 days trigger notifications to the family and doctor.
Doctor dashboard: Doctors can view each patient's taken and missed doses and chat with each patient's family.

How We Built It

Cabinet: A Raspberry Pi 5 with 7 slots, servo-driven doors, and a touchscreen running our Next.js cabinet page in Chromium kiosk mode. An ESP32-S3 handles the speaker (MAX98357A) and microphone (INMP441) and communicates with the Pi over USB.
Backend: FastAPI running on a laptop with a Tiger Data (PostgreSQL) database. The backend makes decisions: a scheduler runs every 2 seconds and handles reminders, missed doses, and alerts. Several laptops can run the backend using one shared database. A lease row determines which one runs the scheduler, and if that laptop stops, another takes over within 5 seconds.
AI: Gemma 4 runs locally through Ollama to answer voice questions and read labels, with Gemini as a fallback for label reading and pill counting. ElevenLabs generates speech. No AI runs on the Pi's 1 GB of RAM. Instead, it sends photos and recordings to the backend.
Web: A Next.js doctor dashboard and cabinet touchscreen interface.
Mobile: An Expo (React Native) app for patients, families, and doctors, with push notifications through Expo and Firebase.
Networking: Everything runs on a 2.4 GHz phone hotspot.

Challenges We Ran Into

Campus WiFi restrictions: Campus WiFi blocks traffic between devices, so the Pi and backend couldn't reach each other. We moved everything onto a phone hotspot.
Running multiple backends: State we had kept in memory, including reminder state and photo counts, caused duplicate alerts when multiple backends ran simultaneously. We moved all shared state into the database, made each scheduler step a conditional write, and allowed only one dose per medicine per pill time.
ESP32 audio unit: USB endpoint limits on the ESP32-S3, serial port issues, and the board reading its own echoed lines all required fixes.
Unreliable local model output: Local models sometimes returned malformed times or values such as "60 tablets" in the wrong field. We clean the output, and anything unreadable is left empty for a person to fill in rather than being made up.
Expo development: An npm install quietly downgraded the Expo SDK, and Android required a development build.

Accomplishments We're Proud Of

Working end-to-end hardware: The full loop works on real hardware: pill time arrives, the cabinet responds, the system checks whether the pill was taken, and the family and doctor receive an alert when necessary.
Safety-first design: AI only reminds and alerts. It never changes a dose or makes up an answer.
Local AI processing: Voice questions and label reading run on a local model, helping keep patient health data local.
Backup mechanisms: Every feature has a fallback for the demo:
- If the servo fails, the patient opens the door manually.
- If camera-based pill counting fails, the patient taps "I took it."
- If WiFi drops, the Pi plays stored sounds.
Team coordination: Four people working on four different parts of the system stayed synchronized through a shared status board and API contract in the repository.

What We Learned

Design for people who won't use a phone. That's why family members receive the same reminders as patients.
Store shared state in the database, not in a server's memory, especially when more than one machine is involved.
Small local models such as Gemma 4 can handle focused tasks effectively when their outputs are validated and cleaned.
Agree on the API contract early. Establishing it in the first hour saved us integration work.

What's Next for ThuocTracker

Pharmacy integration: Connect with real pharmacies so refills can be ordered automatically when supplies run low.
Adherence analytics: Help doctors identify missed-dose patterns over weeks and generate adherence reports before appointments.
Multilingual voice support: Expand voice interaction to more languages, starting with Vietnamese.
Hardware improvements: Build a smaller enclosure and add battery backup.
Real-world testing: Test ThuocTracker with families, caregivers, and elderly patients to improve reliability and usability.