PillCheck

Photo-verified medication tracking for elderly patients: OpenCV 5 sees which blister pocket was emptied, and an AI agent records the dose or alerts the family.

Comment

Inspiration

In India most tablets come in blister strips, and many elderly parents live apart from the children who worry about their medicines. Reminder apps ask the patient to tap "taken", which proves nothing. A blister strip already records the truth: an empty pocket. PillCheck reads it.

What it does

After each dose the patient takes one photo of the strip. PillCheck:

  • checks the photo (blur, light, glare, a re-sent old photo),
  • recognises which medicine strip it is and flags the wrong one,
  • aligns it to the setup photo and decides pocket by pocket which tablets are gone,
  • reads the expiry date printed on the strip,
  • and an agent turns that evidence into actions: record the dose, remind, ask for a retake, alert the caregiver (missed dose, possible double dose, wrong medicine, expired strip), or ask a human to decide.

The caregiver sees only what needs attention, with the evidence photo and the agent's reasoning. Hindi and English.

How we built it

  • OpenCV 5: cv.ALIKED keypoints and cv.LightGlueMatcher on the new DNN engine, findHomography with USAC_MAGSAC, GrabCut strip segmentation, HoughCircles (HOUGH_GRADIENT_ALT) + fitEllipse for pocket finding, Lab + SSIM pocket comparison, img_hash.PHash for re-used photos, and the DNN text models (PP-OCRv3 detector + CRNN) for the expiry date.
  • Agent: facts (which dose a pill belongs to, what is overdue) are computed in code; a planner (rules, or Amazon Bedrock Nova Pro) picks from 6 tools; a policy layer blocks dosing advice and untrusted dose records; a safety floor runs any mandatory action the planner skipped. Every run is stored as a trace.
  • AWS: Lambda (arm64 container) behind a Function URL, DynamoDB, S3, SNS, EventBridge Scheduler (15-minute reminder checks), Bedrock, CloudWatch, all in one CloudFormation template with spending caps.
  • Frontend: React + TypeScript, designed in Figma first.

Results

On 211 real phone photos of Indian strips (public MedCheck dataset, re-shot with random camera changes): 77% correct, 19% asked for a retake, 3% wrong: 96% accurate when it answers, median 2.3 s per check. Same strip: 199–860 matched points; different strips: ≤10. Full method and failure cases are in REPORT.md.

Challenges

  • Our synthetic benchmark said 100%. Real photos immediately broke it: missed doses on silver foil, strips on black cloth rejected as "too dark", real strips not recognised. Each fix was re-measured on real photos.
  • We found that OpenCV 5.0.0's LightGlueMatcher crashes the whole process (SIGFPE) when two images share no matches, and added a guard so a wrong strip can't take the server down.
  • Opaque silver strips remain the hardest case: full and empty pockets differ only by crumpled foil.

What we learned

Test on real data early, design for "I'm not sure" (a retake is better than a wrong dose record), and keep the AI planner on top of deterministic facts and hard safety rules.

What's next

Real before/after photos from users, a live in-app viewfinder with framing guidance, and pharmacist refill integration.

Built With

Share this project:

Updates

Submission history