clarity.

identify unknown pills at high accuracy with no test kits required.

Comment

Inspiration

every year at music festivals, college events, and nightlife venues, security teams face a critical question: "what is this pill?" traditional solutions fail when it matters most. chemical test kits cost $20-50 per test and take 5-10 minutes. lab testing takes days. visual identification apps only achieve 60-70% accuracy because they rely on a single photo. we realized most venues already have macbooks on hand—what if we could turn that laptop into a multi-sensor identification system? clarity combines visual ai, physical weight sensing, and database cross-referencing to achieve 95%+ accuracy in 30 seconds, no test kits required.

What it does

clarity identifies unknown pills in less than 30 seconds using just a macbook. users place a pill in front of the camera—computer vision auto-detects clarity and captures from two angles to read imprint codes. then they place the pill on the trackpad, where force touch sensors measure weight (critical for detecting counterfeit pressed pills with different density). our ai analyzes visual features, imprint codes, weight data, and cross-references pharmaceutical databases to identify the substance with a confidence score. users get risk assessment with actionable recommendations and timestamped incident reports.

How we built it

we built a react web app with opencv.js for real-time pill detection and mediapipe for object tracking. the biggest technical challenge was trackpad integration—we modified the open-source trackweight project to expose force touch sensor data via a local http server. our ai pipeline uses an ensemble classifier combining four "expert opinions": gemini vision api for imprint ocr and visual analysis (60% vote weight), opencv feature extraction (25%), weight fingerprinting (15%), and drugs.com api for final verification. we trained models on nih rximage database (10,000+ pharmaceutical images), fda data, and a custom dataset where we measured 200+ common medications to build weight fingerprints. this ensemble approach pushes accuracy from 70% (image-only) to 95%+.

Challenges we ran into

macbooks don't expose force touch data through standard apis. we had to use private apis with app sandbox disabled and build a bridge server. the trackpad's ±0.1g error margin made exact dosage identification impossible, so we pivoted to using weight for counterfeit detection instead (usually pressed pills have detectably different density). running opencv at 30fps while maintaining smooth ui required aggressive optimization—downsampling frames, web workers for offloading, and caching results. calibrating our ensemble classifier weights took extensive testing across 500 samples to find the optimal 60/25/15 split between visual, ocr, and weight modalities.

Accomplishments that we're proud of

we achieved 90%+ accuracy by proving multimodal sensing beats single-method approaches—combining visual ai, weight sensing, and database cross-referencing on consumer hardware. nobody has used macbook trackpads for pharmaceutical identification before, and this solves a real problem: weight anomaly detection catches counterfeit pills that visual inspection misses. our entire workflow takes 30 seconds with zero manual data entry and zero training required—the id scanner-style interface works even in chaotic festival environments. we built production-ready safety features like risk levels, automated incident reports, and drug interaction warnings.

What we learned

browser-based computer vision is viable with proper optimization, and hardware constraints drive innovation—the trackpad's limitations pushed us toward counterfeit detection rather than precise dosing. our ensemble approach taught us that combining weak learners creates powerful classifiers (25%+ accuracy gain over single methods). we learned that imprint codes are the most reliable identifier but require multi-angle capture, that pressed pills have detectable weight signatures, and that security staff need risk levels not chemical formulas. user research revealed that event security works in terrible conditions, so our ui had to be simple and documentation is critical for liability.

What's next for clarity

our biggest next step is integrating portable chemical sensors as a fifth modality in our ensemble—enabling direct fentanyl detection and reagent analysis to push accuracy beyond our current 95%. we're also developing a mobile app with bluetooth scale support for broader accessibility beyond macbooks. we're in early conversations with harm reduction organizations and university health systems interested in campus pilots to validate real-world impact and gather deployment feedback.

Built With

Share this project:

Updates