-
-
Start with a bottle, your medicines and your goal. Build a source-linked brief for your pharmacist.
-
Language AI connects "my cholesterol pill" to a suggested statin group and the relevant NIH source.
-
Measured medicine understanding and label matching, plus a separate research benchmark and Ray's AI-on/off comparison.
-
Medicine-specific cards preserve NIH's original wording, source links and questions to discuss with a pharmacist.
-
On-device AI ranks research for the bottle and goal, with original study excerpts beside the product label.
-
Combine NIH product records, reviewed photo text, medicine names or descriptions, and your supplement goal.
-
Ambiguous ginseng species and undisclosed blend amounts become specific questions for the pharmacist.
-
A QR link opens the source-linked brief on the pharmacist's screen, ready to review together.
Inspiration
“My cholesterol pill” is meaningful to the person taking it, but it is missing from an exact-name medicine lookup. Supplement labels create another gap: unfamiliar plant names, blends with undisclosed ingredient amounts, and the same ingredient appearing in several bottles.
We built FoliaTrace AI for older adults, caregivers and pharmacists working through these details together. The goal is to turn a bottle and a person’s own words into a useful starting point for a pharmacy consultation, with the evidence attached.
What it does
FoliaTrace AI creates a personalized, source-linked pharmacist brief:
- Add a bottle through live NIH label search, pasted ingredients or a photo with editable text recognition.
- Describe the medicines using names, brands, typos or everyday descriptions. Add the reason for taking the supplement and relevant circumstances.
- Build the brief to see medicine-group suggestions, NIH source cards, ambiguous plant identities, blend-dose questions and ingredients repeated across bottles.
- Explore research ranked for each herb and the entered goal, with the study’s population, preparation and comparator excerpts beside the label.
- Share with a pharmacist through a QR link, printing or a download with selected personal details.
Ray demonstrates the core AI contribution. With language AI disabled, “my cholesterol pill” remains unresolved. With it enabled, the system suggests a statin group and adds the relevant NIH red yeast rice card. The original wording, suggested group and question to confirm the exact medicine stay visible.
How we built it
The browser combines two neural models with a source-matching engine:
- Medicine language:
all-MiniLM-L6-v2sentence embeddings compare unresolved descriptions with medicine-group prototypes after exact-name, brand and typo matching. - Personal research: BM25 retrieves candidate studies from 423 open-access abstracts. The
ms-marco-MiniLM-L6-v2cross-encoder reranks them for the herb and user goal. A relevance threshold controls which leads appear. - Source cards: deterministic rules select from 74 verified NIH passages covering 25 herbs. Research excerpts retain the original abstract wording.
- Photo input: Tesseract.js reads label text locally for the user to review.
ONNX Runtime Web, WebAssembly and browser workers run model inference on the device. NIH receives the separate product-search query or label ID. QR handoff carries the case inputs in a compressed URL fragment. The full Python/FastAPI/SQLite application adds consultation history and pharmacist reading decisions.
Hao developed the product with substantial Codex and Claude assistance, evolving an earlier EthnoHack literature workspace into this bottle-to-pharmacist workflow.
Challenges we ran into
Understanding a description without inventing a drug identity. We use confidence and similarity-margin thresholds, preserve the person’s original words and expose the suggested group for confirmation.
Finding studies for the actual question. A paper mentioning an herb may concern a different preparation, goal or even plant cultivation. We split goals, filter candidates, apply neural reranking and retain literal study context.
Preserving the details that matter. Ambiguous species and undisclosed blend amounts become explicit questions. Export controls recompute the brief so removing medicines also removes their names from cards and personal questions.
Accomplishments that we're proud of
We completed four browser workflows: Mei’s warfarin and water-pill description; Sam’s sertraline and birth-control combination; Ray’s cholesterol-pill description and overlapping supplements; and a live NIH label plus photographed bottle through AI interpretation, research, fresh-browser QR handoff and selective download. The example profiles and photo label are fictional; inputs and observed results are included with the source.
- Medicine understanding: across 54 synthetic English phrases, 37 of 39 medicine descriptions reached the correct group, zero reached a wrong group and two remained unassigned. All 15 negative or other descriptions remained unassigned.
- Label matching: 96.3% exact-species precision on 789 held-out NIH product labels.
- Research retrieval: in a separate full-local SciFact benchmark, neural reranking improved nDCG@10 from 0.812 with BM25 to 0.866 across 188 annotated claims and 5,183 abstracts.
What we learned
The most useful AI contribution was recovering meaning from incomplete input. Ray’s description changed which evidence reached the brief. Research ranking needed the same attention to context: the bottle, goal and study preparation all mattered. Keeping these connections inspectable made the result easier to discuss and evaluate.
What's next for FoliaTrace AI
Our next step is pharmacist review and usability sessions with older adults and caregivers. We will use those findings to refine medicine-group confirmation, expand herb and medicine coverage, and develop multilingual briefs and barcode-assisted product lookup.
Built With
- bm25
- cross-encoder
- css3
- fastapi
- html5
- huggingface
- javascript
- minilm
- nih-dsld
- onnxruntime-web
- python
- sqlite
- tesseract.js
- webassembly