AR HealthFirst

This is the Webpage of our Project , Publicly deployed using AWS EC2

Inspiration

1.5 million medication errors happen every year — not because patients are careless, but because they have no real-time safety information when they actually need it: the moment they pick up a medicine. We wanted to fix that with a single camera point.

What it does

AR HealthFirst lets you point your phone at any medicine strip and get a plain-English safety verdict in under 3 seconds — overlaid directly on your camera. It tells you how to take it, whether it dangerously interacts with your current medications, and one practical reminder. Green = safe. Amber = caution. Red = danger.

How we built it

A 4-step Nova-powered pipeline:

Nova Lite reads the medicine label image and extracts drug name, dosage, and warnings as structured JSON Nova Multimodal Embeddings semantically matches the extracted name (including brand names and misspellings) against our drug database using FAISS vector search Interaction engine cross-checks against the patient's current medication profile via JWT-authenticated session Nova 2 Lite reasons over all signals and generates the 3-line plain-English AR verdict

Stack: Amazon Bedrock · FastAPI · FAISS · WebAR · Python

Challenges we ran into

Real-world medicine labels are messy — glossy reflections, small fonts, partial occlusion, multilingual text. Getting Nova Lite to return clean structured JSON consistently across all these conditions took significant prompt engineering. Latency was also a challenge — we parallelized the embedding call and profile retrieval to keep the full pipeline under 3 seconds.

Accomplishments that we're proud of

Full end-to-end multimodal pipeline built in 48 hours Nova Lite accurately reads real medicine labels across varied conditions Brand-to-generic drug matching with zero lookup tables — pure semantic embeddings Zero cloud image storage — photos processed in-memory, never persisted

What we learned

Nova Lite is far more capable than a standard vision model for structured extraction tasks when given the right medical context in the prompt. Nova Multimodal Embeddings resolved brand-to-generic drug name matching out of the box — no fine-tuning needed. That alone saved us a full day of work.

What's next for AR HealthFirst

Expand drug database to 50,000+ pairs using open DrugBank data Add Nova 2 Sonic for a voice-first verdict mode for visually impaired users Multi-language verdict support for regional markets Pilot with Indian pharmacy platforms (1mg, PharmEasy) — 1.4B potential users