MedIntake

Scan
Audit-1
Audit-2

Inspiration

MedIntake was built to automate brown bag medication reviews, a routine intake protocol mostly done for elderly patients and people with complex chronic conditions when admitted to the hospital. Manually sorting through bags of pill bottles, OTC boxes, and supplements takes clinicians up to 45 minutes. We wanted to automate that entire intake in seconds, catching dangerous omissions, supplement risks, and adverse drug interactions before the first inpatient dose is ordered.

What it does

MedIntake is a cross-platform mobile clinical decision-support app that automates admission intake into a simple scan-and-audit workflow:

Mobile Census & Baseline Intake: Manages patient worklists with documented physiological flags (renal, hepatic, pregnancy) and a strict tri-state allergy model (documented, confirmed NKDA, or unknown).
Multi-Modal Packaging Intake: Clinicians photograph container labels (amber prescription vials, OTC boxes, or dietary supplement bottles).
Vision Extraction & Federal Verification: Multimodal models parse complex packaging text into structured data, preserving verbatim sigs while translating foreign directions (such as Spanish) into standardized English.
Federal Verification: Prescriptions and OTCs are matched against openFDA for exact NDCs, active ingredients, and black-box warnings (with direct monograph links constructed via spl_set_id); vitamins and botanicals are verified against NIH DSLD.
Reconciliation & Safety Audit: Instead of a simple inventory list, the engine compares scanned containers against the patient’s documented baseline to catch omitted maintenance medications and unrecorded OTCs. It delivers actionable clinical directives (CONTINUE, HOLD, CLARIFY) alongside bedside interview questions to clarify dosing. Gemini synthesizes the findings into a four-tier triage summary (Critical, Warning, Verified, Unmarked) with black-box warnings and side-by-side packaging comparisons before archiving the signed report.

How we built it

Mobile Client: Built with React Native, Expo (SDK 57), Expo Router, TypeScript, Gluestack UI, and NativeWind.
State Management: Handled via Redux Toolkit and RTK Query. Auth is handled via Clerk.
Backend: Python and FastAPI using structured Pydantic schemas.
AI & Data Pipeline: Gemini 3.6 Flash powers primary multi-shot vision extraction and clinical interaction audits and structured summaries. Federal data is queried live via openFDA and NIH DSLD REST endpoints.

Challenges we ran into

The Loose Pill Imprint Dead-End: We initially planned to identify loose pills on trays, but discovered that government APIs (openFDA, DailyMed) lack reverse imprint-search endpoints. Realizing that maintaining tens of thousands of generic imprint mappings was fragile and clinically detached from hospital intake policies, we choose to cut loose-tray scanning to focus on container packaging OCR and clinical safety auditing.
Dietary Supplement Blind Spots: Traditional drug databases omit botanicals and vitamins, requiring us to integrate the NIH DSLD API alongside openFDA to catch dangerous herbal interactions (like St. John's Wort or Ginkgo).
Vision Token Tile Caps: Multi-shot photos of curved vials quickly hit Groq's 7,000 TPM free-tier rate limits due to flat per-image tile token pricing. We re-architected the primary vision pipeline to Gemini 3.8 Flash, bypassing rate limits.

Accomplishments that we're proud of

Establishing a reliable, end-to-end multi-modal pipeline that routes drugs and supplements to their respective federal databases without human data entry.
Engineering a clinically grounded UX
Designing a four-tier semantic triage system that translates complex pharmacology, black-box warnings, and contraindications into scannable cards in seconds.

What we learned

Real-world clinical workflows depend far more on verifying container labels, active dosages, and patient physiological clearance than identifying anonymous loose pills.
Structuring strict Pydantic schemas upfront and testing against FastAPI's interactive OpenAPI docs prevents countless runtime integration headaches when wiring RTK Query.

What's next for MedIntake

Barcode & 2D DataMatrix Scanning: Adding instant UPC camera barcode parsing for packaging to achieve faster NDC lookups alongside OCR.
EHR Integration: Building FHIR/HL7 interfaces to pull patient baselines directly from hospital systems (like Epic and Cerner) and push back signed reconciliation reports.