decodRx

Inspiration

Healthcare information is only useful when people can understand and act on it. A medication label might seem straightforward, but small print, unfamiliar terminology, language barriers, and smudged instructions can make it difficult to read confidently.

We designed decodRx with people who have low vision, limited health literacy, or limited English in mind, along with the caregivers who support them. We also recognized that help often comes from family members and volunteers. We wanted to make that assistance easier to request and coordinate while keeping the patient in control.

Our goal was to connect medication information with human support and relevant community resources.

What it does

decodRx turns a medication-label photo into structured, editable fields, including the medication name, strength, dosage form, quantity, and original directions. Users review those fields alongside evidence from the label. Unreadable details remain unresolved, and the application requires confirmation before opening the explanation view.

When users need reading assistance, they can request help from a volunteer. The volunteer proposes a transcription or requests a clearer image, and the patient decides whether to accept the proposal. Questions about ambiguous instructions remain questions for a pharmacist.

Users can also share selected information with trusted supporters and coordinate practical tasks, such as arranging transportation to a pharmacy.

Our Community Resource Board helps users discover nearby workshops, assistance events, and volunteer opportunities. Listings retain their source links, and the application distinguishes information it can validate from details that remain uncertain.

Large-text options, keyboard navigation, and layouts for smaller screens make the experience more accessible.

How we built it

We built the frontend with React and TypeScript and the backend with Node.js and Express. Shared validation schemas helped us develop frontend and backend components in parallel while maintaining consistent data formats.

Google Gemini extracts structured information from medication-label images. Reference lookups connect users to resources from NLM RxNorm, DailyMed, and MedlinePlus.

For community discovery, SerpApi retrieves event listings, and Meta’s Muse Spark organizes retrieved information into structured details. OpenCage supports venue geocoding, while application logic checks geographic distance and dates.

We used Vitest, Supertest, and Playwright to test individual components, API behavior, and browser journeys. Fictional medication examples allowed us to develop and demonstrate the workflow without using real patient information.

Challenges we ran into

One of our biggest challenges was designing for uncertainty. An unreadable number needed to remain unresolved throughout extraction, review, and explanation, rather than becoming a plausible-looking answer.

We also learned that a successful API response does not guarantee a useful result. Location searches could return unrelated places, requiring additional geographic validation. Community listings introduced similar challenges around incomplete dates, duplicate events, and missing venue information.

Human assistance created another design challenge: volunteers needed enough information to help without automatically receiving access to the patient’s full profile. We also needed to preserve patient review and prevent volunteer proposals from silently replacing existing information.

Accomplishments that we're proud of

We built a connected workflow that combines medication-label review, human transcription assistance, selective sharing, and community discovery.

We are especially proud of making uncertainty visible, giving patients control over volunteer proposals, and building revocable supporter access.

We also isolated the community pipeline from medication processing, so a failure in event discovery would not interrupt label review. Automated checks helped us verify these behaviors as we integrated the application.

What we learned

We learned that healthcare accessibility involves more than larger text. People also need clear explanations, understandable next steps, and a way to involve someone they trust.

We learned to validate external data rather than treating model output or search results as inherently reliable. Sources, missing information, and review states all need to remain visible.

Most importantly, we learned that AI can make human assistance easier to access while preserving the patient’s role in deciding what to share and accept.

What's next for decodRx

Our next priority is usability testing with people who have different accessibility needs and with the caregivers who support them.

We want to evaluate extraction on a broader range of label images, improve volunteer onboarding, and expand reviewed language support and audio features. We also plan to investigate plain-language rewriting of directions through a careful review process.

For the Community Resource Board, we want stronger organizer partnerships and better handling of updated or canceled listings.

Before real-patient use, we would strengthen authentication, privacy protections, and accessibility testing, then evaluate the application with appropriate healthcare expertise.