LabelLens

LabelLens reads a herbal product’s label and tells pharmacists whether the EU monograph evidence applies to that exact preparation, plus what the label leaves out.

Comment

Inspiration

When a study or an EU herbal monograph says a herb helped, that evidence covers one specific preparation: a particular plant part, extract type, drug-extract ratio (DER) and extraction solvent. A shop product that just says "St John's wort" or "Ginkgo" may be something quite different, so the evidence may not apply to it at all. Pharmacists are asked about these products every day, but comparing a label line by line with a monograph takes time most of them don't have.

We wanted a tool that answers one narrow, honest question: does the evidence for this herb apply to this product? It checks only what is printed on the label, never what is inside the product, and it never treats missing information as good news.

What it does

A pharmacist photographs a herbal product's label, or pastes its text, and LabelLens:

  1. Reads the label with a vision model, copying ten fields (plant, plant part, preparation type, extract ratio, solvent, marker, dose, other ingredients, batch, expiry). Each field is stated, not stated or unreadable, with a confidence and the exact label text it came from.
  2. Lets the pharmacist check and correct every field before anything is compared.
  3. Compares the label with the preparations studied in EU monographs (EMA/HMPC), using deterministic rules rather than the AI, and returns one of:
    • Matches a studied preparation
    • Partly matches
    • Cannot say whether the evidence applies
    • Not yet covered (the herb isn't in the reference table)

Internally the rules can also find that a label differs from every studied preparation. Pharmacists see this as "Cannot say" until our experts decide how to present it; validation scores the exact internal verdict.

  1. Explains the answer field by field with source links, and lists what the label leaves out, with a question to ask the manufacturer or the patient for each gap. The manufacturer questions come as a ready-to-send message the pharmacist can copy with one click.

Around that core:

  • History and rechecks: every scan and dated verdict is kept, and a saved scan can be checked again against the current reference table.
  • Compare two products side by side. It shows which fields differ and never ranks the products.
  • Patient link and printable summary in plain language. The link expires and can be revoked.
  • Ask an expert: a pharmacist can send an uncertain result to an expert reviewer. Reviewers see results without the pharmacist's name, and their opinion is shown next to the verdict as advice. It never silently changes it.
  • Three roles with clear jobs:
    • Pharmacists scan.
    • Experts review verdicts, maintain the reference table with a full change history, and own the validation labels: the photos, the correct fields and the expected verdicts.
    • Admins manage accounts and run validations. They can read the reference table and the validation labels but cannot change them, so the people running LabelLens never write its answer key.
  • Built-in validation: a validation run measures field accuracy (including values the reader invented), verdict accuracy and expert agreement, over labels an expert has marked Final. It compares LabelLens against two baselines (keyword matching, and a plain AI prompt without our rules), and every number comes with a 95% confidence range.

Every page says "Research prediction, not medical advice". LabelLens gives no dosing or treatment advice.

Try it: open the live demo and register as a pharmacist to scan a label. Expert and admin accounts are invitation-only by design, so the expert and admin screens are shown in our video and image gallery. Judges who would like to try those roles can ask us for an invitation.

How we built it

  • Frontend: React and TypeScript with TanStack Query, Tailwind with our own design tokens, and Recharts for the validation charts. All interface text lives in one English strings file. Verdicts never rely on colour alone, and we never use red/green: each status pairs colour with an icon and words.
  • Backend: FastAPI, SQLAlchemy 2 and Alembic on PostgreSQL. Role checks are enforced in the backend, and passwords are hashed with Argon2. Sessions use a secure cookie, with optional "Remember me". Logins are rate limited, and important actions go into an audit log.
  • Label reading: a Groq-hosted vision model (qwen/qwen3.8-27b) with a strict JSON schema.
    • Every field is checked on its own. A field that breaks a rule is cleared for the pharmacist to fill in, and the rest of the reading is kept.
    • Every reading is stored with its prompt version, provider and model, and cached, so it can be replayed later at no cost. Without an API key, a cached mode replays saved readings.
  • Rules engine: pure, versioned Python with no web or database code, fully unit-tested. It returns codes only, all wording lives in the frontend, and each verdict stores the rules version that produced it.
  • Reference data: seven source-linked draft preparations for St John's wort, ginkgo and German chamomile, transcribed from final EMA monographs and loaded on first start. They are clearly marked "Sample, not checked" until a qualified reviewer verifies them.
  • Validation labels: 12 real product labels for ginkgo, St John's wort and chamomile, 11 of them from the NIH Dietary Supplement Label Database (DSLD). An expert entered the correct field values and expected verdict for each and marked them Final.
  • Privacy by design: label photos are turned upright and stripped of location and camera data. Patient-link tokens sit after # in the URL, so they never reach server logs, and only their hash is stored. Users can delete scans or their whole history.
  • Validation statistics: accuracy ranges use the Wilson score interval, which behaves well for the small samples a hackathon allows.
  • Deployment: Docker Compose for local runs; the live demo runs on Amazon Lightsail behind Caddy with a Let's Encrypt certificate.

Challenges we ran into

  • AI readers invent things. Small print, glare and multilingual packaging produced wrong or made-up values. On one real chamomile label the reader took "25g" for a preparation. We added an evidence check: a value marked "stated" is rejected unless it appears in the label text the model quotes as its evidence. Placeholder answers are rejected too, and the pharmacist reviews every field before comparison.
  • One bad field used to throw away the whole reading. At first a single field that failed the evidence check discarded every field, so labels came back empty. We now repair or clear only the failing field. We also learned that the model sometimes quotes a value correctly but labels it "not stated", so such a value is kept when it appears word for word in its quoted evidence.
  • Unreliable output and limits on a free API.
    • Strict-schema responses sometimes failed, so we retry once in JSON mode and validate the same schema locally.
    • When Groq is busy or rate-limited, we wait briefly and retry once.
    • If a reading still fails, the pharmacist sees the error and nothing empty is saved.
    • Every successful reading is cached.
  • Real labels rarely say enough. A real ginkgo product left out its extract ratio and solvent, so the honest answer was "Cannot say". Designing the interface so that "Cannot say" reads as useful information, with concrete questions to ask, not as a failure took several iterations.
  • Codes that look like batch numbers. On real labels the reader took item numbers and product codes (for example "VS-3301") for batch numbers and expiry dates. Our validation counts these as invented values, which is exactly the kind of error that metric exists to catch.
  • Being honest about what we can claim. Without an external expert we could not verify the reference rows, so the app labels them as sample data. We quote only accuracy numbers from a stored validation run over expert-checked labels, always with the sample size and range.
  • Getting the roles right. Our first design had admins typing reference rows for experts to check, and setting the expected answers used to grade the system. We reworked it so that each piece of work has one owner. Experts own everything that needs a monograph or clinical judgement, including the answer key, and admins only run the system. That way the people running LabelLens never grade their own work.
  • Switching the label-reading provider mid-project. We moved label reading to Groq and removed the previous provider's code and dependency while keeping saved readings replayable.

Accomplishments that we're proud of

  • The AI never chooses the verdict. It only reads; transparent, versioned and tested rules decide, and every reason links to its source.
  • Missing information is never good news. Gaps lower the verdict and become a checklist of questions, not silent passes.
  • A complete, working workflow for three roles: scanning, review, expert escalation, patient sharing, product comparison, reference maintenance with history, and validation. It is deployed over HTTPS.
  • A validation framework designed for honesty: baselines, invented-value tracking, confidence ranges, and stored runs with a frozen copy of the answers they were scored against.
  • First measured results. On 12 real labels marked Final by an expert, LabelLens gave the expected verdict for 11 of 12 (92%, 95% range 65–99%). A plain AI prompt without our rules managed 9 of 12 (75%, 47–91%), and keyword matching ("the herb is named, so the evidence applies") 1 of 12 (8%, 1–35%). The set is small and the ranges are wide, so we treat these as a first signal, not a claim.
  • Careful wording and accessibility: no dosing advice, no claims about what is in the product, and status shown with icon, text and colour together.

What we learned

  • Vision models are good at reading labels but must be made to show their evidence. Asking "where on the label does this say so?" catches a surprising number of invented values.
  • Checks must be precise, not blunt. Rejecting a whole reading for one doubtful field hid good values; checking each field on its own kept them.
  • Keeping AI and decision logic apart makes the system explainable and testable: we can unit-test the rules exhaustively, and measure the reader on its own.
  • In health-adjacent tools, "I can't tell" is often the most responsible answer, and the interface has to make that answer useful.
  • Role design is product design. Deciding who owns which decision removed most of the confusion in our early versions.
  • An accuracy figure means little without a sample size and an uncertainty range. With 12 labels, the range is wide, and we say so.

What's next for LabelLens

  • Expert verification of the reference table by a pharmacist or pharmacognosist, row by row against the monographs.
  • A larger real-label validation set, so the confidence ranges narrow and we can compare against our baselines with real certainty.
  • More herbs and monographs, and plant-name lists in more languages, including Albanian.
  • Better small-print reading, including telling item numbers and barcodes apart from batch numbers and expiry dates, measured against the validation set rather than judged by eye.
  • Production features: two-step sign-in for experts and admins, usage statistics, an audit-log screen, automatic data retention, and licence checks for pharmacist accounts.
Share this project:

Updates

Submission history