RxBridge

Read aloud in the patient's own language
Pipeline: docTR, our DistilBERT tagger, validated contract, attribution, PWA
Drug, dose and timing as a pictogram day-grid
Shuffled-label control: 0.937 vs 0.000 entity F1
Privacy and safety by design
The problem: a real handwritten prescription (Cuba, MINSAP form)
Held-out handwriting benchmark, 10 languages: zero-shot vs fine-tuned

Inspiration

A prescription is the single most important instruction a patient receives for their own body, and a large share of patients cannot act on it. The handwriting is illegible, the form is in a language they do not read, or they have low literacy or low vision.

We started from a real example: Cuba's pre-printed RECETA MEDICA form (MINSAP model 53-05-04). The printed scaffolding is machine-readable, but the drug name, dose and frequency are handwritten in blue ballpoint. Off-the-shelf OCR reads Clotrimazol (500 mg) as Mletzina 20. The patients most affected are the ones with the least access to an interpreter, pharmacist counselling, or a family member who can translate.

What it does

RxBridge turns a photo of a prescription into a spoken, structured medication schedule.

Photograph the prescription from a phone. It's a single-screen, installable, mobile-first web app (108 kB first load).
Drug, dose, frequency, duration are extracted and shown as a pictogram day-grid (Morning / Midday / Evening / Night, a pill glyph and a large numeral per dose), so a patient who cannot read a word still knows how many pills to take and when.
Read aloud in the patient's own language. The summary is spoken with the browser's Web Speech API, so no cloud speech call is needed.
Every value is attributed. For each drug, dose, frequency and duration the API names the exact source tokens that produced it, their mean confidence, and which field came out weakest, so a pharmacist sees why a value was read, not just a number.
It never invents medical advice. It only reads back the prescription. If a line can't be read it is left out, never guessed, and an unreadable photo returns a friendly "unreadable" result, never a crash.

Verified end to end on a real printed prescription: 20/20 OCR lines at 0.87-0.99 confidence, 3/3 drugs extracted with dose, frequency, day-grid times and duration.

How we built it

phone photo
 -> [1] docTR            detect + recognise text (in memory: no disk write, no cloud call)
 -> [2] DistilBERT       token-classification slot tagger, trained by us
                         DRUG / DOSE / UNIT / ROUTE / FREQ / DURATION
 -> [3] Pydantic         BIO spans -> validated PrescriptionSchedule (69 contract tests)
 -> [4] normalise        "twice daily" -> [morning, night]; q8h, BID/TID/QID/PRN; EN/ES/FR
 -> [5] Next.js PWA      pictogram day-grid + Web Speech read-aloud

docTR for detection and recognition, PyTorch / transformers for the DistilBERT tagger, FastAPI for the extraction service, Next.js 15 / React 19 / TypeScript for the PWA, Pydantic v2 for the validated contract with exported JSON Schema, pytest (96 tests) and Playwright for end-to-end tests, all trained and served on 4x RTX 5060 Ti.
Two honest fallbacks keep it usable. If the learned tagger emits an invalid span, a rule-based lexicon retries and the response reports which one answered. Unreadable input is a result, not a 500.

Measuring that the model reads rather than memorises

A metric can be high for the wrong reason, so we trained the same DistilBERT tagger twice on an identical split of 12,000 prescription sentences: once with the real labels, once with every label replaced by a random draw from the label distribution.

model	entity-level F1 (seqeval, n = 960 held out)
trained on real labels	0.9367
trained on shuffled labels (control)	0.0000

The control collapses to exactly zero, so the 0.937 measures the token-to-label mapping, not an artifact of the data generator.

Handwriting in ten languages: a fine-tuned edge VLM and a benchmark that can say no

Handwriting is the hard part, so we fine-tuned LiquidAI/LFM2.5-VL-3B, a vision-language model small enough for edge devices, with a rank-16 LoRA on line-level handwriting in ten languages and six scripts (German, French, Vietnamese, Persian, Arabic, Russian, English, Thai, Hindi, Urdu). Then we built the thing that matters more than the model: a held-out benchmark (benchmark.py) that never touches training data, is never used for model selection, and grades every language against a fixed bar of CER 0.25. It's a verifier, so it's allowed to say FAIL.

It did, three times, and each time it pointed at a real bug in our pipeline:

Corpus imbalance. Two languages were 92% of the gradient, so we capped by characters per language and shuffled globally.
A leak in our own harness. The generation budget was sized from the reference length of the answer being graded. We removed it, re-measured, and published the corrected numbers.
A missing stop token. The training labels never contained <|im_end|>, so the model learned to transcribe a line correctly and then keep going (Thai, Hindi and Urdu had ~40% exact lines but huge error). One line fixed it: the validation stop rate went from 54% to 100%.

Final result on the held-out test set, with zero-shot and fine-tuned on the identical harness and budget:

language	zero-shot CER	fine-tuned CER
German	0.124	0.067 (pass)
French	0.115	0.109 (pass)
Thai	0.988	0.163 (pass)
Vietnamese	0.377	0.262
Hindi	0.551	0.535
English	0.585	0.582
Arabic	0.796	0.792
Persian	0.798	0.801
Russian	0.934	0.923
Urdu	1.028	1.045
languages passing	2 / 10	3 / 10
mean CER	0.630	0.528

The fine-tuned model beats zero-shot in 8 of 10 languages, Thai error falls 6x, and a third language now clears the bar. It does not yet pass all ten, and we say so: the paragraph-form languages (Russian, Persian, Arabic, English) have references of 1,000-3,700 characters that a fixed 128-token budget can't cover, and a pre-declared supplementary run at 1,024 tokens is reported separately in eval/SUPPLEMENTARY_1024.md. That's why the handwriting model stays out of the production pipeline until the benchmark passes for a language: the verifier, not our optimism, decides what ships. Every number is reproducible from the repo (python3 benchmark.py --adapter ...).

Challenges we ran into

Handwriting. docTR reads printed scaffolding at ~0.99 confidence and fails on cursive. Zero-shot LiquidAI/LFM2.5-VL-3B read the blue-ink form exactly where docTR failed, but also hallucinated on a red-ink form, so we fine-tuned it and refused to ship it into the pipeline until a benchmark says it's reliable.
Our own benchmark caught our own bugs. It caught a corpus imbalance (two languages were 92% of the gradient), a harness budget that leaked the reference length into generation, and a missing end-of-sequence token in the training labels that taught the model never to stop. Each one is written up, with numbers, in submission/EVIDENCE.md.
Privacy. Real prescription text is never persisted: images are decoded in memory, and committed benchmark files are scrubbed to {language, CER} only.

Accomplishments that we're proud of

A complete, working path from phone photo to spoken schedule, with attribution on every field.
A tagger result backed by a shuffled-label control, not just a number.
A multilingual benchmark that's strict enough to report FAIL, which we kept honest even when it made our headline worse.

What we learned

Measure the signal, not the artifact. Every time a number looked good or bad for a reason we didn't understand, the cause was in the pipeline (data mix, decoding budget, a missing stop token), not in the model. Building the verifier first is what let us find them.

What's next for RxBridge

Promote the fine-tuned handwriting model into the pipeline for the languages where the benchmark passes, and keep it out where it doesn't.
Pharmacist-review mode built on the per-field attribution.
Fully on-device recognition: LFM2.5-VL-3B is small enough for edge hardware.

Target users and social impact

Primary: patients who cannot read their own prescription: low-literacy adults, elderly patients handed a prescription in a language they don't read, language-barrier patients who'd otherwise need a family member to translate dose and timing, and low-vision patients who can see a pictogram and hear speech but not small print. Secondary: community pharmacists and clinic staff confirming that a patient understood the label before dispensing.

Medication errors from unreadable instructions are a documented cause of preventable harm. RxBridge doesn't replace the doctor or pharmacist. It makes sure the patient can actually hear what they were told.