Inspiration
Midwives record pregnancy and postpartum follow-up on paper registries. The paper works for them, but the data stays on the page: it can't be linked across visits, shared, or counted. The challenge asked for a way to digitize these registries without changing how midwives work and without putting any patient's identity at risk. We wanted an assistant that says when it is unsure, because a wrong value stored silently in a medical record is worse than a question.
What it does
A WhatsApp-style chat agent. The midwife sends photos of registry pages, and the agent:
- reads each page with a vision model and gives every field a status (known, unknown, blank, illegible, not applicable, needs review) and a confidence;
- asks about every field it is unsure of, one by one or all at once, and lets her confirm, edit, retake a photo or enter a page by hand;
- works offline: photos are saved encrypted on the device first, and reading and synchronization resume when the connection returns;
- links a woman's registries by a code and only suggests matches; the midwife decides;
- never stores a name, phone number, ID or address.
Extras: an on-device image-quality check, a French/English interface, an anonymized dashboard, and a WhatsApp Cloud API connector (tested against a simulated service only).
How we built it
Python and FastAPI, with an encrypted SQLite database (SQLCipher) and AES-GCM for images. Gemini (gemini-3.5-flash-lite) reads the pages behind an extractor interface, with a prompt generated from the form's schema. The chat is a plain HTML/JavaScript page.
We built the evaluation before trusting any number: reference values come from the registry PDF, patients are split into dev and test, and the follow-up threshold is chosen on dev only. For a threshold \(t\):
$$\text{silent error rate}(t) = \frac{\text{wrong values stored without asking}}{\text{all values stored without asking}}$$
We take the lowest \(t\) that keeps this under 2% while asking about at most 25% of filled fields.
Results on held-out test patients (French, clean images): 97.3% of filled fields read correctly, 99.4% of blank fields left blank, no invented values, and 0.3% silent errors while asking about 4.3% of filled fields.
Challenges we ran into
- The model was overloaded for hours. Pages sat on "pending" with no explanation. The agent now says why, retries with back-off and keeps synchronising.
- A prompt change made things worse. Telling the model about Arabic and English dropped French accuracy on dev from 98.3% to 90.0%. We measured it, reverted it, and kept it as an option.
- Whole table columns went missing. 21 of 26 test errors were columns the model never returned. No field existed to be unsure about, so the agent now asks about every table.
- No Arabic or English data. We generated synthetic pages in both languages (97.2% and 97.1%). Reading the model's answers showed the Arabic weak point: Arabic-script digits, with ٢ and ٣ confused at high confidence. An optional stricter check on numbers cuts silent errors there from 2.5% to 0.6% on dev.
What we learned
Accuracy was the easy part. Most of the work was in what happens around the model: missing answers, over-confidence, outages, and proving each claim with a test. Measuring before and after every change caught mistakes we would otherwise have shipped.
What's next
Real handwriting in Arabic and English, degraded photos, a side-by-side review screen, and a real WhatsApp pilot.
