log-a-life

Mobile app to log a life in a database without internet.

Comment

💡 Inspiration

In many Moroccan health centers, a pregnancy is still followed on a pink paper form, the Fiche de surveillance de la grossesse et du post-partum. Midwives fill in blood pressure, weight, gestational age, lab results and newborn measurements by hand across 8 pages per patient. Later, someone has to retype everything into a digital system. That step is slow, error-prone, and often skipped, so the data that could detect pre-eclampsia or anemia trends never reaches the people who could act on it.

The DayOne challenge (CodeML 2026) asked a simple question: can a phone photo of that paper become structured, trustworthy data, without ever inventing a value? In maternal health, a misread "164/98" that becomes "104/98" hides a hypertensive emergency. Our goal was a tool that reads fast and, just as importantly, knows when it isn't sure.

🩺 What it does

Log-A-Life is a mobile app for midwives:

  1. Photograph any page of the registry. The model detects which of the 8 page types it is.
  2. Extract every field (text values, table cells and ticked checkboxes) into structured JSON. Each field gets a status: CONNU, NON_FOURNI, À_RÉVISER or ILLISIBLE.
  3. Review in a chat. Every uncertain reading is shown with its confidence and the model's best alternative reading. The midwife confirms it, picks the alternative, marks it empty, or types a correction.
  4. Save offline. The validated page is stored on the phone, linked to a patient, and waits for synchronization.

🛠️ How we built it

Stack: React Native / Expo (camera and chat UI) → FastAPI server → Donut (an OCR-free document transformer: Swin encoder + mBART decoder), fine-tuned on Google Colab.

1. Ground truth from the PDF itself

The specimen registry comes as a PDF and as PNG scans. We discovered that the PDF keeps a text layer. Printed labels are set in Helvetica, handwritten values in handwriting fonts (Caveat, Gaegu, Reenie Beanie…), and ticked boxes are vector ink strokes. From geometry alone we rebuilt the exact content of every page:

  • a handwritten run is assigned to the printed label immediately to its left;
  • a table cell is identified by its row label and column header (e.g. ta_c5 = TA — Visite 2 (T2));
  • a checkbox belongs to a group label (e.g. État des lochies : Fade, claires) when it sits indented below it; otherwise it is a yes/no field.

This produced a fixed schema of 272 fields across 8 page types, with exact labels and no manual annotation.

2. Unlimited synthetic training pages

We only had 10 patients (80 pages): 7 for training and 3 held out for testing. Fine-tuning a vision transformer on 56 images invites memorization. So for each training page we:

  1. erase the handwriting and ink marks (PDF redaction),
  2. redraw the empty checkboxes,
  3. write new, random but plausible values at the same positions, in the same handwriting fonts and ink colors, with per-character jitter,
  4. tick new random boxes.

Because we generate the values, the labels are exact by construction. Each training sample is drawn as

$$ x \sim \begin{cases} \text{synthetic page} & \text{with probability } p = 0.7 \ \text{real page} & \text{with probability } 1 - p \end{cases} $$

then passed through "smartphone photo" augmentations: perspective warp, a background around the sheet, uneven lighting, sensor noise, blur and JPEG compression.

3. Target format

Each page becomes one token sequence, with one special token per field:

<s_registre><s_page_7><s_ta>107/71<s_pouls>75<s_poids>64 kg … </s_page_7>

The model predicts the page type itself; the app can also force it.

4. Knowing when it's unsure

For a field whose value is decoded as tokens $t_1, \dots, t_n$, we compute

$$ p_{\min} = \min_i P(t_i \mid t_{<i}, \text{image}), \qquad m = \min_i \big(P_{(1)}^{(i)} - P_{(2)}^{(i)}\big) $$

where $P_{(1)}^{(i)}$ and $P_{(2)}^{(i)}$ are the two most likely tokens at step $i$. A field is flagged À_RÉVISER when

$$ p_{\min} < \tau \quad \text{or} \quad m < \mu \quad \text{or} \quad \bar{p} < \tau + 0.05, $$

with defaults $\tau = 0.85$ and $\mu = 0.30$. To build the alternative reading, we swap the weakest token for its runner-up and decode again (e.g. "107/71" → "101/71"). The midwife sees both.

5. Training on a time budget

Colab sessions are short, so training runs against a wall-clock budget $T$ (20–35 min) rather than a fixed number of epochs. The learning rate follows a warmup, then a cosine decay driven by elapsed time $t$:

$$ \eta(s, t) = \eta_0 \cdot \min!\left(1, \tfrac{s}{s_w}\right) \cdot \left(0.05 + 0.95 \cdot \tfrac{1 + \cos(\pi \, t / T)}{2}\right) $$

where $s$ is the optimizer step and $s_w$ the number of warmup steps. Training therefore always ends cleanly on time, whatever GPU we get. We train in fp16 with gradient checkpointing at 1280×960, and save the weights in fp16 (≈ 390 MB) so they download quickly.

🧗 Challenges we faced

  • Our first model scored 0%. We had aligned the images with a provided CSV, assuming it described the same patients. It didn't: the page says age 31, the CSV says 24, and most requested fields weren't even on the pages. The model was learning impossible targets. Rebuilding the ground truth from the PDF was the turning point of the project.
  • Missing glyphs. Some handwriting fonts in the PDF have no Unicode mapping for accented characters, so Césarienne came out as C\x00sarienne. We recover them by matching each damaged word against the vocabulary of clean values from the same document (é, è, °, —).
  • Subset fonts. The fonts embedded in the PDF only contain the glyphs that were used, so they can't write new values. We download the full open-source versions, and fall back to another font character by character when a glyph is missing.
  • Tables are tricky. Long dates in adjacent cells merged into a single "word", and two tables side by side shared one header row. We split text at column boundaries and group header rows by table.
  • Hardware and versions. Training didn't fit in a laptop GPU's memory. The previous model had been saved with a newer transformers release than the server could load. We pinned versions and made the server reject obsolete models with a clear message, instead of returning garbage.
  • Honesty over demos. The early prototype returned made-up "success" data whenever the server failed. In a medical tool that's dangerous, so we removed it: errors are shown as errors.

📚 What we learned

  • Look at the data before tuning the model. A 0% score was a labeling bug, not a hyperparameter problem.
  • Synthetic data works best when it matches the real distribution exactly. Same templates, same fonts, same layout: only the values change.
  • Uncertainty is a feature. A model that says "I read 107/71, but it might be 101/71" is more useful to a midwife than one that is silently 95% accurate.
  • Practical skills: PDF internals (text spans, redaction, vector drawings), training under strict time budgets, and building a chat review flow with React Native.

🚀 What's next

  • Fine-tune on real photographed registries (we already have a few real Moroccan forms).
  • Run inference on the phone so it works with no network.
  • Sync validated records to DHIS2.
  • Add clinical alerts, e.g. TA ≥ 140/90 combined with proteinuria → flag possible pre-eclampsia.

Built With

Share this project:

Updates