Care AGent

Patient tab
Patient dashboard
Chat interface and midwife verification

Inspiration

In low-resource clinics, midwives record pregnancy care in a handwritten paper registry. Once written, the information stays on the page: it doesn't follow the woman from visit to visit and never reaches the health system.

The DayOne challenge (CodeML hackathon) asks for one thing above all: change nothing in the midwife's routine. So the paper registry stays, and the digital record grows on top of it. We wanted an assistant that says what it is unsure about instead of silently writing a wrong value into a medical record.

What it does

Care Agent is an offline-first app (French and English) that works on a phone. A midwife photographs a registry page, and the app reads the handwriting and fills a structured record. She only has to check the fields the app doubts: confirm, correct, retake the photo, or leave it illegible. The app then asks which patient the page belongs to. It never creates or merges a patient on its own, so each woman ends up with a single record that grows with every visit. The original photo stays linked to the record, encrypted.

How we built it

A PWA (React, Vite) on the phone. It checks photo quality on the device with OpenCV.js and keeps captures in an encrypted offline queue.
A local server (Hono, SQLite) on a laptop on the same Wi-Fi. No cloud, and no patient data leaves the network.
Two readers. Gemma 4 runs locally through Ollama. A small PaddleOCR model reads cell by cell, much faster.
A schema of the real form, with validators for each field. Every field gets an explicit status (known, unknown, not provided, illegible, not applicable, needs review).
Privacy by design. The app has no field for names, phone numbers or addresses. Those zones are blacked out before any image is processed.

Challenges we ran into

No GPU. Reading one zone with Gemma took about 20s (~5 min for a full page) on a CPU laptop. Switching to a cell-by-cell reader brought a dense page down to about 5 s.
Gemma lost track of cells when asked to read a whole grid, so we now send only the cells that contain ink.
Honest confidence. Gemma gives no usable per-token probabilities. We built our own confidence score and a calibration step.
Real photos behave differently from synthetic ones. Our real form exposed checkbox errors and a misread digit in a fiche number. We kept it strictly held out and never tuned on it.

Accomplishments that we're proud of

A full flow that runs on a phone and a laptop, with no internet: capture, offline queue, local reading, review, patient linking, longitudinal record.
Good accuracy on a CPU-only machine: 92.2 % of handwritten cells right with Gemma, and 99.1 % with the cell reader on our tuning split (98.0 % on three unseen patients), with blank cells always left blank.
A bot that shows its doubts in plain words instead of percentages.
Privacy built in from the start, with no identifier stored anywhere.
A strict patient-level evaluation split, with real photos never used for tuning.

What we learned

A wrong value that is flagged costs the midwife one tap. A wrong value that is silent corrupts a record. So we track wrong-but-trusted values above all.
Cutting the problem small (schema, zones, ink detection, a small reader) beats asking one big model to read a whole page.
Synthetic data flatters you. Real photos show what actually breaks.

What's next for Care AGent

Finish the full evaluation and calibrate the cell reader on real handwriting.
Support the rest of the real booklet (double-page spreads) and Arabic and English pages.
Test with midwives on real phones over HTTPS.
Stay within scope: Care Agent digitizes the record. It never predicts risk or makes clinical decisions.