HearMe

Work phone screen
Office PC screen

Inspiration

All of us work as a senior care worker at FOBE (Friends of Bright Eyes), a Luton charity supporting 100+ disabled children. Before a child can join, a family worker meets the parents for up to 4 hours, takes handwritten notes, then spends about 2 more hours writing a care plan by hand: diagnosis, medication, allergies, how the child communicates, what calms them, what triggers them. That's 200+ hours a year of writing, and she spends the meeting writing instead of listening.

Many FOBE families speak Urdu, Bengali or Sylheti at home. The information is children's health data, so it should never go anywhere near the cloud.

What it does

Record the visit (with a consent checkbox first) on the family worker's phone, or photograph or type the notes.
The work phone sends files peer-to-peer straight to the office PC over Holepunch. No email, no WhatsApp, no server. If the office is offline, files are queued and sent later.
The office PC transcribes and translates locally. Our real demo clip is a mum speaking Sylheti, which Whisper turned into gibberish, so we use IndicConformer plus a local Qwen model and a small Sylheti glossary.
A local LLM drafts the care plan in FOBE's exact format: about me, mobility, communication, toileting, diet, specific words and phrases, challenging behaviours and control measures.
It never guesses. Every field must quote its source, or it's marked missing. Health facts that come only from machine-translated speech are marked needs review. In the demo, medication is flagged missing from the recording, then filled in when the handwritten notes are added. A vision model reads those notes, also locally.
The family worker / supervisor edits, then signs off with a YubiKey passkey (WebAuthn). Every version is kept in an append-only database, and annual reviews are flagged automatically.
Print-ready A4 view for FOBE's secure care-plan folder.
NFC lanyard: tapping the child's YubiKey on any Android phone shows vital "if I'm lost" info (name, egg allergy, mum's number) offline. Diagnosis and ethnicity stay off it unless parents opt in.
A read-only backup of every plan lives on a second device and updates live.

Children's data never leaves the building. We ran the whole pipeline on the office laptop with Wi-Fi switched off.

How we built it

QVAC (Tether): all AI runs on-device through the QVAC SDK on an Intel i5 laptop.
- Speech: IndicConformer 600M (Bengali mode) for Sylheti, with Whisper large-v3-turbo as a fallback.
- Translation and handwriting reading: Qwen3.5-4B vision.
- Care plan drafting: Qwen3-4B, using six small JSON-schema calls with a shared cached prompt prefix. Small models are much more reliable that way.
Pears / Holepunch: Hyperbee on Corestore is the local care-plan database. Hyperswarm handles pairing and phone→office file transfer. A read-only Hypercore replica is the backup. Optional encryption at rest uses a passphrase-derived key. A local HyperDHT bootstrap node is the fallback when venue Wi-Fi blocks peer discovery.
UI: an Express app on localhost with no CDNs, and WebAuthn passkeys on YubiKey 5C NFC for login and sign-off. The "work phone" is a separate local web app with a 16 kHz WAV recorder, a notes camera, an outbox and the backup view.
NFC: the YubiKeys' NDEF slot is programmed with ykman, so the child's key reads as plain text on any phone.
Teamwork: three people building in parallel, each owning one folder with a frozen shared contract and checkpoint files, so we never had a merge conflict.

Challenges we ran into

Sylheti. Every Whisper size and language setting produced gibberish. Switching to IndicConformer in Bengali mode gave a near-perfect Bengali-script transcript, but translation still turned "egg" into "banana" until we added a small glossary. That's exactly why anything health-related from translated speech is flagged for human review.
Windows on ARM. QVAC only supports x64 on Windows, so the AI runs on one laptop and the Snapdragon laptop plays the work phone.
The venue Wi-Fi blocks laptop-to-laptop traffic, so we added a local DHT bootstrap and a hotspot fallback.
Small local models invent things, so we made "quote your source or mark it missing" a hard rule in code, not just in the prompt.

Accomplishments that we're proud of

A real Sylheti recording → a structured care plan, fully offline, in about two minutes, against a typical 2 hours by hand.
The model refuses to guess medication. Missing really means missing.
A genuine P2P workflow that replaces how sensitive files actually get moved today (email and WhatsApp).
It was built around a real charity's real form, by someone who uses it.

What we learned

Local AI is ready for real social-care work if you design for its limits: small schema-bound calls, source quotes, and a human always signing off. Minority languages like Sylheti are where off-the-shelf speech AI fails first, and those are often the communities that most need the help.

What's next for FOBE Care Plan AI

Pilot with FOBE's family worker on real (consented) visits.
Better Sylheti support with community-recorded data and fine-tuning (QVAC supports LoRA fine-tuning on-device).
A native mobile app for the work phone, built with QVAC's Expo support and Bare.
Share the tool with other small disability charities, who face the same paperwork with even fewer staff.

All child data in the demo and repo is fictional.