Juno

Juno is an iMessage assistant for after a hospital stay. It reads the discharge papers from one photo, tracks every follow-up, and brings in family when Mom needs help.

Comment

Inspiration

When an older parent leaves the hospital, they go home with a folder of discharge papers: new medicines, a list of warning signs, and three or four follow-ups buried in a table that says "patient to call to schedule." Those follow-ups are the part that slips. Nearly 1 in 5 Medicare patients is back in the hospital within 30 days, and about half of the non-surgical ones never saw a doctor in between.

Their kids want to help, but they usually live in another city and hear about it after the fact.

We didn't want to build another app. The people who need this most won't download one, make an account, or remember a password, but they all text. So Juno is just a contact in iMessage, and we spent the hackathon getting Photon's Spectrum SDK to make the conversation itself do everything an app would.

What it does

Mom texts Juno a photo of her paperwork. That's the whole interface.

  1. Reads it. OpenCV finds and flattens the page, and Gemini pulls out the diagnoses, medicines, warning signs, and every follow-up with its phone number and deadline.
  2. Explains it. Warning signs come first ("Call 911 if…"), then a plain-English summary and what changed in her medicines, including what to stop taking.
  3. Turns it into a live checklist in the chat. She taps a step to call the clinic, read a two-line script of what to say, or mark it booked with a date picker. Texting "booked heart doctor Tue 2pm" or sending a photo of the appointment card works too, and she gets a calendar invite back.
  4. Follows up. At most two gentle reminders per step. If Mom has read both and still hasn't booked, Juno quietly texts her daughter, who can book it from her own chat. Mom's card flips to Booked · with Tvisha.
  5. Makes lab results readable with a tappable body map, and answers questions like "when is my heart appointment?" or "what's Eliquis for?" straight from her own papers.

Juno never books appointments and never gives medical advice. It explains what the paper says and makes sure the next steps actually happen.

Juno's live iMessage cards: the checklist, a step's detail, booking, and the lab body map

How we built it

Bun + TypeScript on Photon Spectrum, Gemini for vision, OpenCV for page cleanup, SQLite for state, Hono for the card pages, resvg for preview images, and a cloudflared tunnel so the cards inside iMessage can reach our server. About 4,000 lines of TypeScript and Python.

Every iMessage feature has a job

  • Live app cards (app(url, { live: true })): the checklist and body map are web apps running inside the message bubble. Taps work, so Mom calls, books, and checks things off without leaving the chat.
  • edit(): cards update themselves in place. We persist each card's miniAppCardSession in SQLite, so edits keep landing even after the server restarts.
  • Read receipts: the escalation trigger. Family only gets pulled in after Mom has actually seen the reminders.
  • Native polls: Mom picks what her daughter can see, and Juno asks "which one did you book?" when a reply is ambiguous.
  • Tapbacks: ❤️ on a reminder gets "Did you book it? What day and time?", and ❓ gets the number and the script again.
  • Attachments: an .ics invite for every booked visit, with a reminder the day before.
  • Rich links: the full care plan and lab view arrive with preview images we draw on the server, never a gray URL.
  • Typing, mark-as-read, and effects: it feels like texting a person, and confetti fires when the whole list is done.
  • Separate 1:1 chats: the daughter gets her own conversation with Juno, with updates and the ability to book for Mom.

From photo to plan

A phone photo of a discharge summary, the page OpenCV detected, and the flattened scan sent to Gemini

Before any AI sees the photo, cv/flatten.py runs Canny edges → contours → the largest quadrilateral covering at least 20% of the frame. Its four corners define a homography \(H\) that warps the page flat, snapped to US Letter proportions:

$$(u, v, 1)^{\top} \sim H (x, y, 1)^{\top}, \qquad \frac{h}{w} = \frac{11}{8.5}$$

CLAHE and an unsharp mask then boost the contrast. If no page edge is found, it falls back to the original photo, so computer vision never blocks the flow.

Gemini reads every page of a burst in a single call and returns strict JSON validated with zod. The prompts follow three rules: extract only what's on the page, never invent a phone number or date, and use plain language.

The model reads, the code decides

Anything that matters is deterministic code, not model output.

Lab colors come from the reference range printed on the report. For a value \(x\) with range \([L, U]\):

$$\delta(x) = \frac{\max(x - U, 0)}{U} + \frac{\max(L - x, 0)}{L}$$

A result is in range when \(\delta = 0\), slightly outside when \(0 < \delta \le 0.1\), and outside beyond that. Each test maps to an organ through a fixed table in code, and out-of-range results become questions to bring to the doctor.

Escalation to family, for a step \(t\) that's still open:

$$\text{escalate}(t) \iff s(t) \ge 2 \wedge \big(r(t) \ge 2 \vee \Delta(t) \ge 24\text{ h}\big)$$

where \(s\) is reminders sent, \(r\) is reminders read (from read receipts), and \(\Delta\) is the time since the last reminder.

Emergencies skip everything. A message like "I have chest pain" is caught by a pattern check before any AI call and gets an immediate "call 911," plus a heads-up to family.

Built to survive a free tier and a live demo

  • Free-tier Gemini allows 20 requests per model per day. Juno falls through a chain of models and API keys on rate limits (reading documents gets the strongest model first, quick jobs start on Flash-Lite), caches photo reads by a hash of the image, and parses common replies like "booked it Thu 9am" locally. A full rehearsal costs zero AI calls.
  • A virtual clock lets us show three days of reminders and escalation in a few minutes on stage.
  • A live dashboard on the laptop shows every step as it happens: the photo, the detected page, what Gemini read, and the care board moving from To do → Booked → Done, with phone numbers swapped for names.

Challenges we ran into

  • Cards are tiny web views. About 271×300 pt, no scrolling, and window.open doesn't work. Every card had to fit on one screen, and "Full view" falls back to Juno texting the link.
  • Albums. Several photos sent together arrive as one group message, and downloads are slow. We register each photo the moment it arrives, download in the background, and batch each chat into turns (2.5 s for texts, 5 s for photos), so a two-page discharge summary stays one turn.
  • Read receipts mark the whole chat read. A receipt doesn't say which message was seen, so we treat it as "read up to here" and credit every reminder sent before it.
  • An SDK bug. Polls echoed back from the server had an empty title, which silently dropped votes. We tracked it down in the SDK source and shipped a one-line patch.
  • Card images have to be JPEG. PNGs get rejected upstream.
  • Gemini quotas, which pushed us to caching, local parsing, and model fallback.

What we learned

  • iMessage works as a real app platform once you treat its features as building blocks. Read receipts, tapbacks, and polls each replaced a screen we would otherwise have had to build.
  • "Read" is a far more useful signal than "sent." Knowing Mom actually saw a reminder is what makes it OK to bring in her daughter without nagging anyone.
  • In healthcare, let the model read and let code decide. Gemini is great at pulling structure out of a messy page; colors, thresholds, timing, and escalation belong in code you can test.

What's next

  • One shared family group chat instead of separate 1:1s (group chats need Photon's paid plan).
  • More documents: prescription labels, ER visit summaries, and insurance letters.
  • Daily medicine reminders from the morning/evening schedule Juno already builds.

Built With

Share this project:

Updates

Submission history