Careloop

Carey Logo
Careloop Logo
Voice Call
Family Message
Question
Doctor Message
Medicine Help

Inspiration

Careloop started with our own families. Some of us have watched grandparents struggle to follow their medical care: a long list of medicines, instructions that are hard to understand, and appointments where the doctor only sees a snapshot. For me, the problem is distance. I don't live anywhere near my relatives. When the only people with any medical knowledge are family, keeping track from far away is really hard.

We're not alone. 43% of US adults over 65 take five or more prescription drugs. Of the 63 million family caregivers in the US, more than half do medical tasks, but only about 1 in 5 has been trained for them. We wanted a tool that makes checking in a daily habit, helps people understand their own medicines, and catches problems early.

The tools at MHacks shaped the idea too. FinchNode's synthetic health records let us build around a realistic patient. Presage can estimate heart rate from a camera, so a check-in can include a vital sign without any extra device.

What it does

Careloop is an AI check-in companion that lives in Relay Messenger. Our demo patient is Harriet, 78, with kidney disease, atrial fibrillation, heart failure, diabetes and 14 medicines. Her record is synthetic and comes from FinchNode's API, and every question Careloop asks is based on it.

Daily check-in by text or voice. Each morning Careloop asks how she's feeling in her own words. Then it asks up to three questions picked from her history. She takes a blood thinner, so it asks about bruising or bleeding. She has heart failure, so it asks about ankle swelling and trouble breathing lying down. She can type, tap a button, or call and talk it through with an ElevenLabs voice.
Camera heart rate. At the end of a call, Careloop offers a quick camera check. It calls her back, records about 30 seconds of video, and Presage estimates her heart rate. The number is always given as an estimate, not a medical test.
Medication helper. Morning reminders list the medicines her record says she takes in the morning, and Careloop reads back what her prescription says. She can photograph a pill bottle and Careloop checks it against her record. If the strength doesn't match, it tells her to check with her pharmacist. When a refill is due, it reminds her and gives her a ready-to-read refill request.
Reactions in proportion. A sore knee gets noted for her doctor. Trouble breathing gets "call your doctor today" and an alert to her family. Chest pain gets 911 right away. Fixed rules decide how serious something is. The AI only helps understand what she said.
Family updates. Each family member gets their own Relay chat with daily status and alerts, limited to what she chooses to share.
Doctor report. The week's check-ins become a two-page summary a doctor can read in a minute. It covers symptoms by severity with dates and her own words, vitals, the questions she wants to ask, and medication concerns with sources. Instead of trying to remember everything at the appointment, it's documented every day.

Personalization is the whole point. An agent is only as useful as its context, so Careloop builds every conversation from her record and what she has told it before.

How we built it

Server: Node.js and TypeScript with SQLite. It's one server that connects every service and passes data between them.
FinchNode: Reads Harriet's record (read-only) and runs fixed rules on it. Examples: metformin with falling kidney function, a blood thinner combined with aspirin and an antidepressant, and potassium creeping up on an ACE inhibitor.
Relay Messenger: Chat, buttons, photos, and voice and video calls over WebSocket and WebRTC.
ElevenLabs: Realtime speech-to-text (Scribe v2) and the spoken voice (Flash v2.5) on calls.
Gemini: Understands her free-text and spoken answers and words the replies. A fixed safety screen runs before it and an output check runs after it.
Presage SmartSpectra: Heart rate estimates from video.
Python: The camera check call-back uses Relay's Python SDK to record clean video for Presage.

We built it with agentic coding tools (Claude Code, Codex and Cursor) working in parallel. We used branches and pull requests, and split the files so the agents didn't overwrite each other's work.

Challenges we ran into

Our biggest challenge was making all these tools talk to each other. They're written in different languages and expect different formats. FinchNode values arrive as strings with units mixed in, and weights can be in pounds or kilograms. ElevenLabs sends audio in tiny 4 to 14 ms pieces, and Relay padded each piece with silence, so the voice sounded scratchy until we regrouped the audio into whole 20 ms frames.

The hardest part was getting video from a Relay call to Presage. Presage needs about 12 seconds of steady, continuous video, but the live call video dropped frames every few seconds. The phone also switched video resolution in the middle of a reading, which crashed the whole server. Our fix was a second, short call. After the check-in, Careloop calls her back through Relay's Python SDK, which repairs lost packets. On the same phone it delivered 1,050 of 1,050 frames. It records the video at a steady 30 frames per second, gives the file to Presage, and then deletes the video.

We also had to manage a lot of data. That meant keeping API keys in one place on the server, deciding what context each model call gets, and storing transcripts and readings but never raw audio or video. Finally, keeping an AI careful in a medical setting meant making sure it never gives dosing advice or a diagnosis.

Accomplishments that we're proud of

Presage on a real call. Presage read a heart rate from video recorded on a Relay call: a stable, confident 85 bpm on our first live camera check.
A full working demo. It works end to end in the chat and on voice calls with ElevenLabs.
Safety testing. We ran 145 realistic messages through Careloop. It caught all 37 emergency and crisis messages and sorted 144 of the 145 correctly.
Automated tests. Over 1,400 tests cover the medication rules, the severity ladder, the calls and the doctor report.

What we learned

We learned a lot about connecting new tools and APIs. Everything passed our offline tests, but live calls on real phones showed problems no test predicted, so test on real devices early.

Context matters, both in the product and in how we built it. Careloop is only helpful because it knows her history, and our coding agents only worked well once we wrote clear docs and rules for them.

We also learned to stay specific about who we're building for. In early tests, "my knee hurts" got the same 911 warning as trouble breathing. That taught us to build a severity ladder so Careloop reacts in proportion. Seniors want a short, warm check-in, not an interrogation, so it asks at most three questions and "Not today" is always an option.

What's next for Careloop

A medical model on our own hardware. Replace Gemini with an open model from Hugging Face trained on medical text, hosted locally so health data never leaves our own hardware.
Better questions and experience. Polish the interface and make the questions more useful. Daily weight comes first: it's one of the most important self-care checks for heart failure.
Clinician review and more languages. Have a clinician review our safety rules, and support languages beyond English.