Interlude

Doctor Dashboard
View Patient Details
Begin Real Time Transcription
Real Time Translation
Patient Dashboard
English Visit Summary
Patient Visit Summaries
Patient Health Data
Patient Notifications
Requesting Call With Agent
Call With AI Agent direct to Patient's Phone

Inspiration

Jiwon: As an exchange student in the United States, I’ve experienced firsthand how challenging it can be to communicate in a second language. Even in everyday situations, expressing exactly what I mean can be difficult. However, these language barriers can be especially consequential in healthcare, where misunderstandings can affect a person's health, safety, and confidence in managing their own care.

Caden: I experienced first hand the dangers of miscommunication when my mother was denied the adequate treatment leading to the development of a shingles infection. A bit of digging revealed that many individuals and families across United States deal with the same consequences--consequences that could have been entirely avoided if the patient was given adequate time to communicate.

We realized that the problem was systematically baked into healthcare which has been transformed by a heavy reliance on electronic health records leading to unnecessary administrative burdens. This shift has drastically reduced the amount of time allocated for face-to-face interactions between the patient and doctor. In fact, doctors in the United States now on average spend more time working on desktops (3.17 hours) than speaking with patients (3.08 hours). For patients whose English is not their first language, communication is becoming more rushed and incomplete; and that is dangerous.

We found that preexisting solutions only focus on specific parts of the healthcare journey, leaving patients to navigate the gaps between appointments, consultations, and follow-up care.

That inspired us to create Interlude: an app designed around the patient's entire healthcare journey. By supporting patients before, during, and after medical visits, we aim to make communication and understanding more accessible at every step.

What it does

Interlude is a companion for the whole care cycle: before, during, and after every medical visit. It uses multilingual AI voice agents to send daily check-in calls before each visit, record/transcribe during the visit, and ensure understanding from the patient after the visit. While data collection from check-in calls works in the patient's own language, our workflow translates this data so that the doctor is provided with a high-level report of the patient's conditions and concerns ahead of the visit. Combined with live recording and translation during the visit, doctors spend less time taking notes and more time caring. After each visit, patients can access a call with an agent in their native tongue which answers anything they didn't understand from the visit and ensures that the patient is familiar with their medication. Interlude's agents contact you through plain phone calls, so patients who don't open the app are still covered.

Before the visit

Interlude uses an AI voice agent to call patients every day at a set time and asks how they're feeling. It remembers earlier calls and follows up on the symptoms they mentioned. Patients can also tap "Call me now."
Patients add questions for their doctor by typing in the app or by saying them during a call.
If a patient describes an emergency symptom, Interlude tells them to call 911 and alerts the doctor.

For the doctor

Check-ins become a symptom timeline with severity, medication events, and the patient's own words, each translated.
The doctor gets a short pre-visit brief and the patient's question list.
The doctor can search the record in any language, or ask a question about it and get an answer that cites what the patient said.

During the visit

Interlude translates the conversation live, with the patient's consent.
It records and transcribes the visit, then writes a summary in both the doctor's and the patient's languages.
It shows which questions were answered and which symptoms never came up. The doctor reviews the summary before releasing it.

After the visit

The patient gets a plain-language summary, a daily medication schedule, and a checklist of next steps.
Patients can call or chat with Interlude about their visit. It answers only from that visit's record.
Unanswered questions carry over to the next appointment, and the daily check-ins continue.

How we built it

Interlude has two front ends that share one backend. Patients use a React Native (Expo) mobile app and also get phone calls. Doctors use a React (Vite) web dashboard. Both apps talk only to Convex. Convex calls Cartesia for voice, Elastic for search, and Claude to extract symptoms from conversations, translate, and write summaries. No API keys are stored in either app.

Convex is our backend. It holds the data, runs every server function and scheduled job, and stores the visit recordings, all in one TypeScript codebase.

Live queries: a symptom a patient mentions on a call appears on the doctor's dashboard within seconds, and a released visit summary appears on the patient's phone right away. We wrote no polling code for either.
Workflow component (@convex-dev/workflow): after each call, a durable workflow runs extraction, saves the results, and indexes them, retrying any step that fails.
Scheduling and storage: crons place the daily calls and generate pre-visit briefs, and file storage holds visit recordings.
Typed API: both apps use Convex's generated types, which let two developers build in parallel against an agreed set of functions.

Elastic is the search and reasoning layer over the patient's own words. Every symptom quote and visit passage is indexed in Elasticsearch on Elastic Cloud Serverless.

Hybrid search: we combine semantic search with Jina multilingual embeddings, which match by meaning across languages ("dizziness" finds "mareada"), with BM25 keyword search, which matches exact drug names and doses. RRF merges the two result lists, and the Jina reranker orders the final results.
Agent Builder: a patient-record agent answers doctor questions using ES|QL tools that can only read one patient's record. Each answer cites the quote it came from.
Workflows: a pre-visit-brief Workflow writes a summary for the doctor before each appointment.
Search during calls: Elasticsearch also finds visit passages for the voice agent while the patient is on the phone.

Cartesia is how most patients use Interlude: through phone calls in their own language.

Line agents place outbound check-in and visit-explainer calls. Each call gets the patient's recent symptoms as context, and the agents call our backend during the call to raise alerts, save questions, and look up the visit.
Ink speech-to-text transcribes visit recordings and live interpreter turns.

Known limitations:

Ink takes one language per request, so a mixed-language visit needs one transcription pass per language.
We poll for call results instead of receiving a call-ended event.

Design: While we built the backend, our designer made the mobile and web screens in Figma. In parallel, we built plain, unstyled versions of every screen so we could test each feature from start to finish. When the backend was solid, we migrated the design from Figma into our codebase, resulting in the final product.

Challenges we ran into

Visits often mix languages, with the doctor speaking one and the patient another. Cartesia's Ink speech-to-text takes only one language per request. To solve this, we transcribe each visit once per language in parallel and have Claude merge the passes turn by turn. Two developers worked on one backend at the same time. Splitting work while minimizing conflicts is difficult. Eventually, we agreed on function signatures up front and gave each developer their own deployment to avoid conflicts.

Accomplishments that we're proud of

Interlude places real phone calls and can hold a conversation in the patient's language. Concurrently, the voice AI agent is able to communicate with the backend and data can be displayed on the doctor's web side application within a minute. We were particularly proud of accomplishing this workflow because no one on the team had a lot of previous experience working with voice agents. Yet, we managed to demonstrate functionality.

What we learned

We learned that in healthcare, it is important to be able to trace each statement to a source. Life-altering decisions should not have an unknown origin. That's why every summary, answer, and timeline point links to what the patient actually said. We also learned how to work together as a team. For most of us, this was our first hackathon/tech competition, and adapting to a team environment has been immensely fulfilling.

What's next for Interlude

We want to add more languages and dialects, including regional accents and patients who switch between languages mid-sentence. To get ready for real clinics, we would like to develop a reliable authentication system and incorporate electronic health records.