TellyHealth

Inspiration

Medical errors cause over 250,000 deaths each year in the U.S., but the more common, and often invisible, problem happens every day: patients leave telehealth appointments confused, overwhelmed, and unsure whether their new medications are safe to take alongside existing ones. Studies show patients forget 40–80% of what doctors say almost immediately.

At the same time, clinicians are under intense time pressure, making it easy to miss drug interactions or signs that a patient doesn’t fully understand their care.

We wanted to build something that protects patients without slowing doctors down: an AI safety layer that listens, understands, and intervenes only when necessary.

That led to TellyHealth: an AI-powered shadow companion for telehealth visits that quietly safeguards medical conversations in real time.

What it does

TellyHealth is an AI patient advocate that sits silently inside telehealth calls to:

• Detect and surface prescribed medications the moment they’re mentioned
• Cross-check new prescriptions against the patient’s existing conditions and medications in real time
• Flag dangerous drug interactions before harm occurs
• Translate complex medical language into patient-friendly explanations
• Generate a clear, structured post-visit summary that patients can actually understand TellyHealth never interrupts the doctor. Instead, it works in the background, acting as a safety net that ensures patients leave informed, confident, and protected.

How we built it

TellyHealth is built as a real-time, privacy-first AI system using the sponsor stack:

• LiveKit powers the telehealth video and audio experience, enabling real-time participation in the call. -Overshoot AI analyzes the video feed to detect non-verbal signals such as confusion, stress, or rushed explanations, converting visual cues into structured signals that the system can reason over.

To preserve patient confidentiality, TellyHealth is designed with federated learning principles, ensuring sensitive medical data is processed locally and never centralized.

Challenges we ran into

• Medical safety boundaries: Ensuring TellyHealth assists without replacing clinical judgment.
• Hallucination risk: In healthcare, a single incorrect suggestion can be dangerous.
• Real-time latency: Combining video analysis, NLP, and medical lookups without slowing the live call.
• Privacy: Designing an always-on assistant that enhances care without storing or exposing sensitive data.

Accomplishments that we're proud of

We successfully built a live, real-time telehealth safety system that can detect medication mentions instantly, cross-check them against a patient’s medical history, and flag dangerous interactions before they reach the patient.

Our demo shows TellyHealth catching a real-world drug conflict and enabling the doctor to immediately switch to a safer alternative. This interaction happens silently, without interrupting the visit.

We’re also proud that TellyHealth goes beyond transcription. It generates a patient-friendly summary that organizes medications, dosage instructions, and explicitly documents which drugs were avoided and why—solving the “I forgot what my doctor said” problem at scale.

What we learned

We learned that healthcare AI must act as an assistive layer, not an authority. The most effective systems don’t talk over doctors but instead quietly protect patients.

Incorporating video-based reasoning through Overshoot showed us how much meaning is lost when systems rely on text alone. Confusion, hesitation, and rushed explanations are often visible before they’re spoken.

Most importantly, we learned that trust is everything in healthcare. Continuous evaluation wasn’t optional; it was essential to proving that an AI system can be safe, responsible, and worthy of real clinical use.

What's next for TellyHealth

Next, we plan to expand TellyHealth’s medical intelligence by integrating additional trusted clinical databases and, with explicit patient consent, connecting to EHR systems for deeper personalization.

We also plan to add multilingual support, allowing patients to receive explanations in their preferred language.

On the safety front, we’ll continue strengthening our federated learning approach to further reduce data exposure while improving performance.

Finally, we aim to pilot TellyHealth with real clinicians and telehealth providers to measure its impact on patient comprehension, medication safety, and overall quality of care.

Built With

• css
• html
• javascript
• livekit
• overshootai
• react
• typescript