Inspiration

Recovery does not fail only in the hospital, it often fails at home, when patients are sent back with dense discharge paperwork, changing medications, and no easy way to sanity-check what comes next. Research shows post-discharge medication discrepancies are common, patient understanding is often incomplete, and better discharge communication can meaningfully reduce readmissions. In the age of AI a lot of people ask a multitude of AIs that might have limited access to supported medical data. Hearthside exists to close that gap with AI support that turns discharge instructions into clear, actionable guidance.

What it does

Hearthside gives post-hospital patients a team of AI agents — one reads your prescriptions and flags interactions, one guides your recovery plan, one knows when to tell you to call 911. Upload a document and it extracts the details. Turn on your camera and it watches your movements to coach physical therapy exercises.

How we built it

FastAPI + LangGraph for agent orchestration, Next.js frontend with SSE streaming, ChromaDB for discharge context, MediaPipe for hand/movement tracking, Florence-2 VLM for reading prescription images and documents. Modal is used for spinning up quick functions and inferencing on these open source models and give them as tool-calls for our agents.

Challenges and accomplishments

Getting multiple agents to feel like a coherent team rather than separate chatbots. Streaming structured data reliably over SSE. Making MediaPipe movement feedback actually useful for PT without overstepping into diagnosis. Overall orchestration is harder than getting models to work individually. Most of the work is in routing, context sharing, and knowing when to hand off through complex decision graphs— not in the model calls themselves.

We were proud of being able to at least have a real-time multimodal pipeline that handles both text and camera input in the same session and agents that are able to talk over each other and bounce ideas with access to many OSS models and apis for backup. Agents that share context about your discharge rather than starting from zero every message.

What's next for Campfire

EHR integration, caregiver view, push alerts for missed medications and larger fine-tuned source-backed databases, expanding PT exercise library with validated movement benchmarks, additional diagnosis abilities through real-time VLMs and seamless handoff to caregiver if required.