MediCue

Inspiration

Imagine being in a hospital bed, fully aware of what's happening around you, but unable to tell someone that you're in pain, uncomfortable, or need help.

For patients recovering from strokes, living with ALS, or temporarily unable to speak after surgery, something as simple as asking for water or explaining where something hurts can become incredibly difficult. Nurses and caregivers do their best, but they often have to rely on gestures, yes-or-no questions, or guesswork to understand what a patient needs.

We we wanted to explore the idea: What if technology could meet patients where they are, instead of requiring patients to adapt to technology?

That's what inspired MediCue.

We wanted to build something that doesn't just give patients a way to communicate, but actively helps them find the words, reduces the effort required to express their needs, and makes sure important information reaches the people caring for them.

What it does

MediCue is an AI-powered communication companion that helps patients express their needs when speaking is difficult or impossible.

On the patients side: • Two taps to start. The patient picks where (Head, Chest, Belly, Surgery spot, Washroom and more, on a body map), then what it feels like. The second step's options are tailored to that patient. • One question at a time, only when needed. MediCue asks the single yes/no question that tells it the most, and stops as soon as it is confident. Some requests, like Washroom, go straight to the nurse with no questions. • Emergencies skip the questions. If a high-risk need such as chest pain or trouble breathing is likely, the nurse is alerted at once. • A screen that adapts. When answering gets harder (slower taps, corrections, repeated "not sure" answers, or silence), the tablet simplifies itself in steps: fewer and bigger choices, then one yes/no question at a time, then asking someone to help. It never loses what the patient has already said. • Voice when it helps. Patients can speak instead of tapping. MediCue matches what they said to a choice on screen and asks them to confirm it before anything is sent. Saying something urgent alerts the nurse straight away. The tablet can also read questions aloud. • Always one tap from help. An Urgent help button is on every screen, and Show to someone puts the patient's message on screen, in their own words, for anyone in the room. • Display and sound settings let the patient choose text size, button size, high contrast, reduced motion and read aloud.

Once enough information has been gathered, MediCue generates a structured, easy-to-understand handoff for nursing staff, helping communicate both what the patient confirmed and what remains uncertain.

On the caregiver's side: • Nurse station: a "Needs you now" queue (urgent first, then check-ins, then routine requests) beside a Unit right now board with one line per patient. Alerts arrive live. • One-tap confirm: the nurse records what the patient actually needed, and MediCue learns from it: how reliable each family member, nurse and the patient's own taps have been, and what this patient usually asks for. • Case packets for unclear requests: when MediCue stops without a confident answer, the nurse gets why it stopped, the needs still in play, and a concrete next step at the bedside, written as an SBAR handoff. • Shift summary for handover, and profile suggestions drawn from nurses' notes that apply only after a nurse approves them. • Patient profiles: chart flags, trusted contacts, chart notes, and what MediCue has learned from confirmations.

Beyond individual interactions, MediCue has two additional capabilities:

Longitudinal patient insights: By tracking communication events over time, MediCue can surface changes in reported needs, discomfort, or communication patterns that may deserve a caregiver's attention.
Autonomous Research Agent: MediCue can identify emerging patterns, generate hypotheses, investigate supporting evidence, and request clinician input. Instead of treating every correlation as a conclusion, it maintains uncertainty and requires human validation before findings are accepted.

Most importantly, MediCue is designed as a communication-support tool, not a replacement for clinical judgment. Our hackathon prototype uses synthetic patient data and does not provide diagnoses or independently validated medical alerts.

How we built it

We built MediCue as a full-stack application. It features a Next.js front end, a Python back end with engines we wrote ourselves, Tiger Data for time-series storage, and Gemini for language.

Frontend — Next.js, React, TypeScript, and Tailwind CSS

We created separate patient-facing and clinician-facing experiences. The patient interface prioritizes large interaction targets, simple prompts, a visual body map, and low-effort communication. The nurse dashboard presents patient requests, conversation summaries, and historical trends in a more structured format.

Backend — Python and FastAPI

Manages live sessions, alerts, prompting, silence checks and the research workflow. Every engine is deterministic Python with no network calls, so it is fast, testable and works without Gemini.

Bayesian Belief Engine

The belief engine keeps a probability for each of 20 needs in 5 groups (pain, breathing, toileting, comfort, and emotional or other). The starting probabilities come from context such as time of day, days since surgery and the patient's confirmed history. Every tap, answer, chart lookup and family answer updates them with Bayes' rule. Each source has a learned reliability (a Beta distribution), for example, a usually correct family member's opinion is weighted more heavily.

Decision Engine

After every update, the engine scores each possible move by how much it would reduce the risk of getting the need wrong, minus the patient's effort and the cost of delay weighted by clinical risk. The loss is asymmetric: missing chest pain costs 12.5 times more than confusing thirst with feeling cold. We also have a safety floor of 25% for high risk needs, which results in immediate escalation.

Google Gemini

Gemini writes SBAR handoffs and case packets, simpler wording for questions, family questions, the shift summary, and research hypotheses. It reads chart notes into patient profiles, ranks the tailored options, and suggests forecast adjustments. Every output is validated before it is used: profile items and forecast changes must quote a real fact and use fixed, capped multipliers. If Gemini is not available, all features have a template fallback. The nurse alert is sent before any Gemini call, so Gemini can never delay or suppress one.

Crucially, Gemini supports the communication layer rather than making independent clinical escalation decisions.

Tiger Data (Timescale DB + PostgreSQL)

Every tap, belief snapshot, chart event and confirmation is stored in hypertables. Continuous aggregates and time_bucket_gapfill power the Insights views, and Timescale Toolkit aggregates compute tap-speed percentiles and quiet stretches against each patient's baseline behavior.

Autonomous Research Agent

We built a research workflow that turns detected patterns and clinician observations into hypotheses. The agent can evaluate whether to investigate stored data, consult a clinician, or wait for additional evidence. It tracks supporting and contradicting information and keeps clinicians involved in validating findings.

Hypotheses begin at 50% confidence. The agent picks check the data, ask a clinician, or wait by expected information gain minus cost, and logs every score. Patterns are found on one half of the data and tested on the other, so the agent can't confirm its own guesses. Evidence is down-weighted so no single source settles a question, and every finding goes to a clinical lead for review.

Together, these components create a system that combines structured decision-making while leveraging generative AI.

Challenges we ran into

One of our biggest challenges was balancing intelligence with patient safety. It would have been easy to send a patient's input to an LLM and ask it to infer what they needed. But in a healthcare-related setting, confident-sounding guesses can be misleading, especially when the patient cannot easily correct them. We addressed this by separating responsibilities: our Bayesian engine tracks uncertainty, our decision engine handles action selection, and Gemini focuses on language and communication.

Another challenge was confidentiality. Because Gemini only rephrases messages our deterministic engines have already decided, it never sees the full patient record or makes a decision, and the app works fully with it switched off; access is also split by role, so a patient's tablet reaches only that patient's own record.

We also had to design around limited patient energy. Asking more questions might produce more information, but every additional interaction can also place a burden on someone who is already struggling to communicate. We had to think carefully about how to balance information gain against fatigue and response effort.

We found that making an interface technically accessible isn't the same as making it genuinely easy to use. Large buttons alone aren't enough if questions are complicated or the interface doesn't respond to the patient's changing needs. This motivated our adaptive communication approach.

Building the Research Agent introduced a different challenge: ensuring that a detected pattern wasn't automatically presented as an established finding. We needed a workflow that could gather evidence, represent disagreement and uncertainty, and incorporate clinician feedback without removing human oversight.

Finally, bringing the frontend, probabilistic engines, Gemini, and time-series data together into a coherent experience was a significant integration challenge, especially under hackathon time constraints.

Accomplishments that we're proud of

Building our own reasoning engines. Rather than making MediCue entirely dependent on an LLM, we designed custom Bayesian inference and action-selection logic. This gave us more control over how information is interpreted, how uncertainty is handled, and when the system should ask another question.

Making communication adaptive. We wanted MediCue to work with a patient, not against them. Designing an interface that can reduce communication complexity as interaction becomes more difficult was one of the most meaningful parts of the project.

Connecting individual communication to longer-term insights. MediCue doesn't treat each interaction as an isolated event. By preserving structured communication history, we can explore changes that might otherwise be overlooked.

Creating a human-in-the-loop research workflow. Our Research Agent goes beyond displaying charts. It creates hypotheses, investigates available information, and brings clinicians into the process before findings are considered validated.

Keeping the human purpose at the center. Even with all the technical components, we're proudest of building toward a simple outcome: helping someone communicate a need they might otherwise struggle to express.

What we learned

This project challenged how we think about building AI systems, particularly in sensitive environments like healthcare.

We learned that a useful AI system doesn't always need to give an immediate answer. Sometimes the right decision is to ask a better question, acknowledge uncertainty, consult another source, or involve a person.

We also learned that accessibility is not a single feature or setting. It's an ongoing interaction between the user and the system. What works for someone at the beginning of a conversation may become difficult a few minutes later, and good design needs to account for that.

On the technical side, we gained experience combining probabilistic inference, decision policies, LLM-based language generation, and longitudinal event analysis into one application. We also learned the importance of keeping those components modular so each can be tested and improved independently.

To tie everything together, we learned that the most meaningful application of AI isn't to automate decisions, but to give people more control over decisions that affect them.

What's next for MediCue

Our long-term vision is for MediCue to become an adaptable communication platform that can support patients across hospitals, rehabilitation centers, and long-term-care facilities.

Some of our next steps include:

Real-world usability testing: Work with clinicians, speech-language pathologists, accessibility specialists, patients, and caregivers to understand which interaction methods are most effective.
More personalized communication: Improve how MediCue learns each patient's preferred communication methods and typical interaction patterns while avoiding unsupported assumptions about their abilities.
Additional accessibility options: Explore eye-tracking, switch controls, and other assistive input technologies for patients with severe motor limitations.
Clinical-system integration: Investigate secure interoperability with electronic health records and existing nursing workflows.
Stronger longitudinal analysis: Explore whether communication changes can provide useful context for clinicians, with appropriate validation and safeguards.
Privacy and clinical validation: Evaluate reliability, security, consent, and safety requirements before considering use with real patient information.

Ultimately, we envision a future where losing the ability to speak doesn't mean losing the ability to be heard.

MediCue: A voice for every patient who cannot speak.