Bedside (table 79)

Bedside home page
Bedside care team page

Inspiration

The riskiest days after surgery happen at home. An older patient leaves the hospital with a dozen medications, a stack of instructions they heard while groggy, and nobody checking in until a follow-up visit two weeks later. That gap is where blood clots, wound infections, missed blood thinners and falls go unnoticed, often until they become an ER visit or a readmission.

The tools meant to cover that gap all wait for the patient to act: a portal to log into, an app to open, a chatbot to ask. Older patients are the least likely to do any of those. When we talked with the FinchNode team, their challenge to us was simple: don't build another chatbot. Build something proactive that actually gets things done.

So we flipped the direction. Bedside doesn't wait to be asked. It calls you.

What it does

Bedside is a recovery companion a hospital prescribes at discharge.

Enrollment from the health record. A nurse picks the patient's record from FinchNode. Bedside reads their medications (with dosing times), conditions, allergies and baseline vitals, and builds a daily recovery plan. For example: on apixaban + aspirin → watch for bleeding; heart failure → weigh daily; kidney disease → never suggest ibuprofen.
A call every morning. Bedside phones the patient, with no app and no login. An ElevenLabs voice agent reminds them which medicines are due, then runs a short check-in in their language.
It acts before hanging up. Deterministic rules turn the answers into actions: alert the care team, book a follow-up visit on the clinic calendar, text and call a family member, or text a link to a camera vitals check. If the patient doesn't pick up after three tries, the family gets a "please check on Mom" call.
Contactless vitals. A 30-second front-camera recording estimates heart rate and respiratory rate (rPPG), compared against that patient's own baseline from FinchNode.
Clinician dashboard. Flags come first, followed by an English SOAP summary of every call, the pain trend, the plan, every message sent, and a plain-language log of everything Bedside did.

Bedside is not a diagnosis tool: it never tells a patient they have a condition. It reminds, asks, measures and escalates.

How we built it

Frontend: React + TypeScript + Vite. It includes a big-button patient app (incoming-call screen, mute, large text, day and night modes), a camera check-in page, a family page, and the care-team dashboard.
Backend: Python + FastAPI + SQLite. A scheduler places daily calls, retries missed ones and watches call outcomes.
Voice: an ElevenLabs Conversational AI agent with 11 webhook tools (get_daily_briefing, record_checkin, book_appointment, notify_family, record_family_response, and more). The patient and conversation IDs come from dynamic variables, never from the model.
Records: FinchNode's normalized patient records drive the plan. Signed records.updated webhooks (sandbox records.advance) trigger a call when medications change.
Reasoning: an LLM (Gemini, or Claude when configured) with structured outputs for symptom triage, plan personalization and clinician summaries. It always runs after a deterministic red-flag layer.
Calling channels: one codebase, three channels. Relay handles in-app calls and messages, Twilio handles real phone calls, or calls can ring in the web app.

Contactless vitals (rPPG + respiratory rate)

MediaPipe Face Landmarker runs in the browser, so video never leaves the device. Each frame becomes a few numbers: the mean RGB of skin patches on the cheeks and the bridge of the nose (well-perfused skin that hair and brows don't cover), the nose-bridge position, and facial blendshapes.

On the server, heart rate comes from CHROM, the chrominance method of de Haan and Jeanne. The RGB trace is cut into 1.6-second windows that overlap by half. Each window is normalized, Cn(t) = C(t) / mean(C), and projected onto two chrominance axes that cancel most of the change caused by motion and lighting:

X = 3·Rn − 2·Gn Y = 1.5·Rn + Gn − 1.5·Bn

Both are band-passed to 0.7–3.5 Hz (42–210 bpm) and combined:

S = Xf − (σ(Xf) / σ(Yf)) · Yf

Each window is weighted with a Hann window and added back into the full trace. The result is band-passed again, and the heart rate is the spectral peak:

HR = 60 × (the frequency with the most power in S between 0.7 and 3.5 Hz)

Respiratory rate is extracted from two signals in the 0.1–0.8 Hz band (6–48 breaths/min): the subtle vertical motion of the head with each breath, and the breathing-driven variation in the green channel. We keep whichever has the sharper spectral peak, and give a bonus when the two agree within 3 breaths/min.

Every reading carries a signal quality index (SQI). For heart rate:

SNR = 10 × log10(power near the peak f0 (±0.1 Hz) and its harmonic 2f0 (±0.2 Hz) ÷ all other power in the band) SQI_HR = (0.6 × clip((SNR + 6) / 12) + 0.4 × c) × q_face × q_light × q_motion

Here c measures how consistent the estimate is across sliding 10-second windows. Low-SQI readings are treated as missing data, and the patient is asked to retake with better light; they are never shown a number we don't trust. A high-confidence reading is interpreted against the patient's record (for example, "HR 112 vs. a typical 74 across 10 visits, on metoprolol"). A normal reading never cancels a red-flag symptom.

Challenges we ran into

The record didn't have everything. By design, FinchNode is read-only, with no family contacts, no surgical history and no scheduling. So we capture the discharge episode and family contacts (with consent) at enrollment, and run our own clinic calendar.
Safety can't depend on the model. Emergencies (chest pain, trouble breathing, stroke signs, self-harm, in English and Spanish) are caught by deterministic rules first. During testing, "hi I'm dying" slipped past the rules. The LLM layer caught it, but our family-call hook only fired on the rules path. We fixed it so every route to "call 911" also calls the family, with de-duplication.
Should the bot call 911 itself? We decided no. A server-placed 911 call shows the wrong location, can't answer a dispatcher's questions, and false alarms send real ambulances. Instead, Bedside tells the patient to call 911 and immediately calls their family with the address on file.
Real-world signal quality. Our first live heart-rate capture scored SQI 0.49 against a 0.5 cutoff. We gave heart rate its own threshold (0.4); mid-confidence readings are shown but never used to raise urgency. We also flag atrial fibrillation, since an irregular rhythm undermines camera estimates.
Telephony is full of sharp edges. ElevenLabs requires flash/turbo v2 voices for English agents. Calls bridged through Twilio need μ-law 8 kHz audio, while the browser needs 16 kHz PCM, so we run a separate phone copy of the agent. Twilio's trial blocks several call parameters, so we poll for call outcomes instead.
Flaky upstreams. The LLM API returned intermittent 503 errors at peak times, so we added retries and a fallback chain of models, plus rule-based fallbacks if every model is down.

Accomplishments that we're proud of

A full loop that acts, not just talks: call → check-in → care-team alert → booked visit → family called, all before the patient hangs up.
Safety-critical logic lives in code, covered by 113 automated tests. They include red-flag detection, the rPPG pipeline on synthetic signals with known rates, scheduler retries, and webhook signature verification.

What we learned

Proactive beats reactive for the people who need care most. The best interface for an 80-year-old is a phone that rings.
Intelligence includes knowing when not to trust your own data. A quality score that says "retake" is more valuable than a confident wrong number.
Ground everything in the record. Generic advice is easy; "take your apixaban, and tell me if you see unusual bruising" is what actually helps.
Integrating real infrastructure (voice, telephony, health records) is mostly about the unglamorous details: audio formats, webhooks, retries and consent.

What's next for Bedside

Validate rPPG accuracy against a pulse oximeter across lighting conditions and skin tones, and measure respiratory-rate error.
Read vitals directly from the video call itself (Relay delivers the patient's camera frames to the agent).
Partner with emergency-data services (such as RapidSOS) to share location and context with 911 the right way.
Pilot with a care team and measure missed complications and readmissions.