Landing page with the 3D anatomy twin (4k)
Kadrian's twin made from a photo, with check-ins, health data and Twin Pod vitals (4k)
Nurse console (4k)
Nurse console in anatomy view, with the risk score, alerts and audit trail
Elena's twin, seated in a wheelchair (4K)
Apple Health, Fitbit and device sources (4K)
Patient privacy and access log (4K)
External-validation ROC: model 0.659, NEWS2 0.627, qSOFA 0.569
TwinPatch Chest with two site pods
The pods placed on the anatomy model
The knee pair and the calf pod
The chest patch over the heart
Exploded view of the chest patch
Cross-section of the chest patch
Blueprint TP-CH-001

Hospitals send patients home sooner every year. A knee replacement goes home in a day, and heart failure, COPD, diabetic wounds and spinal cord injuries are managed from the living room. That is also where complications start: a blood clot in the calf, an infected incision, fluid building up in the lungs. Today the safety net between visits is a phone call.

Our team is a software engineer and a nursing student. Kadrian has experience in a medical ICU as a nurse intern, and she has seen where care breaks down:

Patients struggle to describe what they feel. "It hurts around here" over the phone is hard to triage, and patients often can't tell "normal" from "call someone now."
Nurses translate vague reports by hand. A follow-up nurse with 25 patients turns each phone call into a structured assessment (OPQRST) and a handoff to the physician (SBAR).
Warning signs arrive between appointments. By the next visit, a symptom has either gone away or become an emergency.

The numbers are hard to ignore:

1 in 7 U.S. hospital stays is followed by a readmission within 30 days (AHRQ HCUP).
1 in 4 Medicare heart-failure patients is readmitted within 30 days (JAMA, 2013).
More than a third of 30-day readmissions happen in the first week home (AHRQ HCUP).
A readmission costs on average 12.4% more than the original stay, and 80% of U.S. hospitals face Medicare readmission penalties in FY2027 (Healthcare Dive).
41% of rural hospitals operate in the red, and 300+ have stopped obstetrics (Chartis).

Our mission: improve rural health outcomes and lower healthcare costs by giving every patient recovering at home a live 3D twin, shared in real time with their care team.

What it does

Digital Twin Biomedical is a live 3D twin of every patient recovering at home, shared in real time between the patient and their nurse. Patients point to where it hurts on their own body instead of filling out forms. A wearable streams their vitals, and their Apple Watch, Fitbit, scale and blood pressure cuff feed the same record. Fixed safety rules watch every reading, AI drafts the clinical handoff, and nothing goes to a doctor until a nurse approves it.

For the patient

Their own body, in 3D. Upload a full-body photo and the twin is generated to look like them. Switch to Anatomy to see muscles, bones and blood vessels, or focus on the area their condition affects: the knee, head, heart, lungs, foot or spine.
Tap where it hurts. A tap drops a pin on the exact body region ("Right calf (back)") and opens a short check-in with large chips and a pain slider instead of free text.
Live vitals from the Twin Pod. Heart rate, blood pressure, respiratory rate, SpO₂ and temperature stream onto the dashboard and onto the twin itself.
Their health record, Apple Health style. Activity rings (Move, Exercise, and Stand, or Roll for wheelchair users), every Apple Health category, day to six-month charts, and logging for blood pressure, weight, glucose, mood, cycles, workouts and medication doses.
Their own devices. Patients import an Apple Health export, which is parsed in the browser so only daily or hourly summaries are uploaded. Fitbit and Pixel Watch connect through the Google Health API.
A health agent that knows them, by text or voice. "How am I doing today?" gets an answer grounded in their own twin and readings. Voice is hands-free: the agent notices when you pause, answers out loud, and stops if you talk over it. That matters for someone post-op, iced and elevated.
Clear status, never raw AI output. Patients see "Your care team has been notified," then "Marcus has reviewed your check-in." Chest pain or sudden shortness of breath shows Call 911 now instantly, from a fixed rule.
Privacy they can check. Patients see who opened their chart and why, and can download their record.

For the nurse

A census sorted by who needs attention, with each patient's condition, live heart rate and alert priority.
The same live twin, with pins. Check-ins appear on the nurse's twin the instant the patient sends them. The nurse's pointer also shows on the patient's twin, so they can talk about the same spot.
Condition-specific trends: daily weight and sodium for heart failure, glucose for diabetic wounds, SpO₂ and respiratory rate for COPD, steps or pushes for mobility, plus missed doses and wearable flags.
A deterioration risk score from a model trained on 40,336 real ICU stays. It shows the chance of sepsis onset in the next 6 hours, its trend, and which vital-sign patterns drove it.
An SBAR handoff already drafted. When a rule fires, Claude drafts the Situation, Background, Assessment and Recommendation. A second model lists what the draft is uncertain about and what to ask the patient. The nurse edits, approves, escalates or resolves it.
A clinical agent, by text or voice, that helps prioritize patients, interpret trends and tighten SBAR wording.
A full audit trail and HIPAA controls. Every check-in, chart open, rule, AI draft and nurse action is timestamped and read-only. The console locks after 90 seconds idle.

Safety by design: AI drafts, humans decide

Fixed rules run first, with no AI. They check for possible pulmonary embolism, blood clot, infection, uncontrolled pain, low SpO₂, tachycardia, fever, tachypnea, abnormal blood pressure, heart-failure weight gain, irregular rhythm notifications and dangerous glucose. Each rule sets a priority floor.
AI can only raise priority. It can never lower a rule's priority and never declares an emergency.
Every AI draft needs a nurse's approval, and edits are recorded.
A second model checks the first. An independent DeepSeek review sees the draft with the patient's name removed.
The risk model is advisory. It can open a review, never an emergency, and never dismiss one.
Least-privilege context. The patient's agent sees only that patient's record, and API keys and OAuth tokens never reach a browser.

HIPAA safeguards

The product implements the technical safeguards expected of a HIPAA-ready monitoring console: unique user access control, automatic logoff, append-only audit controls, purpose-of-use logging on every chart open (treatment, payment, operations or break-the-glass), encrypted transmission, minimum-necessary AI context, an accounting of disclosures patients can read, and right of access through Download my record.

How we built it

SpacetimeDB is the whole backend. Every table, reducer and safety rule lives in one TypeScript module on SpacetimeDB Maincloud. Clients subscribe to tables, so a patient's tap shows up on the nurse's screen with no polling and no separate API server. Alert rules run inside the same transaction that writes the check-in or vital sign, so a red flag can't be missed between services.

Network calls happen only in scheduled procedures. When a rule opens an alert, runSbarJob builds a structured, name-free record of rule results, recent check-ins and vitals statistics. It asks Claude for an SBAR draft through a strict tool schema, asks DeepSeek on Bitdeer to review it, and writes the result back for nurse approval. If a model is unavailable, a template draft keeps the nurse unblocked. runTwinJob sends the patient's photo to Meshy image-to-3D and saves the textured model on the patient.

The 3D twin uses React, Vite, three.js and React Three Fiber. Photo scans are normalized to a standard height, so tap-to-region mapping, hotspots and pins work on any body, standing or seated in a wheelchair. The Anatomy view is a meshopt-compressed model derived from BodyParts3D (skin, muscle, bone, arteries and veins).

Voice chat uses OpenAI gpt-audio through OpenRouter. An energy-based turn detector in the browser adapts to background noise and sends each turn as 16 kHz audio. The server streams the reply back as 24 kHz PCM, which starts playing on the first chunk, and a parallel pass transcribes both sides into the chat.

Predicting deterioration. A patient at home doesn't have labs, notes or diagnosis codes, so we limited the model to what a wearable measures: heart rate, SpO₂, temperature, blood pressure, respiratory rate, mean arterial pressure, shock index and age.

Data: the open PhysioNet/CinC 2019 sepsis challenge, 40,336 ICU stays from two hospitals, predicting sepsis onset within 6 hours.
Features: 89 per hour, including latest values, 6 h and 24 h statistics, slopes and change from baseline. We left out labs and charting frequency so the model learns physiology, not hospital workflow.
External validation: trained on hospital A and tested on hospital B, which it never saw. AUROC was 0.659, against 0.627 for NEWS2 and 0.569 for qSOFA. At the "elevated" threshold it flags 49% of septic patients a median of 8 hours before onset.
It runs in the database. The 161 XGBoost trees are exported to TypeScript and scored inside the SpacetimeDB module in the same transaction that writes each reading. Tests match XGBoost to within 0.000001.

Predicting hypotension from a pulse waveform. In a companion study (Smith, 2026, Zenodo) we authored, a 85-million-parameter transformer was pretrained on open VitalDB waveforms from 6,388 surgeries on 2× NVIDIA B200 GPUs. It was then fine-tuned to predict hypotension within 5 minutes from 30 seconds of a finger pulse. On held-out patients the AUROC was 0.693, against 0.676 for gradient-boosted trees.

The hardware: TwinPod. The simulated Twin Pod has a real design behind it, complete to Rev A with schematics, power budgets, a BOM, the BLE contract, blueprints and 3D models (3D viewer).

TwinPatch Chest (54 × 28 × 8 mm, 12.9 g): single-lead ECG, heart rate and RR intervals, respiration and chest skin temperature on an nRF5340, with about 7 days per charge.
TwinPatch Site (22 × 28 × 7 mm, 4.8 g): ±0.1 °C skin temperature and a 6-axis IMU on an nRF52840. A pair above and below the knee gives knee angle.
Kit Dock: charges one chest patch and three site pods over USB-C.

Deployment. The site runs on Vercel, with the agent, voice and twin-download routes as Node functions. To keep the public demo affordable, each visitor gets a fixed number of AI turns, enforced by a separate SpacetimeDB database that stores only hashed keys.

Built with: SpacetimeDB, React, Vite, TypeScript, three.js, React Three Fiber, Anthropic Claude, DeepSeek on Bitdeer, OpenAI gpt-audio via OpenRouter, Meshy, XGBoost, scikit-learn, the Google Health API, Apple Health export, Vercel and Cursor.

Challenges we ran into

The SpacetimeDB sandbox. new URL() throws for every input inside the server runtime, so the twin job rejected every finished Meshy asset. We replaced it with structural URL validation that works in any runtime.
CORS on 3D assets. Meshy's asset CDN sends no CORS headers, so browsers couldn't load the model. We built a same-origin, allowlisted proxy for GLB downloads.
Silent render crashes. Two uncaught GLTF loader errors killed the render loop with no message. A safe loader now falls back from personal scan to reference model to anatomy skin.
Honest machine learning. Predicting sepsis 6 hours ahead from vitals alone is hard. Tuning only moved external AUROC between 0.65 and 0.67, and the model overestimates risk on hospital B. So we made it advisory: it leads with level and trend, and can only raise a review.
Real-time without a real sensor. Each simulated 5-second reading stands in for one hour, so a pattern that builds over a day plays out in about two minutes during a demo.
Keeping AI costs bounded on a public demo, with signed per-visitor and per-network turn limits and a daily cap on AI SBAR drafts.

Accomplishments that we're proud of

A patient's tap and the nurse's twin stay in sync in real time, with safety rules running in the same transaction that saves the data.
A deterioration model that beats NEWS2 on a hospital it never saw, scored inside the database with no model server.
A real person's 3D twin generated from one photo, working end to end.
HIPAA-style technical safeguards, on by default.
A complete Rev A hardware design for the TwinPod kit.
A peer-citable waveform study trained entirely on open data.

What we learned

Safety has to be architecture, not a disclaimer. Fixed rules first, AI second, a human last.
Putting logic in the database removes whole classes of sync bugs between patient and nurse.
Report model limits plainly. External validation and calibration matter more than a headline number.
Patients at home need low-effort input. Tap and talk beats forms, especially after surgery.

What's next for Digital Twin Biomedical

Build the TwinPod boards and add the ingest reducer, so measured readings replace the simulated pod.
Native HealthKit and Health Connect sync through a companion iOS and Android app.
Voice check-ins that log a symptom on the twin ("my left calf is tight and warm") as a check-in the nurse can act on.
Validated thresholds reviewed by clinicians, and a triage evaluation set labeled by nurses.
A home-trained outcome model, retrained on pilot outcomes with wearable trends and labs.
A pilot with about 20 patients at one orthopedic practice for six weeks, compared with its phone follow-up. It would measure days from the first warning sign to nurse contact, ER visits and readmissions, nurse minutes per patient, and patient satisfaction.
EHR integration through FHIR and SMART on FHIR, reimbursement through Medicare remote monitoring codes (CPT 99453, 99454, 99457 and 99458), and the FDA Software as a Medical Device 510(k) pathway.