-
-
Architecture
Inspiration
Monica is my girlfriend's grandmother, and we all love her. When someone you love is getting ready to leave the hospital, everyone asks the same question: "Is she ready to go home?" But we realized there is another question that is just as important: "Is her home ready for her?"
Her care team can see her chart, diagnoses, medications, and discharge plan. What they cannot see is the narrow bathroom doorway, the loose rug, the dark hallway, or whether her walker will actually fit where she needs it. The family wants to help, but most of us do not know what to look for. A proper home-safety visit requires a clinician and often happens too late.
That is why we built SafeStep Ready. It lets a family member use a LiDAR-enabled iPhone to walk through the home before discharge. The system compares what the care plan requires with what the home actually allows, then brings clear evidence back to the care team.
The demo uses fully synthetic patient data, but the reason we built it is personal. We wanted to make the trip home safer for Monica and for every family having the same conversation.
What it does
SafeStep Ready starts with the discharge plan in Medplum. It reads the patient's FHIR Patient, Encounter, discharge note, and after-visit summary, then turns them into specific things that must be true at home: enough clearance for a walker, grab bars, night lighting, no unsafe stairs, and the right caregiver support.
Twilio texts the family a walkthrough link. Riley, an ElevenLabs conversational agent, guides them room by room without overwhelming them with clinical language. While they walk, three agents work together:
- The vision agent uses Claude Haiku for quick frame-by-frame callouts and Claude Sonnet for deeper grading against CDC STEADI and HSSAT guidance.
- The geometry agent reads Apple RoomPlan LiDAR data to measure rooms, openings, and doorways.
- The arbitration agent compares each discharge obligation with the evidence and marks it
verified,at_risk,blocked, orunverified.
In our demo, Monica is preparing to go home with a 28-inch walker. SafeStep measures her bathroom doorway at 27.1 inches. That one measurement changes the decision. The issue is no longer a vague "fall risk." The discharge plan, as written, will not work in her home.
SafeStep sends the exception to OT and the discharge team with the evidence, an owner, and a deadline. It also drafts FHIR Observations, ServiceRequests, and Tasks for review. DME requests include HCPCS codes and Medicare coverage notes. Nothing is ordered or sent automatically. A clinician stays in control and must approve every action.
How we built it
The iPhone app is written in Swift. It uses Apple RoomPlan and ARKit to capture the space, LiDAR measurements, and camera frames. Core ML runs YOLOv3 on-device to draw live detection boxes, while the backend handles the reasoning.
The backend is built with Python and FastAPI. Claude Haiku performs the fast vision pass so Riley can react during the walkthrough. Claude Sonnet handles the deeper review of hazards and supporting evidence. The same backend is connected to Riley through an OpenAI-compatible streaming endpoint, so the voice agent can use the chart context and live scan events in one conversation.
Medplum is the clinical source of truth. SafeStep reads FHIR R4 resources from the chart and creates a draft Bundle containing Observations, ServiceRequests, and Tasks. Twilio sends the walkthrough link to the family. A Next.js interface gives the care team the final report, evidence, floor plan, and draft actions.
We kept the phone intentionally simple. It gathers evidence and gives the family clear instructions. The reasoning, obligation tracking, escalation logic, and FHIR drafting live on the backend.
We also built a deterministic evaluation suite for the parts that should not depend on a model's opinion. It generates simulated RoomPlan exports, checks doorway-clearance math, tests escalation routing, validates every draft FHIR Bundle, and runs stateful twins of the Medplum and Twilio request contracts. Our committed baseline runs 6,994 assertions with zero failures across 500 simulated homes.
Challenges we ran into
The hardest part was turning a discharge note into something measurable. "Home is safe" is too vague to test. We had to break the plan into obligations and define what evidence would verify or block each one.
Geometry was another challenge. RoomPlan produces useful structure, but a doorway and a wide open span are not the same thing. Small unit or threshold errors can completely reverse a walker-clearance decision. We built edge cases around the decision boundary and tested malformed measurements instead of trusting the happy path.
We also had to balance speed with depth. A family member needs feedback while pointing the camera, but a careful safety review takes longer. We split the work into a Haiku fast pass for the live conversation and a Sonnet deep pass for the final evidence-backed report.
The clinical write-back needed strong guardrails. We did not want an AI system silently changing a chart or placing an order. Every downstream resource is marked as a draft, tied to its evidence, and held for clinician review.
Finally, connecting live voice, camera frames, LiDAR geometry, chart context, SMS, and FHIR write-back into one demo was a coordination problem. Each part worked at a different speed and had a different failure mode. Getting them to feel like one flow took most of the work.
What we learned
We learned that the useful output is not another risk score. A care team needs to know what failed, why it matters for this patient, what evidence supports it, who owns the next step, and when it must be fixed.
We also learned that families can collect valuable clinical evidence if the system asks simple, neutral questions. Riley does not tell someone that a room is dangerous while they are still scanning it. Riley helps them capture the evidence, and the care team gets the judgment with context afterward.
Most importantly, the chart and the home cannot be treated as separate worlds. A 27.1-inch doorway only becomes meaningful when the system knows Monica is coming home with a 28-inch walker. The value comes from connecting those two facts before discharge, when there is still time to change the plan.
What's next for SafeStep Ready
Next, we want to test the workflow with occupational therapists, discharge coordinators, patients, and caregivers. We want to learn which obligations matter most across different recoveries and how much evidence a clinician needs to act with confidence.
We also want to support phones without LiDAR by combining guided reference measurements with computer vision, add multilingual walkthroughs, and improve the remediation loop so a family can rescan a failed area and show that it has been fixed.
The long-term goal is simple: before someone we love is sent home, the care team should know that the home can support the plan.
Log in or sign up for Devpost to join the conversation.