Panoramic

GIF

Inspiration

My grandmother lived in a residential care center. She was active, clear-minded, and still went for walks after dinner. While her caregiver was away, she spilled water on the floor in her room. Then she slipped and broke her hip.

Two weeks later, she died on my seventeenth birthday.

Every birthday brings me back to the same question: how could something as ordinary as spilled water take away someone who still had so much life in her?

The gap was not simply whether a caregiver had been assigned. It was the time between a change in her surroundings and someone knowing that help was needed. Being clear-minded does not mean a person will notice and respond to every hazard in time. Care should not depend entirely on a resident recognizing danger, finding the words, and asking for help.

A staffing ratio also cannot tell us who is actually available. Even with one-to-one care, a caregiver cannot be beside a resident every second. Technology cannot replace enough hands on deck, but it can help make the need for those hands visible sooner.

What if a resident's living space could help notice a new hazard, warn locally, and get the right information to an available caregiver?

That is why I am building Panoramic.

What Panoramic does

Panoramic is a caregiver-facing prototype for noticing possible environmental hazards, coordinating available help, and retaining what happened. It is motivated by the time between a change in a resident's surroundings and an available caregiver knowing that help is needed.

The prevention workflow is: observe → assess the walking route → open a concern → assign help → respond → check safety → retain a handoff. An available, qualified caregiver is chosen by a supervisor; being nearby alone does not mean that person is free. Assignment, acceptance, arrival, and outcome are separate events. A recorded outcome does not close a concern: supervision must confirm a physical safety check, followed by an independent nursing-station check.

What the prototype demonstrates today

Resident Floor shows four sample suites and a 3D floor view with room-level concern states. A route marked in a staged image gives a possible hazard a level of L1 (outside the route), L2 (near it), or L3 (intersecting it). With no marked route, the concern remains unassessed.
Bathroom playback uses an annotated, staged A101 recording. Its authored water event opens a concern and shows the response sequence from supervisor assignment through caregiver arrival and the two station checks. The recording does not detect water from its pixels or alert a real care team.
Scene review accepts a deliberately selected room photo or video frame. A server-side Gemini integration requests structured candidate observations and a brief, with validation and uncertainty. The integration is deployed, but a successful live model analysis and its accuracy have not been verified.
Panoramic AI is designed to answer questions about authorized, facility-scoped records and cite the underlying evidence. It may suggest an assignment, but only a supervisor can confirm one; the model cannot dispatch, sign off, or close a concern.
Care session retains the Prepare, Support, and Handoff workflow. It blocks a routine with missing support confirmations, separates help acknowledgment from arrival and resolution, respects refusal and withdrawn consent, and exports unresolved concerns.

This is a fictional-data prototype. It is not a live camera system, a trained water detector, a medical device, or validated fall prevention. There is no SMS, phone push, or external emergency dispatch.

How I built it

React, TypeScript, and Vite power the workspace. Three.js renders the interactive floor. The bathroom recording uses manually marked regions tracked with OpenCV optical flow; its water event is authored playback, not automatic recognition. Explicit TypeScript transitions govern the local care session.

For shared incidents, Supabase PostgreSQL stores facility-scoped records and server-checked response commands. Row-level security limits record access. Supervisor assignment checks availability, role, and competing assignments. The database checks response deadlines; the app uses Realtime and polling for updates. The Gemini API sits behind a private Vercel function, so the provider key is not in browser code. Downloadable handoffs are assembled from recorded events without inventing a model summary.

OpenAI Codex helped with implementation, debugging, documentation, and tests. It is a development tool used to build Panoramic, not a live OpenAI feature inside it. Daily notes and audio capture were removed from this release. The repository documents the earlier SteadySide idea and the new implementation's history.

Challenges I ran into

The hard part was connecting detection to an actual response. A notification is not an acknowledgment; an acknowledgment is not arrival; a caregiver's outcome note is not proof that a floor is safe. That is why the prototype keeps those stages separate and requires supervision and nursing checks before a concern leaves the active queue.

Visual evidence has limits too. A cup on a table does not prove that water is on the floor, and the original dry reference photo does not prove the room was cleaned afterward. The demo labels staged input and leaves physical safety to people. I also had to keep an unavailable caregiver out of the assignment path and ensure a new hazard invalidates earlier clear evidence and approvals.

Accomplishments and what I learned

The prevention-focused build has 141 passing local tests, including workflow transitions and isolated PostgreSQL checks for wrong roles, stale updates, facility isolation, invalidated approvals, and dual sign-off. The production build passes, and the caregiver workspace is deployed. A local browser walkthrough completed the staged bathroom sequence from L3 concern to outcome, supervision check, nursing check, and retained handoff.

These checks verify the prototype's logic, not its performance in a care facility. Hosted multi-account behavior and successful live Gemini output still need verification. The lesson shaping Panoramic is simple: useful care technology must show who can respond, what has actually happened, and what remains unresolved.

What's next

Validate live Gemini output on staged, non-sensitive images; test authenticated multi-client delivery and private evidence; evaluate hazard detection against labeled examples; and work with care professionals on privacy, accessibility, staffing, and real-world safety requirements. No real resident data belongs in this demo.

The idea grew from earlier exploration and a separate prototype. Panoramic's repository preserves that provenance; hackathon eligibility for earlier concept work still needs organizer clarification.

Try it

Open Panoramic · Enter the caregiver workspace · View the source code

For an account-free walkthrough, open Resident Floor and use A101's Play tracking. The staged event opens a concern. Assign a responder in the playback, record acceptance, arrival, and outcome, then review the dry reference and complete the separate supervision and nursing checks. The playback is isolated from shared care-team records and clears on refresh.

The repository explains the prevention loop, verification limits, and project provenance.