Sensara

Elderly patients have died unnoticed due to delayed care in nursing homes. Our AI system detects critical events in real time and alerts caregivers instantly, enabling faster, life-saving responses.

Comment

Inspiration

  • In Ohio, an 82-year-old nursing home resident went missing and was found dead outside the facility after 2 days; staff did not realize she was gone.
  • In Iowa (2026), a resident was put to bed while struggling to breathe and later died, leading to fines and violations against the facility.
  • In Florida, an 83-year-old man with dementia was found dead in a freezer after going missing from his room, discovered only after family noticed.

What it does

Our project uses AI-powered real-time monitoring to detect critical events instantly and trigger alerts the moment something is wrong. By combining live detection with secure video capture and rapid notifications, caregivers can respond immediately, turning delayed reactions into fast, life-saving interventions.

How we built it

  • Frontend & Framework: Next.js 14 App Router, TypeScript, and Tailwind CSS.
  • Vision: Google MediaPipe Pose Landmarker runs in-browser at about 30 FPS, tracking 33 body landmarks for fall, posture, and wandering detection.
  • Audio: ElevenLabs Scribe v2 STT analyzes rolling 6-second microphone chunks to detect distress signals like thuds, gasping, or “help.”
  • Recording: The browser MediaRecorder API captures camera video and microphone audio into a single .webm clip.
  • Storage: Video clips upload directly from the browser to AWS S3 using presigned URLs, so the backend does not proxy large files.
  • Backend: Next.js API routes handle ElevenLabs, S3 presigning, and event CRUD.
  • Database: MongoDB Atlas stores structured safety events and S3 file references.
  • Detection Logic: Rule-based classifier with cooldowns, safe-zone debounce, torso-center tracking, and configurable thresholds.

Challenges we ran into

  • Reducing false positives during real-time detection.
  • Keeping video and audio capture synced in the browser.
  • Uploading large video clips securely without overloading the backend.
  • Balancing fast alerts with reliable event detection.

Accomplishments that we're proud of

  • Built a real-time elderly safety monitoring system.
  • Combined computer vision, audio detection, alerts, and secure video storage.
  • Used S3 presigned URLs for scalable and private clip uploads.
  • Created a solution focused on faster caregiver response.

What we learned

  • How to build low-latency browser-based AI detection.
  • How to combine MediaPipe, audio processing, MongoDB, and AWS S3.
  • How presigned URLs keep uploads secure and scalable.
  • How important speed and reliability are in healthcare safety tools.

What's next for Sensara

  • Improve detection accuracy with more testing.
  • Add SMS, email, and mobile push alerts.
  • Build a caregiver dashboard for live monitoring.
  • Test the system in real care environments.

Built With

Share this project:

Updates