Inspiration

We started with a simple question: could an ordinary webcam help someone check in after a possible fall?

We wanted a system that gives the person a chance to respond, keeps camera processing local, and connects to a caregiver through a familiar app.

What it does

Our application watches body movement through a webcam. When it detects a possible fall, it plays a spoken check-in and starts a response countdown.

The person can cancel by saying “I’m okay,” pressing a key, or clicking a button. If the countdown expires without cancellation, the application attempts one caregiver alert through Telegram.

The interface also shows live posture estimates, including upper-body orientation when the legs are outside the camera view. Ranked activity scores animate as the estimates change.

How we built it

We built one Python application with three parts: detection, safety, and device controls.

  • OpenCV captures webcam frames.
  • MediaPipe Pose extracts body landmarks.
  • A pretrained Transformer, running locally through LiteRT, analyzes movement sequences.
  • A state machine manages check-ins, cancellation, countdowns, and alerts.
  • sounddevice plays recorded prompts, while Vosk recognizes cancellation phrases offline.
  • The Telegram Bot API connects the application to a caregiver.
  • A local HTML, CSS, and JavaScript interface provides camera selection, live feedback, and controls.

Camera frames and microphone audio stay in memory. Caregiver notifications contain alert text rather than recordings.

Challenges we faced

Connecting a pretrained model to live camera input required careful attention to landmark order, coordinates, normalization, and timing.

Voice cancellation brought another challenge: selecting the correct microphone and recognizing a short phrase consistently. Comparing microphone inputs and using a focused response vocabulary helped us recognize a live spoken response.

We also needed to handle incomplete views honestly. Instead of guessing whether someone is standing or sitting when their legs are hidden, the interface switches to upper-body posture estimates.

What we learned

We learned that a useful AI application needs more than a model score. Clear feedback, responsive controls, and a well-designed response flow matter just as much.

Separating detection from incident handling also helped us keep the countdown and cancellation controls responsive while audio and notification work happened in the background.

What’s next

We want to improve posture estimation across camera angles, make microphone setup easier, and refine the experience through more real-world demonstrations.

Built With

Share this project:

Updates

Submission history