Life Vision

• 

  main screen

• 

  isaac simulation

• 

  life vision logo

• 

  fine tuning results

Inspiration

Every summer, a camera watches a pool and sees nothing. A toddler slips into the water while a parent looks away. There is no commotion, just silence. Drowning is the leading cause of death for children ages 1 to 4. After decades of decline, U.S. drowning rates began rising again in 2020.

What it does

LifeVision turns the cameras people already have into a second set of eyes. Over 61% of U.S. households already own security cameras, and that number is growing 22% every year. Using standard smart-home APIs, LifeVision sends live camera feeds straight into our detection model, so a camera that just records becomes one that actually watches. An adult still has to respond, but the silence never goes unnoticed.

How we built it

Since we can't stage real drowning situations, we used two sources of data. The swimming pool footage we had was fairly low resolution, so we used it to pick out the people in each frame and build ground truth data, which we then used to fine-tune our machine learning model. On top of that, we simulated drowning scenarios in NVIDIA Isaac Sim and Google DeepMind's MuJoCo to get tracking data we couldn't film. This helps the model keep tracking a person even when they're hard to see underwater. We paired it with Google's Gemini to send alerts that are hard to ignore. LifeVision isn't a lifeguard replacement. It's the camera, finally paying attention.

Challenges we ran into

• Deciding which model to select, and which models were capable of fine-tuning.
• Generating a dataset, since footage of kids and babies drowning is difficult to find. This led us to simulate those scenarios in NVIDIA Isaac Sim and Google DeepMind's MuJoCo.
• Learning the simulators themselves to set up the conditions we wanted to test.

Accomplishments that we're proud of

• We actually used the simulations to create more datasets.
• We went through the fine-tuning process on real swimming pool footage, and then applied the fine-tuned model in the Isaac Sim environment to test our edge cases as more data was generated.

What we learned

We learned how to:

• Run simulations to generate datasets, with the right physics settings for each scenario.
• Use those generated datasets to fine-tune our model.
• Apply the trained model to a real-life situation using swimming pool footage.

What's next for LifeVision

Next, we want LifeVision to go beyond children in the water and also detect people tripping or getting injured in industrial settings like factories, such as falling or getting stuck in a machine. Those are the moments we want to catch live, so more lives can be saved. On the technical side, we want to keep using the datasets we generated and the Isaac Sim simulations to improve our model.

Right now, we're using a very lightweight model that runs on a local machine, but our team would like to try heavier models that give higher accuracy. We could then give swimming pools and factories the option of running it locally or on our server.

Built With

• bytetrack
• computer-vision
• fastapi
• isaac-sim
• javascript
• machine-learning
• mujoco
• object-detection
• python
• pytorch
• rf-detr
• yolo