Each year, thousands of people die at sea each year because rescuers can't reach them in time.

Rescuers can be aided by drones, but watching live footage is a huge bottleneck. To closely watch live drone footage for anomalies in the water, you'd need many human operators all tuned in at the same time.

What if we could use IBM Watson's Visual Recognition to cut the signal from the noise?

What it does

Watson Search and Rescue uses Watson's Visual Recognition API with a realistic 3D simulation to model how machine learning and a drone swarm could aid rescue efforts at sea.

How we built it

Watson Search and Rescue models one square kilometer of open ocean to scale in Unity3D. We built a simulated swarm of camera-mounted drones. Somewhere in the open ocean is an overturned boat.

These drones traverse the open ocean like a search party, sending images and their coordinates every 100 meters to the Watson Visual Recognition API.

We trained a Watson custom classifier on sample images of aquatic accidents versus open ocean.

Watson filters out images of open ocean.

But when Watson spots an image that might be an emergency, it sends it to a simple dashboard built with Loopback.

Humans have the final say about what coordinates are likely to have an emergency. Occasionally false positives appear on the dashboard, which the operator can easily spot. A human operator sees any photos flagged as potentially having anomalies, looks at them, and can click them.

When the user flags a photo as an emergency, the dashboard forwards the image to Watson as a hit, improving the classifier. The user's click also sends coordinates to the drone swarm to converge on that location.

Challenges we ran into

Interfacing between Unity and the 'outside world' via HTTP is a challenge. All the important work that our application does is asynchronous, so we needed to carefully coordinate everything. For instance, we don't want a drone to advance before it's photo had been successfully created and sent. Since the simulation only renders what is on-screen at any given time, we had to make sure that only one camera was active at each moment a photo was taken.

Accomplishments that we're proud of

Our custom classifier accurately detects an overturned boat in the open ocean. If we could keep training it (limitation of the free API), we imagine it could become very accurate.

What we learned

Watson's API was very easy to use and we were impressed at how few photos (< 40) we had to feed it before we received accurate image recognition. Neither one of us had ever used Unity's HTTP support, so we were thrilled when the feed from the drones hit our server successfully.

What's next for Watson Search and Rescue

  • Upgrade API to allow for model retraining from operator dashboard
  • Add more anomalies like trash or sea life to see how well Watson can abstract an "emergency" situation from normal ocean anomalies

  • Simulation Repo

  • Operator Dashboard Repo

Share this project: