First Flight

Realtime ML to speed up search and rescue operations.

Comment 1

Inspiration

I was inspired to make this project after I learned about the difficult yet crucial job of first responders during natural disasters. First responders must locate victims regardless of treacherous and dangerous terrain, often in bad weather. Moreover, speed is crucial - minutes delaying search and rescue efforts can impact the survival of victims. I decided to build a project that can assist first responders and reduce the time it takes to find victims.

What it does

First Flight uses machine learning to locate people in aerial imagery from autonomous drones. Drones can be pre-programmed to follow a certain flight path and fitted with cameras to record and stream video footage. The footage from the drone doesn’t have to be super high-res, in fact, the lower the resolution, the faster the inference time for detecting people! For this project, I used previously-recorded footage because I didn’t have access to a drone (I actually tried to build one and use a GoPro, but it couldn’t fly – maybe the best hack that didn’t work?!).

After streaming video footage, I used the YoLoV8 object detection algorithm to draw bounding boxes around people, animals, cars, boats, and more in real time. The YoLoV8 model uses convolutional neural networks to locate and detect over 30 classes of objects in a video stream up to 120FPS – that’s less than 50 milliseconds per inference!

After identifying people in the image, the GPS data from the drone is used to pinpoint the location of the victims. This GPS data – in the form of latitude and longitude coordinates – are used to drop pins on a google maps view, which helps first responders locate victims.

How we built it

First Flight is built using the Flask framework. I chose Flask because the machine learning code for object detection uses Python, and Flask provided a simple yet effective way to integrate everything into a web app.

The core of the project revolves around computer vision via OpenCV, which I used to capture and process the video feed. The video stream is then processed by TensorFlow to run the YoLoV8 model, which was trained on the COCO2018 dataset—a collection of 1.2 million labeled images across various categories.

The web application serves as the front-end, allowing first responders to view both the real-time video feed and the Google Maps view, where pins are dropped based on the GPS coordinates tied to detected people. Google Maps integration plays a crucial role in pinpointing the exact location of potential victims, making it easier for rescue teams to respond quickly.

Challenges we ran into

One of the main challenges was dealing with aerial imagery. My first attempts at capturing aerial imagery involved building a plane with an onboard camera. I tore up some cardboard boxes and built a make-shift plane, but the cardboard was inconsistent, weak, and too heavy. My next attempt used a cheap foam glider I purchased for 10 dollars. Although the glider flew perfectly, the GoPro I had lying around was too heavy for it to fly. Crunched on time, I had to rely on pre-recorded aerial footage.

Another big hurdle was learning Flask from scratch. I had no prior experience with the framework, and integrating it with the machine learning code while also serving video streams in real time was definitely a challenge. Optimizing the object detection model for real-time performance was also tricky, especially since I was running everything on a local machine, which has computational limits.

Accomplishments that we're proud of

I’m really proud of successfully training the YoLoV8 model on such a large dataset and getting it to work for real-time object detection. It’s crucial for this application that the model is fast and accurate, and I’m happy with the results. Another big accomplishment was building the Flask app from scratch, even though I had never used Flask before. I managed to get the whole system up and running, from capturing video to displaying detection results on a web app. Finally, integrating Google Maps with the object detection results was a key part of this project, and I’m proud that it worked seamlessly. It really enhances the functionality of the app and makes it easier for first responders to visualize locations.

What we learned

This project taught me a lot about using Flask for web development, especially how to integrate Python-based machine learning models into a live web app. I also learned how to train and deploy object detection algorithms like YoLoV8 using TensorFlow, and gained experience working with large datasets. Additionally, I now have a better understanding of how to process and display real-time video streams, which was a crucial part of this project’s success. Finally, I got a bit of experience scoping out problems and completing tasks even with inadequate materials and resources.

What's next for First Flight

The next step is to integrate an actual drone into the system so I can use real-time, live footage for detection. This will allow me to test the system in real-world conditions, rather than relying on pre-recorded videos. I also want to explore further optimization of the model to improve its performance on lower-power devices like onboard processors or edge devices. This would make it easier to run the entire system directly on the drone itself, rather than relying on external computing power. In the future, I’d also like to add features like automatic flight path generation based on high-risk areas and integrate thermal cameras for detection in low-visibility conditions or during nighttime operations. Finally, I want to test the system in an actual natural disaster or public safety emergency.

Built With

Share this project:

Updates