FirefightAR

• 

  Architecture of FirefightAR

• 

  Screenshot of camera viewed in Spectacles

• 

  Screenshot of camera viewed in Spectacles

Inspiration

Firefighters face many challenges when entering a burning building. They use the left/right-hand search technique, a brute-force approach in which they crouch and crawl through the rooms, holding each other's legs while searching with their hands. This method can be slow and increases the risk of injury, disorientation, or getting trapped. We wanted to create a solution that enhances firefighters' safety, speed, and efficiency during rescue missions.

What it does

FirefightAR is an augmented reality (AR) app designed to assist firefighters with critical, real-time information about their surroundings during fire rescue operations. The app maps hazards, identifies objects in the environment, and directs firefighters to exit routes, trapped victims, or dangerous areas using an intuitive AR overlay. Firefighters can use the Snap AR's Spectacle to navigate safely and quickly, improving the chances of a successful mission while reducing the risk of injury.

How we built it

We developed FirefightAR using Lens Studio and Spectacles Interaction Kit for the AR interaction. We utilized a pre-trained YOLOv5 model on fire-related datasets to detect heat sources and people. The app processes real-time video feeds and overlays crucial information like room layout and shortest path to the nearest person. Firebase storage enables team collaboration and data synchronization, ensuring all team members share the same situational awareness.

Challenges we ran into

• We had to quickly learn a brand-new technology that had only been out for one week, and we had no experience with AR before. It wasn't easy to find solutions. Thankfully, the Snap AR team has been very helpful and encouraging with their feedback.
• Because of time constraints, we didn't train our model but used pre-trained ones, leading to extra complexity with different versions and deprecated files.

Accomplishments that we're proud of

We are proud to utilize our knowledge and experience for the first responders. The AR interface requires minimal training, making it accessible to firefighters without extensive technological expertise. We are also happy with our product and ability to adapt to cutting-edge technology.

What we learned

Throughout this project, we learned about the intricacies of developing AR applications for real-time, high-stakes environments. We gained insights into the limitations and possibilities of object detection models in low-visibility scenarios and the importance of efficient data processing to ensure real-time performance. We also realized the critical role of user experience design in creating tools that can be effectively used in stressful situations.

What's next for FirefightAR

In the future, we aim to expand FirefightAR by integrating advanced thermal imaging capabilities and improving the object detection model. Eventually, we hope to expand the system's functionality to support other emergency responders, such as paramedics and search-and-rescue teams.

Built With

• ar
• javascript
• lensstudio
• python
• typescript
• yolo