-
-
navigation app
-
3d map to find shortest path
-
7000 rgb data points
-
Inspiration
When trying to save people during a fire, Firefighters usually haven't been in the buildings they are going into. So if someone is stuck in a specific room they would have a hard time even knowing how to get there! In addition, over 51% of people don't know where fire exits are in buildings that they live in.
What it does
So we created Firesight, a way to navigate uncharted buildings for both firefighters and every day people. Using our app, it uses the phone's camera to detect where in the building you are based on what the camera sees, and guides you to a safe exit or a specific room in the building.
Perfect for firefighters looking for a specific room, or for individuals in a rush to exit the buidling in the safest and fastest way!
How we built it
Use MiSensorKit to get a 3d rbg data point cloud of the entire floor of DCC. Used Vultr's GPUs and CPUs to process the 7000+ points of data. Created a react native app to find the best path to a specific room or exit and guide the user using audio and AR arrows. Built a fastapi backend for camera localization, sending camera frames to the backend and returning back the routes. Fish audio generates spoken route instructions.
Challenges we ran into
Getting our reconstruction pipeline running on Vultr a lot of experimenting with dependencies, compute resources, and processing workflows.
Aligning the camera scans, 3D reconstruction, and floor-plan coordinates was another major challenge. Small inconsistencies in the data could locate a user in the wrong room.
Accomplishments that we're proud of
We turned self-created scans of a real campus building into a working indoor navigation guide. Proud of using so many new technologies.
What we learned
We didn't expect this to work in the beginning, and it took us a long time before we made any progress, but progress is exponential and we learned to keep perservering!!
What's next for Firesight
We'll combine our working camera localization and 3D map with the rest of Firesight's features, so the phone, firefighter and commander screens all share one live position and rerouted path from a single server. We also plan to make localization more robust by fusing the camera model with motion sensors and room-number reading, so it keeps working in low light and smoke.
The end goal is to create a wearable device that attaches to the chest so firefighters dont' have to hold up the phone. If we get these scans all over public buildings like malls, colleges, etc, all people and firefighters will have an easier time navigating these buildings during an emergency!
