Inspiration
In an emergency, the people searching a building have to make fast decisions in places they may never have seen. Most training is slideshows or guided walkthroughs that show you what to do but never make you decide. We wanted something that lets you practice the decisions themselves and get honest feedback on how you did, without real-world stakes.
What it does
Firebox puts you inside a simulated building. As you move through it, each room can ask you questions that depend on what kind of room it is. A kitchen, a stairwell, and an office each have their own risks and priorities. Once you've answered everything and made it back to the exit, you get a score based on your answers and your route. Then Firebox shows you the best route through the building next to the one you took, so you can see where you lost time or doubled back.
How we built it
Firebox loads Matterport scans through the Showcase SDK, so users get to walk through real spaces.
Frontend: built with React + Vite. The Matterport SDK returns to us sweep positions, room data, and floor info in real time as you move.
Backend: a small Python server that serves the SDK config, the per-room questions, and the optimal route.
Questions: each checkpoint is anchored to a real sweep and room in the scan. A scenario bank assigns room-type appropriate decisions (a kitchen question in the kitchen, a stairwell question at the stairwell) and scores answers by rank. The best choice earns 3 points, the worst choice earns 0, with partial credit in between.
The Route Planner: serves as the backbone of the application. Matterport gives you sweeps but no notion of doors, exits, or which sweeps connect, so we export the scan's sweep graph straight from the SDK at runtime, hand-mark the exits (since scans have no "door" or "exit" concept), and run an exact route optimizer over the checkpoints that respects walkable adjacency and multiple floors. That's what lets us show users' path next to the optimal one and compute where you lost time.
Exterior view: we pull live aerial imagery (Vexcel, keyed off the property's real coordinates) so you can scout the building from above before going in.
Debrief agent: Instead of a scoreboard, an agent uses three tools and a multi-step process to analyze the user's performance: a scorer that knows exactly which decisions you got wrong and why the better choice was better, a route calculator that compares the path you took to the mathematically optimal path through every checkpoint, and a replay of the user's actual recorded path. It determines which tools would be most relevant to the analysis and calls them accordingly before summarizing its findings.
Challenges we ran into
Buildings don't come with a map. The SDK handed us camera positions but no graph of what connects to what, no doors, and no exits. We had to reconstruct the navigable graph ourselves from raw sweep adjacency, then hand-mark exits, before any routing was possible.
Multi-floor routing broke our planner. Our first optimizer assumed every floor plan was one floor. But when we loaded a two-story building, half of the checkpoints became "unreachable." In order to fix it, we had to find the stair connections in the graph and let the route cross floors.
Unlabeled rooms. Some scans don't name their rooms, which made our questions generic and irrelevant (a "door" scenario dropped into a living room). We built a path to pull the real room labels the SDK already knows so questions match where you're standing.
Accomplishments that we're proud of
Turning real buildings into interactive training environments Using Matterport lets users practice navigating real spaces.
Connecting interior and exterior views Bringing Matterport walkthroughs and Vexcel ariel imagery into one experience.
Rendering obstacles inside the walkthrough Getting custom 3D models to appear in Matterport.
What we learned
Learned to combine Matterport walkthroughs, Vexcel aerial imagery, and interactive scenarios into one training experience.
Learned how graph algorithms apply to physical spaces, using rooms and viewpoints to plan routes and compare training performance.
What's next for Firebox
Expand training scenarios with more buildings, hazards, and difficulty levels.
Add realistic, dynamic obstacles that challenge users to adapt their routes.
Partner with firefighters and instructors to validate scenarios and guide future development.
