FlyFireFighter
A rescue drone with a fly’s reflexes and a firefighter in command.
Inspiration
On September 27, 2026, a house fire on 13th Street in Troy killed two parents and their 11-year-old daughter. Their teenage son escaped through a second-floor window with help from people at the scene. The fire happened just blocks from RPI. It left us with a question we could build around at HackRPI: Could a small drone give firefighters a better picture of what’s inside while they’re deciding how to get someone out? WAMC’s reporting on the Troy fire
What it does
FlyFireFighter puts you in command of a simulated rescue. A report comes in that someone may be trapped upstairs. You send a drone to investigate and follow its camera, simulated thermal feed, and observations from a 3D mission console.
When the drone proposes breaking a window or spraying water, you see the target, the evidence behind the request, and the area it would affect. You decide whether to approve it. The drone carries a finite water supply, and opening a window changes how smoke moves. After suppression, it has to look again before guiding the occupant toward an exit.
The mission ends with a concrete outcome: Did the occupant reach the outdoor assembly point? An independent evaluator checks where they ended up. The replay shows what the drone saw, what the commander approved, and what happened afterward.
How we built it
- Flight inspired by a fruit fly. We feed the simulated drone camera into FlyVis, a model built from the measured wiring of the fruit fly’s visual system. We turn signals from its motion-sensitive T4 and T5 neurons into braking and steering commands. A conventional optical-flow controller gives us a baseline to compare against. FlyVis research
- A Troy training scene. We built the scene in Blender and rendered it with Three.js. The exterior takes its cues from RPI’s Heffner Alumni House on Peoples Avenue. The interior, fire, and occupant are fictional.
- Approvals enforced in code. The planner selects from actions the simulator can check. Each breach or water discharge needs its own approval, tied to that specific action. If the evidence goes stale or conditions change, the system can block the action even after approval. The local planner runs offline, with optional Gemini and OpenAI planning and MongoDB Atlas storage for completed runs. Ethics takes most priority.
Challenges we ran into
Getting the drone to move was easier than getting it to know when to stop. A view that supported a spray a few seconds ago might no longer tell you who’s in its path. We made the drone gather fresh evidence, and we built an evaluator that can distinguish seeing someone, guiding them, and actually getting them outside.
What we’re proud of
In one recorded Troy simulation, the optical-flow controller with range assistance carried out an approved window breach, used 0.6 liters across two approved sprays, and guided the occupant to the outdoor assembly point with zero recorded violations. The evaluator separately recorded that the drone did not return.
We also tested six controller configurations across 24 held-out obstacle courses. The trained FlyVis controller made more progress along the planned path than optical flow without range assistance, but hit obstacles in 22 of 24 runs versus 10 of 24. Range assistance cut FlyVis collisions to 1 of 24, and course completion remained poor across all configurations. That result gave us a concrete next target: teach the drone to recover after it stops or gets stuck.
What’s next
We want to put the commander console in front of firefighters for a tabletop drill and see which evidence they need before they’d approve an action. On the flight side, the next test is a controlled indoor course with a physical drone, where we can measure how the controller handles poor visibility, tight openings, and recovery after a stall.
Built With
- blender
- flyvis
- free-wili
- google-gemini
- mongodb-atlas
- next.js
- node.js
- openai
- openstreetmap
- python
- pytorch
- react
- three.js
- typescript
- web-serial
- websockets
