FlyBy

Building smarter ground control for search and rescue - save seconds, save lives.

Comment 1

Inspiration

Finding someone is not the same as reaching them. After an earthquake, people can be trapped under rubble or hidden behind debris while rescuers race to locate them. A published review cites three earthquakes where 85–95% of survivors rescued from rubble were pulled out within the first 24 hours. Time spent sorting information is time a rescue team cannot get back. Source

Monish led a high-school drone team that worked on search-and-rescue missions. They could cover ground quickly and use ML to surface leads, but the footage, data, and decisions were fragmented. The drone moved fast, but the workflow around it didn’t.

This is personal for us, too. We all come from earthquake-impacted areas of India, where flooding can leave people stranded and difficult to reach. Small drones must navigate cluttered spaces with limited computing power, while responders sort real leads from a flood of detector flags. FlyBy came from connecting those two delays: helping the drone react at reflex speed, assisting rescuers in reaching a decision sooner, and keeping the final call with humans.

What it does

A sighting only matters if someone can act on it. FlyBy combines a fly-inspired visual reflex with AI ground control to move a possible survivor from a drone’s camera to a responder’s decision.

In our simulated coastal disaster, a drone searches the area and flags possible people alongside false leads like debris and animals. FlyBy turns those flags into a ranked queue. It weighs each sighting against incoming reports, known hazards, and the latest incident picture, then suggests what to do next: take another look, inspect up close, dispatch a crew, or set the lead aside. Uncertain possible survivors stay visible for human review. New information can change the ranking. Responders can review or override every recommendation, and no crew is dispatched without human approval. The system also prepares a brief so the crew has the location and relevant context in one place.

For the close-up search, FlyBy uses a fly-inspired visual reflex that reads camera motion and tells the drone when to brake or turn. A carport simulation lets us watch and test those reactions around obstacles. Together, the reflex and ground control are designed to make the journey from sighting to close-up inspection to human-approved rescue decision one connected mission.

How we built it

We built FlyBy in two layers:

  • Ground control: A Python and FastAPI simulator runs repeatable searches through a flooded coastal neighborhood, streaming possible sightings to a TypeScript and Three.js dashboard. Our fine-tuned Laya model runs locally to recommend the next action for each lead. Grok, through the xAI API, turns messy field reports into a changing incident picture and drafts crew briefs. MongoDB Atlas logs mission events, while code keeps locations and numbers exact and requires human approval before dispatch.

  • FlyBrain reflex: A separate service passes camera frames through the pretrained flyvis visual model. We fitted our own readout to its motion signals so the simulated drone can brake, turn away, and continue toward its goal. We tested those reactions in repeatable flights through a 3D carport, and adapted the MIT-licensed fly-brain viewer to show recorded activity lighting up mapped neurons.

We fine-tuned Laya’s scoring layer because the original model could give uncertain possible survivors low or no priority. The fine-tuned model keeps those leads visible for human review while helping responders decide what to inspect next. Together, these layers are designed to carry a lead through one continuous mission, from first sighting to close-up inspection and a human-approved rescue decision.

Challenges we ran into

  • Defining what makes FlyBy different - Rescue drones with thermal cameras, AI detection, and obstacle avoidance already exist. Our challenge was figuring out how our fly-inspired reflex and AI ground control could help within the same rescue mission. That meant designing the handoff from a flagged sighting to close-up inspection, then returning useful evidence for a human decision.

  • Making the reflex work beyond a clean demo - The pretrained fly model gives us visual motion signals, not a ready-made collision alarm. Our early readout missed hazards and sometimes braked for safe paths, so fitting and testing it against debris, beams, and posts took far longer than expected.

  • Making uncertainty impossible to ignore - Laya needs to help rescuers decide what to inspect next, not quietly dismiss a lead because the evidence is incomplete. The original model sometimes gave possible people low or no priority when it was unsure. Fine-tuning it for drone sightings and incoming field reports took time because we had to make uncertain leads visible for human review while still giving responders a useful order of action.

Accomplishments that we're proud of

  • Building more than a dashboard - Our simulated drone searches a flooded neighborhood and flies back to inspect uncertain leads. New field reports can change the order of the queue, and responders can approve, override, and review each decision.

  • Fine-tuning Laya for rescue triage - The original model could push an uncertain possible survivor to the bottom of the queue or leave it out entirely. We trained its scoring layer for our use case so those leads stay visible for human review, then used the fine-tuned model in ground control.

  • Getting the fly reflex to work in a flight loop - We connected camera frames, fly-vision activity, and brake-and-turn commands, then built repeatable carport flights to test the readout against different obstacles.

  • Making the fly brain visible - Mapping flyvis activity onto tens of thousands of neurons in a 3D viewer was difficult, but it gave us a way to show what the visual system responds to as a scene changes.

What we learned

  • A fly brain does not automatically know what a collision is - flyvis gave us motion activity, but we had to build the part that turns it into a brake or turn. A beam the drone could safely pass under sometimes looked as threatening as one it would hit, while narrow posts could appear too late. We learned that camera view, obstacle shape, and braking distance matter as much as the model.

  • Uncertainty should trigger investigation, not dismissal - The original Laya model could give a possible survivor low priority when the visual evidence was unclear. Fine-tuning helped keep those leads visible, but a model still cannot confirm a person it cannot see clearly. That is why FlyBy combines model recommendations with field reports, another look from the drone, and human review.

  • Autonomy has to fit the reality of a rescue mission - We learned that an onboard reflex is assistance, not permission for an unmonitored drone to search wherever it wants. In the U.S., real operations involve a pilot, visual-line-of-sight and airspace rules, and sometimes emergency authorization. That reinforced why FlyBy keeps dispatch decisions with responders, too.

  • Agentic coding taught us to be precise - This was Rupesh’s first hackathon, and AI agents made it possible to work across a simulator, two models, and a ground-control interface in one weekend. The useful prompts specified what each component should do, how it should connect, and how we would test it. We learned to treat generated code as a starting point to verify, not proof that the system works.

What's next for FlyBy

  • Test the complete mission - We want to evaluate the full workflow, from the first sighting to close-up inspection and a human-approved rescue decision. We’ll compare it with manual review on the same scenarios to see whether FlyBy helps responders act sooner while keeping uncertain possible survivors in view.

  • Test Laya against harder cases - Our fine-tuned model keeps uncertain possible survivors visible, but we need to test it on more varied scenes and field reports. We want to measure missed leads as carefully as response speed, so a faster queue never comes at the cost of overlooking someone.

  • Move the reflex onto a real drone - The carport simulation lets us repeat difficult flights, but real obstacles, lighting, and camera motion will be tougher. We want to improve the reflex and test it on hardware, measuring response time and power use before trusting it in a rescue environment.

  • Build with rescue teams - We hope to test FlyBy with search-and-rescue teams, including teams in monsoon-affected areas of India, and explore compatibility with platforms from companies such as Skydio, DJI, BRINC, and Flyability. The goal is a tool that fits how crews already work and helps them act on a lead sooner.

Built With

Share this project:

Updates

Submission history