Inspiration
Last September, Hurricane Helene struck our home state of North Carolina, unleashing widespread destruction and storm surges that devastated our communities. Several of our classmates, friends, and family were displaced from their homes, left struggling to access even the most basic necessities like food, clean water, and shelter. Rising waters turned roads and neighborhoods into flooded wastelands, washing away or stranding many residents.
However, we noticed that the areas hardest hit were also the areas most difficult to be served. Isolated towns like Busick, North Carolina struggled to receive timely and affordable relief, with resources centered around urban areas like Asheville. Upon further research, we discovered a key issue: current disaster relief is scarce in rural and low-income areas due to limited infrastructure, prioritization of urban centers, and technological gaps. In many disasters, rescue teams rely on expensive methods including expensive helicopters or boats that are difficult to deploy quickly and at scale, leaving smaller communities underserved when they need it most. This pattern is seen globally as well. For instance, during the 2022 Pakistan floods, rural villages were submerged with no access to rescue boats or helicopters, and poor warning systems left residents unprepared. Relief efforts were concentrated in cities, leaving rural communities to rely on local and informal aid networks, meaning residents of these neighborhoods were left missing for weeks. Limited transportation and governmental resources also played a major factor, as it was difficult for search teams to travel to and receive funding to aid those in need.
Furthermore, for rescue teams, search is dangerous, time-consuming, and expensive for them to embark on. Teams often face harsh conditions like flooding, landslides, or unstable structures, which put their lives at risk. Coordinating efforts across vast, inaccessible areas requires significant logistical planning and resources, including helicopters, boats, and trained personnel. These operations can cost more than $30,000 per day, limiting the frequency and scale of relief efforts.
These challenges illustrate the urgent need for a more affordable, accessible aquatic search-and-rescue solution capable of reaching isolated communities quickly to aid in the rescue process during floods, storm surges, and coastal disasters.
What it does
Kingfisher is a drone-based search and rescue system designed to help locate survivors in aquatic disaster environments such as floods, storm surges, and coastal emergencies. By combining natural-language control with real-time aerial object detection, Kingfisher enables rapid, accessible, and cost-effective search operations, particularly in rural or resource-limited communities where traditional rescue infrastructure may be unavailable.
At its core, Kingfisher allows users to control a drone using simple text commands rather than specialized technical interfaces. Through a cloud-based large language model, operators can issue flexible instructions such as directing the drone to search specific areas or stop when a survivor is detected. This natural-language interface lowers the barrier to operation, enabling volunteers, local responders, or community members to deploy the system without extensive training.
While the drone navigates the environment, an onboard computer vision system powered by a fine-tuned YOLO (You Only Look Once) object detection model continuously analyzes the drone’s video feed. The model has been trained on thousands of search-and-rescue images and can identify humans and related indicators of human activity, including swimmers, floating individuals, boats, and lifesaving equipment. Once a potential survivor is detected, the system processes positional data and alerts the operator, allowing rescue teams to quickly direct assistance to the location.
Kingfisher is designed to operate efficiently during prolonged searches. Unlike human teams that may fatigue after hours of scanning water or debris fields, the automated detection system maintains consistent accuracy across long operations and large areas. In testing with simulated aquatic rescue footage, the model was able to detect people over 60 meters away with approximately 95% accuracy.
Judge & User Guide
- Turn on the DJI Spark drone and connect it to its remote controller.
- Launch the API on the Raspberry Pi and connect it via Wi-Fi to the remote controller.
- Connect the computer to the Raspberry Pi interface via Ethernet and navigate to http://localhost:50001 In the command box, type natural language instructions, like “Find something edible,” or “Tell me how many people you can see”
- Once a command is typed, it is sent to a cloud-based LLM, where it generates drone controls using a custom programming language shown visible in the example.
- The command the drone is executing can be stopped at any time by clicking the stop button.
- The UI shows the real-time drone imagery with bounding boxes drawn with the object detection model showing what the drone understands in its surroundings.
How we built it
Our development began with the DJI Spark drone, which we chose because of its low price, portability, and its camera attached by a 2-axis gimbal. However, the Spark was not designed to communicate directly with a laptop or edge computer, as it does not have a simple computer-based API, instead typically relying on both a remote controller and a smartphone for command and control. This hardware limitation created a key obstacle, as we needed a way for our drone to be able to interact with our AI system without manual phone control.
To solve this, we introduced a Raspberry Pi wired to a computer in order to expose the drones control interface through a network connection. By modifying the remote control to allow the Raspberry Pi to connect to it over a direct Wi-Fi connection, and then hosting a DJI control server on the Raspberry Pi, we converted the smartphone-only interface into a programmable API that the rest of our system, running on both cloud and edge-based computers, can access.
With the drone now programmable, we built on top of TypeFly to enable the drone to respond to natural language commands while recognizing its surroundings. Because TypeFly was not originally designed for complex search-and-rescue scenarios, and was not designed for the DJI Spark, we developed a custom wrapper class that mapped our new Spark API to the system, and retrained the object-detection system over new search-and-rescue data.
Our natural language command system enables intelligent, autonomous drone operation by combining large language models with vision intelligence. The system introduces a custom lightweight language designed specifically for LLMs and drone control: when a user describes a task (for instance, “search the forest area and look for people”), the system sends that description to a cloud-based LLM, which generates a program in our custom programming language that represents the task. The system allows both flexible drone commands to operate quickly, while taking advantage of the LLM’s reasoning capabilities; for instance, the program might ask the LLM about how it should optimally search the forest area or help in determining if images have signs of life in them.
The system additionally facilitates coordination between drones; this means that not only can operators without formal training quickly and efficiently control drones, but they also are able to control a large fleet of drones with a single command.
For human detection, the drone’s onboard camera streamed video through the Raspberry Pi, where our computers processed the footage using a fine-tuned human detection model based on YOLO (You Only Look Once). We trained the object detection model for search-and-rescue scenarios, prioritizing both speed and accuracy so that human targets can be identified in real time. The system has both higher accuracy and reliability in real search-and-rescue scenarios by continuously analyzing drone footage with the model. Unlike human operators, who may miss visual details after long periods of monitoring video feeds, our system can scan every frame of incoming footage without fatigue, ensuring that potential detections are far less likely to be overlooked.
When the model identifies a potential human presence, the system can immediately flag the detection with the location of the drone through the interface, allowing rescue officials to quickly locate the area where presence was flagged if they determine that rescue is necessary, significantly reducing response times and false positives by directing search teams to precise locations instead of requiring them to manually cover large areas of land. The automated AI model allows for predictable, testable performance over long search operations, designed to operate in conjunction with human search teams ensuring there is no opportunity loss from adopting the technology.
To evaluate Kingfisher beyond theoretical testing, we additionally conducted a field trial using a Spark drone in a real forested environment. While we weren’t able to test the system in a water-based scenario due to logistical concerns, we used the system in real-world conditions, including uneven lighting, dense vegetation, and real camera motion. Our tests demonstrated that the model can function outside of testing environments and dataset to provide validation that Kingfisher works in practice and can be applied to real search-and-rescue situations.
Challenges we ran into
One of our primary challenges was hardware integration. The DJI Spark was not designed to communicate directly with a laptop or edge computer, requiring us to engineer a custom solution using a Raspberry Pi to expose the drone's control interface programmatically. Connecting YOLO, our model, to the live drone footage also proved complex, as natural language instructions needed to be accurately parsed and translated into precise, low-latency flight commands. Training a reliable human detection model was another hurdle, as search-and-rescue imagery is scarce and our dataset was limited in diversity.
Accomplishments that we're proud of and what we learned
Through this process, we learned that building accessible technology required constant iteration on both the interface and the underlying control logic. We also gained a deeper appreciation for the gap between controlled testing and real-world deployment, which motivated our field trial in a live environment.
Impact
Current disaster relief is often scarce in rural and low-income areas for several reasons: limited infrastructure and geographical isolation delays the arrival of aid and rescue operations; the prioritization of urban centers lead to under-funding and resourcing in rural areas; and technological gaps lead these areas unable to detect early warning signs or develop robust communication networks.
Furthermore, from the rescue teams’ perspective, it is dangerous, time-consuming, and expensive for them to embark on disaster relief initiatives. Teams often face harsh conditions like flooding, landslides, or unstable structures, which put their lives at risk. Coordinating efforts across vast, inaccessible areas requires significant logistical planning and resources, including helicopters, boats, and trained personnel. These operations can cost more than $30,000 per day, limiting the frequency and scale of relief efforts.
This gap underscores the need for a low-barrier and safe means of disaster relief. Kingfisher addresses this issue by equipping Unmanned Aerial Vehicles (UAVs) with a voice-enabled search and rescue model, designed to work in resource-limited environments. This system uses speech recognition and AI-powered human detection, allowing individuals to help without requiring access to expensive technologies or extensive training. In doing so, Kingfisher empowers underserved communities to build a self-resilience against future crises.
What's next for Kingfisher
Future development will focus on expanding the human detection model with more diverse disaster environment data including floods, landslides, and turbulent water to improve real-world accuracy, as well as adding thermal imaging support for locating survivors at night or beneath debris. Implementing an offline mode would remove dependency on internet connectivity, which is often unavailable in disaster scenarios. Multi-drone coordination is another key direction, enabling a single operator to deploy a fleet of UAVs that autonomously divide and cover search grids. Longer term, piloting the system with NGOs in disaster-prone regions will help gather field data and refine the system based on feedback from real rescue operators.
Log in or sign up for Devpost to join the conversation.