Inspiration
We started with a question: if a drone camera spots someone who may need help, how does that footage become an incident a dispatcher can act on?
A camera feed can show what happened, but a dispatcher still needs to find the important moment, review the evidence, choose who should respond, and track who received the assignment. We built RescueDrone to connect those steps, especially when several cases need attention at the same time.
What it does
RescueDrone is a local rescue-dispatch simulation. A Samsung camera stands in for Drone 1, and recorded footage represents Drone 2. The dashboard gives each open incident its own row and puts urgent cases first. A dispatcher can open the evidence, view the source’s live feed or replay, and get directions to a location explicitly labeled as a simulated map pin.
The dispatcher can assign the same case to EMS / Ambulance, Fire / Rescue, or Police. Each team has its own queue and records its own acknowledgment. Closing a case removes it from the active queues and preserves its assignments and acknowledgment times in history.
The prototype also includes a fictional patrol roster. Its police-unit selector ranks available units by straight-line distance to the training pin and lets the dispatcher confirm the suggested assignment.
How we built it
| Tool | How we used it |
|---|---|
| Python, OpenCV, and NumPy | Captured video, calculated torso movement from pose keypoints, rendered evidence snapshots, and served the local dispatcher. Python also ranks fictional patrol units by distance. |
| Ultralytics YOLO and ByteTrack | Detected people, estimated body posture, and tracked identities across frames. We also compared ByteTrack with BoT-SORT in recorded-video experiments. |
| HTML, CSS, and JavaScript | Built Active cases, the camera viewer, incident details, diagnostics, and the responder queues. The local viewer can start Drone 1 with one click. |
| SQLite and JSON | Saved incident history, priorities, responder assignments, acknowledgments, and closures. A separate JSON file points to the latest alert. |
| Tiger Data, PostgreSQL, and Psycopg | Synced timestamped camera telemetry from SQLite to the cloud and queried the two sessions used in the overlap analysis. |
| MATLAB, Simulink, and Stateflow | Plotted feed overlap and tracking interruptions, then replayed saved dispatcher decisions in a state chart. |
| Gemini API | Reviewed recorded fall footage and analyzed possible gunshot sounds. Saved audio assessments let the dispatcher review the same verdict alongside the recording. |
| TensorFlow Hub, PyTorch, and Transformers | Ran YAMNet and the Audio Spectrogram Transformer on labeled gunshot, fireworks, and other sharp sounds. |
| TinyFish | Retrieved public agency information from the official City of Tempe page and displayed the extracted services in the dispatcher. |
| Windows PowerShell | Ran the camera and replay tests, checked the APIs, and created a double-click launcher for the demo. |
Challenges we ran into
- Slow movement triggered a fall alert. A controlled-lowering clip met the original rule. We traced the movement and raised the replay threshold from 0.35 to 0.75 torso lengths. The updated replay cleared that false alert and preserved the review events in all six tested UR Fall color clips.
- A single latest-alert slot could hide earlier activity. Samsung’s result appeared in that slot after a replay incident. SQLite preserved the replay in history, and the Active cases table gave each open incident its own row.
- The live viewer went blank when the camera test ended. The original test stopped after 30 seconds. We added a mode that keeps publishing frames and connected the viewer to a one-click camera start.
- Gunshots and fireworks were hard to distinguish. YAMNet scored one fireworks clip higher than any of the labeled gunshot clips in that batch. The second model missed one of two new gunshot clips. Gemini changed its verdict on the same recording. We cached each usable assessment and made uncertain results appear as Needs review · Unverified.
Accomplishments that we're proud of
Best Use of MATLAB
MATLAB measured 4.09 seconds of overlap between the live and replay feeds using a 23-row export from Tiger Data. A separate report showed a 0.50-second tracking gap, from 5.97 to 6.47 seconds into a replay clip.
We also connected MATLAB to the dispatcher API. Simulink and Stateflow used a saved incident to replay Waiting, Flagged, Reviewed, and Assigned. The logged output matched the saved two-team selection of EMS / Ambulance and Fire / Rescue. Both reports are available from the dashboard.
Best Use of Tiny Fish
The dispatcher’s Refresh agency reference button asks TinyFish to read the official Tempe Fire Medical Rescue page and extract the department name and listed services. Python validates the response before the dashboard displays it. The integration returned eight service entries, including advanced emergency medical services, city-run ambulance service, hazardous materials operations, and technical rescue.
[MLH] Best Use of Gemini API
The dispatcher retrieved Gemini’s assessment for a saved fall incident. Gemini flagged a possible fall around 5.33 seconds into the clip. We also verified that a saved uncertain audio assessment becomes a Needs review · Unverified case with the recording and Gemini’s observation available for human review.
[MLH] Best Use of Tiger Data
We verified 205 telemetry samples in Tiger Data across the Samsung and replay sources. SQL let us retrieve the exact sessions we needed and export the timestamps for the MATLAB overlap report.
Best Overall
We demonstrated the local handoff from an urgent incident to a saved closure. The same incident appeared in the EMS and Fire / Rescue queues, both teams recorded separate acknowledgments, and the dispatcher preserved those timestamps in the closed case. The one-click Drone 1 viewer let reviewers open the live feed from the queue.
What we learned
The most important lesson was to separate what the system observed from what a dispatcher decided. A lost track or a sharp sound can give someone a reason to review a clip. The original evidence, timestamp, and uncertainty label help the dispatcher make that decision.
We also learned to test both sides of every change. Fixing the slow-lowering false alert mattered because the updated system still surfaced the recorded fall reviews.
What's next for RescueDrone
The current demo uses one live camera and one recorded replay. Our next tests are two physical cameras running side by side, a fall recording where the same track ID stays visible, and audio collected under realistic drone microphone conditions. We also want to verify the busy-unit selector with two simultaneous simulated police cases. Remote responder access would be a separate deployment with authenticated accounts and an authorized connection to agency dispatch systems.