Eye in the Sky

Inspiration

When a person is missing in the woods, stranded by a flood, or running from the police, the people on the ground have a drone and no time. A DJI Mini 2 can see a lot from thirty metres up, but somebody still has to watch the feed, work out where the person is on a map, decide whether the crew can get there, and say it out loud to the team. Overwatch is the copilot that does that work next to the pilot: it finds people in the video, puts a beacon on them, works out a safe way in, and says every decision with its evidence, so a human can confirm it in one look.

What it does

Overwatch is a first-responder aerial copilot with six situations you can launch from one page:

Pursuit and city chase. The drone launches from the station pad, searches for the person the operator described ("red shirt, jeans"), locks on by clothing colour, follows, loses them behind carports or downtown towers, predicts along the street graph, and reacquires only when the match is verified. You can drive the runner from your phone through a Cloudflare tunnel shown as a QR code, and a police unit is routed to intercept.
Search and rescue. On a 1.5 km forest with 15,920 trees, the drone flies lawnmower sweeps over 500 m sectors with a Bayesian probability-of-area prior. A sweep lowers the probability only where the camera actually looked, so "no detection" is never "clear".
Rescue sweeps (forest, flood, wildfire). Every person the detector sees gets a numbered beacon. The moment a beacon is placed, an A* planner assesses the crew's way in over the hazard grid: around standing water deeper than 15 cm, around burning cells and their arrival-time margin, through the one break in a ring of fire. The route is drawn in the 3D world immediately, the drone keeps sweeping, and the crew walks to the beacons the operator confirms, one after another.
Callouts. Short spoken phrases, newest first: "Found. Beacon 2.", "Lost.", "Target 46 metres north-east of you, your 2 o'clock." Positions are relative to the responder, not the drone.
Responder phone. A PWA that joins the incident, hears the guidance (AirPods through system audio), acknowledges alerts, and replays anything missed on reconnect.
Evidence everywhere. Every detection, decision and callout is logged with provenance (live, replay, simulated or manual), stored in SQLite and mirrored to Tiger Data, and replayable to the same hash.

How we built it

Perception. YOLO26n from Ultralytics on a 1080p feed, a multi-object tracker with a tracker epoch so stale IDs are never reused, HSV clothing colour names per track, and ArUco calibration that turns a box bottom into a world position with an uncertainty radius. Every position is image-derived: the simulator's hidden actor positions are never read.
Muse. Meta's muse-spark-1.3 picks the person whose crop matches a clothing description, accepted only at 0.9 confidence or higher, with a deterministic clothing-rules fallback when the key is missing or the API times out. In our test it picked the yellow-jacket hiker and the blue-shirt walker correctly and declined descriptions that did not fit.
Missions. Pursuit, dispatch (coverage legs sized from the camera footprint), rescue (beacons plus A* access routes), SAR search (probability of area), a decisions router that is rules-first with validated OpenAI proposals, and a safety supervisor that clamps every command to the geofence and altitude limits.
Worlds. A synthetic renderer in NumPy and OpenCV draws the suburb, the city towers, the forest, the static flood with dry mounds and the wildfire with charred trees. One map per situation; scenarios are scripts on it.
Serving. FastAPI and uvicorn, WebSockets with sequence numbers for state, 3D frames and incident events, MJPEG streams, an SQLite incident store in WAL mode, and a TimescaleDB mirror on Tiger Data with hypertables and continuous aggregates for events, telemetry and detections.
Frontend. React 19, Vite and Tailwind. A three.js world with a procedural Mini 2 (spinning props, gimbal camera), trees, towers, water and fire layers, beacons with their amber A* routes, the crew's green route and callout lines. A survey map, Recharts metrics, zustand state, a joystick page and the responder PWA. ElevenLabs renders the callouts server-side; the browser falls back to Web Speech.
A second engine. test12 is a self-contained police-dispatch engine (growing search, YOLO plus colour, dead reckoning, a 360° scan until found, police intercept) mounted at /test12 with its own speed control.

Challenges we ran into

Seeing people from altitude on a CPU. The original 960×540 feed at a 960 px YOLO input saw the person in 0 of 8 frames at 30 m. We measured the options: 1080p at 1280 px saw 2 of 8, a full 1920 px pass 6 of 8, a tiled pass 8 of 8 but at four times the cost, which dragged the simulation to a third of real time. We settled on the full-resolution pass at 5 fps so 2x and 3x speeds still run ahead of real time.
A person inside a ring of fire has no route. The A* assessment correctly reported "blocked", which is the right answer and a useless demo. We gave the ring a break on the trailhead side and the planner threads it: 166 m from the crew's position.
The drone that leaves. In a rescue, matching one description meant the drone would see a person, decide they were not the one, and fly on. Rescue sweeps now tag everyone and keep sweeping; pursuit still matches one person, because that is what a chase is.
A 3D drone lying on its side. The procedural Mini 2 was built Y-up while attitude was applied in a Z-up body frame. Rebuilt in the body frame.
Callouts that talk over each other. One shared list, the newest pre-empts, urgent ones interrupt at once, and no feet or degrees.

Accomplishments that we're proud of

Twenty-seven commits in one day across six worlds, thirty-plus scenarios, a 3D viewer, two phone pages, a database mirror and a second engine, with the demo page always one click from any of it.
Positions that come from pixels and calibration, never from the simulator cheating.
Honest labels: everything simulated says SIMULATED, nothing in the software says "safe", and real flight stays on the sticks.
The beacon-to-route moment: a person appears, a beacon lands, and the crew's way in is on the map in the same frame.

What we learned

Measure before you tune. One eight-frame benchmark replaced an hour of guessing about resolution, tiling and altitude.
A planner that says "blocked" is doing its job; the scenario has to give it a way in.
The same vision model behaves differently for a chase and a rescue. Matching one person and tagging everyone are two missions, not one setting.
Voice is a UI: short, newest first, relative to the person listening.

What's next for Overwatch

Fly it on the real Mini 2 with the companion relay, keeping the advisory-only guard until the safety case is written.
Deploy the backend on Vultr with a GPU for YOLO so the search feed runs at full frame rate.
Terrain relief and second looks in the forest sectors, and the responder phone speaking the beacon routes turn by turn.
Thermal and near-IR sources, labelled as processing, never as temperature.