Sightline

Drone photos and radio reports in, where to search next out. OpenCV 5 finds people in search-and-rescue photos, an agent proposes the next move, and a person approves every one.

Comment

Inspiration

A search-and-rescue drone brings back hundreds of photos per flight, and a missing person is about 70 px wide in a 4000×3000 photo. Volunteers reviewing by eye take about 33 s a photo and, in the published HERIDAL study, found 68 % of the people. Meanwhile the search manager decides where drones and ground teams go next, from radio reports and a paper map. This is not a remote-wilderness problem: North Shore Rescue, twenty minutes from downtown Vancouver, answers more than a hundred calls a year in the mountains above the city, and the difference between finding someone and not is often time. Plenty of tools flag pixels; none I found turns every photo, negative sweep and radio report into one updated probability map, with a logged approval gate between a suggestion and a drone or team.

What it does

  • Detects each photo as it arrives with OpenCV 5 cv2.dnn, and places every detection on the map from the photo's GPS and camera pose.
  • Keeps a probability map of where the person could still be. Every photo, review, searched sector and clue updates it. Searched ground shows how well it was searched ("searched at POD 60 %"); the software never rules an area out.
  • Takes a second look at uncertain detections: OpenCV 5 re-look tools (native-resolution zoom, CLAHE, re-detection) build better images, and NVIDIA Nemotron VL on Nebius Token Factory gives a verdict.
  • Proposes the next move (promote a detection, fly or re-task a drone, assign a sector, alert the teams) with its reason. A person approves every one, logged with who and when.
  • Talks to ground teams on a secondary radio channel and logs a clue only after a readback.
  • Writes the files a command post files: ICS 204 per team, ICS 214 from the trace, CalTopo GeoJSON.

On the live site you are the search manager in a three-minute guided simulation: the AI proposes, you decide, and the end screen compares a manual search with yours (Without Sightline vs with Sightline, every assumption with its source). Open the console runs the same mission hands-on from the call.

Run a real search: #new starts a live mission on the AWS API: drop aerial photos and each is detected with OpenCV 5 cv2.dnn on the server and placed on the map.

How I built it

  • OpenCV 5 sees. YOLO11s fine-tuned on 1024 px HERIDAL tiles, exported to ONNX and run on cv2.dnn with no PyTorch at run time: identical scores to PyTorch on the test set, 363 of 363 boxes matched one to one. Re-look tools, one per failure mode: zoom_native for small people, CLAHE for clothing that matches rock, redetect_region for groups and people lying down; cv2.fillPoly and cv2.blur for coverage and canopy on the map.
  • NVIDIA models read, reason, listen and speak: Nemotron 3.5 Super VL (second look), Nemotron 3 Super (radio turns with tools), Nemotron 3 Nano (briefings, conditions), Parakeet and Magpie (speech).
  • Nebius runs every model call (Token Factory) and the batch detection (a Serverless Job: 711 photos in 483 s, about $0.10). Tavily brings the forecast, warnings and park alerts that gate the drones.
  • Every AI step in the demo is a real Nemotron answer from Nebius Token Factory: the demo mission and all four scenarios were re-recorded on Oct 26 with live calls (34 new calls, 50.1 s in all): three Nemotron 3.5 Super VL second looks per mission and one Nemotron 3 Nano briefing per assignment, replayed word for word, each labelled on screen with its model, "Nebius Token Factory" and its time; the Trace view counts them per mission.
  • AWS: S3, an SQS FIFO queue with a dead-letter queue, an EC2 worker running cv2.dnn, CloudWatch metrics and Secrets Manager; 7.1 s per photo from upload to processed. COOL on Graviton: the detector and API also run on Cloud Optimized OpenCV with bit-identical boxes (538 of 538).
  • The agent proposes and never acts: every tool call, model call, verdict, proposal and decision is one line of the mission's trace.
  • The web console (React, TypeScript, MapLibre) is laid out like a command post: resources, the map, Needs you, and the timeline with the radio.

Challenges I ran into

  • Small people in big photos. Shrinking a photo to the model's input drops test recall to 0.234; tiling at native resolution works. The second look mostly answers "unclear" on people under 60 px, and I kept that.
  • import ultralytics sets cv2.setNumThreads(0), which made cv2.dnn 5× slower in any process that also loaded Torch.
  • Honest evaluation: every threshold chosen on validation, test scored once per decision; two early choices saw test, and the report says so.

Accomplishments that I'm proud of

  • A real chain where OpenCV changes the decision: a person lying down, scored 0.18 (below the 0.29 threshold), zoomed from native pixels, called "yes, lying" by Nemotron VL, and proposed for review.
  • The detector finds 86.4 % of people on the HERIDAL test set (strict VOC; 89.6 % on the study's looser scoring, against 68 % by eye).
  • The whole loop live on AWS, and a technical report where every number links to its evidence.
  • What breaks first, stated up front: the detector misses 46 of 337 test people (groups, rock-coloured clothing, people sitting or lying); the second look adds only 4 people for 15 false alarms; the fleet is simulated over real HERIDAL photos.

What I learned

The detector is no longer the hard part; what happens after a pixel is flagged is. Measuring that honestly gave small gains where I hoped for big ones (the second look; the planner, which beats a fixed order only when there is a clue), and I report them as measured. COOL taught me the same: image processing 1.14 to 1.35× faster, cv2.dnn not faster.

What's next for Sightline

Thermal imagery for night searches; photos arriving straight from the drone platform; a word error rate on real volunteers' voices over real handsets; and a field trial with a search-and-rescue team.

Built With

Share this project:

Updates

Submission history