Sightline

One search map for drones, ground teams and the radio. NVIDIA Nemotron on Nebius Token Factory reads the photos and voice reports and proposes where to search next; a person approves.

Comment

Inspiration

A search-and-rescue drone brings back hundreds of photos per flight; a missing person is about 70 px wide in each, and people reviewing by eye found 68 % of them in the published study. Meanwhile the search manager decides where drones and teams go next, from radio calls and a paper map. Drone tools stop at a flagged pixel; mapping tools track clues. None I found keeps one probability map that photos, sweeps and radio reports all update, and that directs both drones and people.

What it does

  • Detects each drone photo as it lands and places it on the map from its own GPS and camera pose.
  • Updates a probability map with every photo, review, searched sector and clue. Searched ground shows how well it was searched; the software never rules an area out.
  • Nemotron VL takes a second look at uncertain detections, from OpenCV re-look images.
  • Proposes the next drone flight, re-task, team assignment or alert, with its reason. A person approves every move, logged with who and when.
  • A radio desk for ground teams: Parakeet speech to text, Nemotron 3 Super with tools, Magpie speech. A clue is logged only after a readback.
  • Conditions from Tavily at run time decide whether drones may fly.

On the live site you are the search manager in a three-minute guided simulation: the AI proposes, you decide, and the end screen compares a manual search with yours (Without Sightline vs with Sightline, every assumption with its source). Open the console runs the same mission hands-on.

Run a real search: #new starts a live mission on the AWS API: drop aerial photos and each is detected with OpenCV 5 cv2.dnn on the server and placed on the map.

Physical AI: drones sense, the agent decides, drones and people act, and each action changes what is sensed next. A team's clue re-tasks the nearest drone; that drone's photo shows the person. The fleet is simulated, flying over real HERIDAL aerial photos through the same photo API a real drone would use.

How I built it

Every language and vision model call goes through Nebius Token Factory, through one OpenAI-compatible client, and every model is an NVIDIA open model:

Job NVIDIA model Runs on
Second look at uncertain detections Nemotron-3.5-Super-VL Token Factory
Briefings; reading conditions pages with quotes Nemotron 3 Nano 30B-A3B Token Factory
Radio turns with tool calls Nemotron 3 Super 120B-A12B Token Factory
Speech to text / text to speech Parakeet TDT 0.6B v2 / Magpie TTS 357M RTX 4080 SUPER (NeMo)
  • Every AI step in the demo is a real Nemotron answer from Nebius Token Factory: the demo mission and all four scenarios were re-recorded on Oct 26 with live calls (34 new calls, 50.1 s in all): three Nemotron 3.5 Super VL second looks per mission and one Nemotron 3 Nano briefing per assignment, replayed word for word, each labelled on screen with its model, "Nebius Token Factory" and its time; the Trace view counts them per mission.
  • The VL verdict takes 1.35 s p50, about $0.00033 per detection, and 461 of 461 replies were valid JSON. A radio turn's first sound comes back in 2.5 s p50.
  • A Nebius Serverless Job detected the simulated fleet's 711 photos with OpenCV 5 cv2.dnn in 483 s on 16 vCPUs, about $0.10; a Serverless Endpoint serves the same detector one photo at a time (5.68 s p50, $0.38 per 1,000 photos).
  • Tavily searches and extracts the forecast, warnings and park alerts; Nemotron Nano keeps each value only with a verbatim quote. Nemotron Nano can also call Tavily itself to read a public missing-person bulletin.
  • AWS runs the live API and photo worker; the web console is React, TypeScript and MapLibre.

Challenges I ran into

  • Small people: below 60 px Nemotron VL mostly answers "unclear". I kept that abstention rather than prompt it away.
  • Keeping a voice model honest: with thinking off, Nemotron 3 Super sometimes put its reasoning in the reply, and once invented weather without calling the tool. Both are now enforced in code.
  • Never letting software rule an area out: tests check every proposal, briefing and screen string.

Accomplishments that I'm proud of

  • Five NVIDIA models in one loop, each doing one job, each call traced.
  • 6 of 6 radio clue reports read back and confirmed before they were logged.
  • With Token Factory fully down, a mission pays about 82 s in all, not minutes per call; the approval gate held in 12 of 12 runs.
  • What breaks first, stated up front: the detector misses 46 of 337 test people; the second look adds only 4 people for 15 false alarms; the fleet is simulated.

What I learned

Tool calling with thinking off needs code-side guards for anything safety-relevant, and thinking tokens count against max_tokens, silently. A Serverless Endpoint needs no TLS or auth code on my side, but it has no scale to zero. Per-tool notes: docs/feedback/nebius.md.

What's next for Sightline

A Jetson field kit without internet (roadmap only, not yet run on a Jetson); thermal for night searches; real drone photos straight from the drone platform; a field trial with a search-and-rescue team.

Built With

Share this project:

Updates

Submission history