Inspiration

At noon on Saturday, when DurHack officially began, we asked each other one question: what are you most afraid of losing? All three of us gave the same answer: our sight.

We tried to picture it properly. You wake up and the room is dark and stays dark. You can't find the door or see the chair someone left in the hallway, and you don't know if the corridor ahead is empty or full of people. Everything we do without thinking, like walking to the kitchen, crossing a street or finding a seat, would suddenly depend on a cane, a guide dog or another person.

Around 256 million people around the world live with severe vision impairment, and they live with that fear every day. We wanted to see how much of the gap we could close in one weekend.

So we started with a simple premise: what if your headphones could see for you?

Everyone already carries a camera, and computers have become very good at understanding images. Depth models can tell how far away every pixel is, and vision-language models can describe a scene the way a friend would. The hard part is doing it fast enough to trust with your body. A warning that arrives a second late is about a world that has already changed. That pushed us into the details: a depth pipeline that never leaves the GPU, a local hazard tier that never waits on the network, and a cloud tier that only speaks while what it says is still true.

We weren't trying to build a demo you watch. We wanted one you could walk through blindfolded and trust. If our biggest fear ever came true, this is the tool we'd want to have.

What it does

Sight turns a phone camera and a pair of headphones into a voice that walks you around obstacles in real time.

You wear the phone on your chest. It films the way ahead, and a voice in your ear tells you what to do: "Keep going straight." "Move a little left." "Stop. Turn right."

  • Hazard alerts: if something is in your walking path, you hear "Caution" and then "Stop", followed by what is in the way.
  • Directions, not just warnings: a white cane only feels what is within about a metre. Sight looks several metres ahead and steers you only through gaps wide enough for a person.
  • Scene understanding: Gemini looks at the picture and the distances and gives richer guidance, like naming a doorway or warning about a road crossing that depth alone can't see.

A web dashboard for judges and debugging shows the camera feed, a live depth map with every object outlined and labelled, the walking line drawn on the floor, everything the system said or chose not to say, and live latency figures for each stage.

How we built it

The system is a pipeline with two speeds:

  1. Camera: a phone browser page streams frames over a WebSocket (through a Cloudflare tunnel) at up to 30 fps. When the connection is slow, it switches from JPEG to H.264 by itself. Speech comes back over the same socket.
  2. Perception (local GPU): every frame goes through Depth Anything 3 Metric-Large, which gives the distance in metres of every pixel. We reduce that depth map to a 3 × 15 grid of zone distances, removing the floor with a ground-plane fit that is updated on every frame. The frame is uploaded once, and only 90 numbers come back from the GPU.
  3. Hazard tier (local): simple, fast rules on those distances decide "Caution" and "Stop". The alert clips are generated once with ElevenLabs at startup and played from memory, so a hazard never waits for the network.
  4. Guidance tier (cloud): when the scene changes, a small frame and the zone distances go to Gemini 3.5 Flash-Lite with thinking turned off. It streams back a short instruction, which we pass to ElevenLabs Flash v2.5 as soon as the first phrase is complete.
  5. Speech arbiter: hazards interrupt guidance, stale advice is dropped, and nothing is repeated needlessly.

For the labelled view we also run a local labeller: RT-DETRv2 detects objects, its boxes prompt SAM 2.1 for exact shapes, and a DINOv2 segmentation head finds walls, doors and stairs.

Engineering: Python 3.13 with PyTorch, prebuilt fastforks CUDA kernels for the depth network, nvJPEG to encode the previews on the GPU, asyncio for networking, and a React dashboard over WebSockets. The code follows a strict hexagonal architecture: every external technology sits behind a port, and every port has a fake. The whole system runs on a laptop with no GPU, camera or API keys, and swapping in the real hardware is a configuration change. Our three team members each owned one vertical slice (perception, guidance, and alerts/audio/UI), connected by contracts we froze at the start. More than 330 tests, including architecture tests, enforce the rules.

Challenges we ran into

  • Latency is everything. A late instruction is worse than none. We set research-based bounds (hazard alerts under 300 ms, under what human reaction-time studies allow for obstacle avoidance) and measured every stage with p50/p95/p99 figures.
  • The cloud was slower than advertised. Gemini answered quickly on images it had already seen and much more slowly on new ones. Our first benchmark looked great only because it reused the same picture. On real walks, response times ranged from 0.6 s to over 10 s. Instead of a fixed deadline, guidance now speaks only while the scene still matches the frame it was asked about, and local steering fills the gap.
  • Monocular depth lies at the edges. Single noisy pixels could trigger "Stop". We take the 5th percentile per zone instead of the minimum, and remove the floor so the ground itself doesn't count as an obstacle.
  • Getting distances right. With the wrong camera field of view, every distance read 20–25 % short. Calibrating it against a tape measure brought the error within about 12 %.
  • Two voices arguing. Local steering and Gemini sometimes contradicted each other. We had to design clear rules for who speaks when.
  • Phone networking. Streaming video from a phone through a tunnel on a shared hackathon network meant building our own flow control so the stream degraded gracefully instead of falling behind.

Accomplishments that we're proud of

  • Hazard alerts in about 16 ms from frame arrival to audio (p50; 22 ms p99), almost 20 times inside our 300 ms bound.
  • Depth on every frame in 15 ms (p99 16.3 ms) with a 0.35-billion-parameter metric depth model on a laptop GPU.
  • A system you can actually walk with your eyes closed, guided only by voice.

What we learned

  • Measure before you optimise, and be suspicious of good numbers. Our most flattering benchmark was flattering because of a mistake in how we tested.
  • Split work by how fast it must be. Anything safety-critical has to be local and simple. The cloud is great for richness, not for reflexes.
  • Never queue frames. In real-time systems the newest frame wins. Working on old input just means being confidently wrong.
  • Calibration matters as much as the model. A state-of-the-art depth network is only as good as the camera parameters you give it.
  • Contracts and fakes make a team fast. Freezing the interfaces first let all three of us build in parallel from the first hour.

Built With

Share this project:

Updates

Submission history