BatVision AI

Unlocking real-time VLM spatial reasoning for the blind using Nebius AI Cloud and NVIDIA Nemotron Nano 2 VL

Comment

Inspiration

It started with a tree branch.

On our way to the office, one of us walked straight into a low, bent branch. Face first. No warning, no sound, just branch. It hurt, it was embarrassing, and then came the uncomfortable thought:

If that happened to us, with working eyes, in daylight, what happens to someone who relies on a white cane?

A white cane is a brilliant, humble tool. It tells you what is at your feet. It does not tell you about the branch at your forehead, the truck mirror at your shoulder, or the e-scooter about to glide silently past you. And the new wave of AI "describe my surroundings" apps are impressive, but they are slow, they need a network, and, worst of all, they can say "the path is clear" with total confidence when it is not.

We are ZeroLatency. (Yes, we know. Keep reading; we address the irony in Challenges.) We decided to build the thing we wished existed: an assistant that protects you in milliseconds, understands the world in sentences, and never lies to you about safety.

What it does

BatVision AI is an active-safety and assistive-navigation co-pilot for blind and low-vision people. It runs on a camera and a pair of headphones, built like a nervous system with two hemispheres:

Hemisphere 1: Reflexes Hemisphere 2: Reasoning
Where it runs On the device, every frame On request; only the model call leaves the machine
Needs internet No Yes (degrades gracefully without it)
Job "Stop. Chair ahead." "Where is the red water bottle?"
Powered by OpenCV 5 DNN, MiDaS depth, YOLOX, optical flow NVIDIA Nemotron on Nebius Token Factory
Trusted with safety? Yes Never

What you actually experience:

  • Spatial beeps that pan toward the obstacle and speed up as you get closer. Your ears do the steering.
  • Head-height awareness. A 3D corridor split into head, torso and ground zones catches the branch, not just the box.
  • Approaching-object alerts. Optical-flow time-to-contact picks out fast, silent threats a cane never touches; slow approaches are ignored so it does not cry wolf.
  • Ask it anything. "What does the sign say?" The cloud agent calls OpenCV tools on the photo (check quality, find text, zoom, re-read), and every call is traced.
  • Offline, always. No network? The reflexes keep protecting you and the device says the cloud is gone.
  • Hands-free controls. Repeat, stop, quiet mode, indoor/outdoor, speech speed, notes, and "where did you last see my phone?"

The part we are proudest of: the lie detector

Every cloud answer is checked by a deterministic SafetyVerifier against the device's own depth sensing, once in the agent loop and again on the device right before speech.

Nemotron:   "The path ahead looks clear."
Edge depth: obstacle at 0.9 m, torso zone
Verifier:   VETO
Spoken:     "Stop. Obstacle directly ahead."

The model is allowed to be clever. It is not allowed to be wrong about whether you are about to hit something.

How we built it

The reflex path (offline, on the device)

  1. Metric depth from a single RGB camera. MiDaS v2.1 small (ONNX) runs through OpenCV 5's cv2.dnn with the new engine, which accepts any multiple-of-32 input, so we run at 128 px instead of the fixed 256 px.
  2. Floor-plane calibration. A trimmed least-squares fit turns relative inverse depth into meters.
  3. A 3D corridor (head / torso / ground) turns depth into "something is 1.2 m ahead at head height."
  4. Looming without false alarms. Pyramidal Lucas-Kanade features, a RANSAC similarity transform to cancel the camera's own motion, and time-to-contact from the residual expansion rate.
  5. Names. YOLOX-S at 2 Hz tells you what the thing is.
  6. Arbiter. Debounces hazards (a hazard must be seen in 2 of the last 3 frames), rate-limits speech, and lets safety messages interrupt everything else.

The reasoning path (NVIDIA Nemotron on Nebius Token Factory)

  • Model: nvidia/Nemotron-3.5-Super-VL, called through Token Factory's OpenAI-compatible API. No model hosting and no server to deploy: the agent loop runs locally and only the model call leaves the machine.
  • A real tool-calling agent: assess_image_quality, find_text_regions, zoom_region, analyze_region_color, measure_region_depth, then final_answer.
  • Answers start with a deterministic instruction computed from geometry ("Turn 15 degrees right, walk four steps"), then the model's description.

Why the split matters: latency is a distance

A person walks at about \(1.4\ \text{m/s}\), so latency becomes distance:

$$ 1.4\ \text{m/s} \times 0.047\ \text{s} \approx 0.07\ \text{m} \qquad \text{vs.} \qquad 1.4\ \text{m/s} \times 2\ \text{s} = 2.8\ \text{m} $$

In the time an on-device reflex fires, you have moved about 7 cm. In the time a cloud model thinks, you have walked 2.8 m, further than our "Stop" threshold of 1.2 m. So safety lives on the device, and the cloud gets the job it is great at: understanding.

Challenges we ran into

1. Our model thought too hard. Nemotron-3.5-Super-VL is a reasoning model. With thinking on, about half of our agent answers ran out of tokens and came back empty. We turned thinking off with chat_template_kwargs (enable_thinking: false). Result: 4 of 4 forced tool calls worked, and latency on those calls dropped from about 3.5 s to about 1.2 s.

2. Team ZeroLatency vs. the cloud. Our end-to-end cloud answer misses our own 1.5 s target (median about 2 s over eight live requests; one demo run took far longer). We are aware of the irony. It is exactly why safety does not wait for the cloud.

3. The verifier kept saving us from the model. When the model said "clear" and the camera disagreed, we wanted the camera to win every time. Getting a verifier that is strict about obstacles but does not nag took real tuning, plus a debounce and a slow-approach filter so a person strolling toward you is not a siren. (Speech itself runs on a priority queue: a warning interrupts everything, and an interrupted answer is replayed afterward.)

Accomplishments that we're proud of

What Result
Edge reflex loop (depth + corridor + flow) p50 47 ms, p95 53 ms on a laptop CPU in our benchmark (OpenCV 5.0.0, 128 px)
Time-to-contact 1.13 s true vs 1.18 s detected; 0.68 s true vs 0.68 s detected; slow 4 s approaches correctly ignored
Floor calibration Under 5% median error on floor pixels (synthetic scenes)
Object detector YOLOX-S: mAP@[.5:.95] 0.349, [email protected] 0.467 on a 60-image COCO navigation subset
Anti-hallucination "Path is clear" overridden when the edge sees an obstacle, in the agent loop and again on the device
Agentic vision OpenCV tools the model calls, with results that change its next step, each call traced
Works offline Cloud down? Reflexes keep running and say so

A few things we are quietly delighted by:

  • A nice demo that only works with a perfect network A system that tells you when the network is gone and keeps protecting you.
  • Our model reading a sign through a zoom-and-re-read loop it chose itself.
  • Tests that run with no camera, no downloaded models and no API key, so anyone can clone and verify.

Honesty box. These numbers come from a laptop and from synthetic or small-subset data. We kept our measurements in the open, including where we miss. BatVision AI is not clinically validated, not a medical device, and not a replacement for a cane or guide dog.

What we learned

  • "Where does this run?" is a safety decision. The most useful question we asked was which part is allowed to be slow? Anything about safety must not be.
  • Latency is a distance. Once you convert milliseconds into meters, product decisions get easy.
  • Verify the AI, do not trust it. A boring deterministic check against a real sensor beats a very smart model guessing.
  • Listen first. We have not yet sat with users. Everything above is a hypothesis until we do, which is why the next step is below.

What's next for BatVision AI

We are doing this full time.

  1. Bengaluru pilots. We are in conversation with NGOs serving blind and low-vision communities in Bengaluru. Plan: 20 to 30 users over about a month, measuring near-misses, alert usefulness, false-alarm rate and answer accuracy.
  2. A wearable. Today the demo runs on a laptop; the pilot target is a Jetson Orin Nano wearable so the reflexes run on a chest-mounted device.
  3. Better language support. English first; Kannada and Hindi next.
  4. Business and funding. A subscription for cloud features plus institutional licences for NGOs and schools, and a raise of approximately $500,000 to run the pilots, build the wearable, validate safety properly, and check the regulatory path in India.
  5. Faster answers. Smaller images, tighter prompts and more Nemotron variants to bring cloud latency down.

Our promise: if the camera can see an obstacle, BatVision will never tell you the path is clear.

Thank you to the Nebius and NVIDIA teams for the models, the infrastructure and the hackathon.

Built with

Nebius Token Factory · NVIDIA Nemotron-3.5-Super-VL · OpenCV 5 · MiDaS · YOLOX · Python

Links

The team: ZeroLatency
Manan Yadav AI/ML Engineer. Orchestrates the system end to end.
Nitin Kumar Soni Data Evangelist. Data modelling and advanced data systems.
Nida Researcher at IISER Mohali, exploring vision models and advanced devices.

Built With

Share this project:

Updates

Submission history