UnCane-y AI

UnCane: White cane at the feet, 47 ms AI reflexes at the chest, and zero head-bumps ahead.

Comment

UnCane-y AI: See Beyond the Cane

UnCane-y AI: See Beyond the Cane

The white cane tells you what is at your feet. UnCane tells you what is coming, what is above you, and where things are, and it never lets a language model say "path is clear" when the camera disagrees.

OpenCV AI Competition 2026, powered by AWS. Physical AI + Agentic Vision. Team UnCane-y.
Reflex latency (edge, offline) p50 47 ms, p95 53 ms on a laptop CPU
Metric depth from one RGB camera < 5 % median floor error, no LiDAR
Safety model Dual deterministic verifier (Lambda + device) overrides cloud answers
Agentic Vision 5 OpenCV 5 tools the cloud agent calls mid-loop; results change its next step
Cloud API Gateway, Lambda (arm64 Graviton), Bedrock Mantle, DynamoDB, S3, CloudWatch, one-command SAM deploy
Tests 42 offline tests, no camera, models, or AWS account needed

Table of contents

  1. The problem, in one sentence and one bruise
  2. What UnCane-y AI does
  3. Architecture: two hemispheres, one brain stem
  4. OpenCV 5 implementation (the nerdy bit)
  5. Agentic Vision: the camera changes the plan
  6. The anti-hallucination verifier
  7. AWS deployment
  8. Evaluation, including where we fail
  9. Judging criteria map
  10. Technical report
  11. Reproduce it yourself
  12. Challenges (a.k.a. things that broke)
  13. Limitations and responsible use
  14. What's next

1. The problem, in one sentence and one bruise

Over 2.2 billion people live with a vision impairment. The white cane is a brilliant, 100-year-old, zero-battery piece of engineering. It is also physically incapable of noticing:

  • Head-height things: low branches, truck mirrors, signs, open cupboard doors. (The cane sweeps the floor. The forehead does not get a vote.)
  • Fast, silent things: e-scooters, bicycles, electric cars.
  • "Where exactly is the door handle?": the last yard that GPS politely ignores.

Meanwhile, vision-language models (VLMs) describe scenes beautifully but take seconds to answer and occasionally announce "the path is clear" with the confidence of a golden retriever walking into a glass door.

A cloud round-trip is far too slow to stop a collision, and a model that can hallucinate cannot be the last line of defence. So we stopped asking one system to do both jobs.

2. What UnCane-y AI does

Think of a human nervous system. Touch a hot stove and your spinal cord yanks your hand away in milliseconds. Your cortex only finds out afterwards and has opinions. We built exactly that.

Hemisphere 1: Reflexes Hemisphere 2: Reasoning
Lives On the wearable In AWS
Network Not needed Needed, degrades gracefully
Speed p50 47 ms per frame A few seconds (honest numbers in section 8)
Job Depth, 3D corridor, time-to-contact, object names, spatial beeps, "Stop." "Where is the red water bottle?", "What does that sign say?", memory of places
Mood Calm, instant, a bit blunt Thoughtful, chatty, occasionally wrong
Trusted on safety? Yes Never

Four things you can hear:

  1. "Stop. Chair ahead." The edge reflex, no internet involved.
  2. "Bicycle coming, right!" Ego-motion-compensated looming detection.
  3. "Turn 30 degrees right, walk three steps." A cloud answer converted into body-centric directions by deterministic geometry, not by vibes.
  4. "Connection lost. Obstacle alerts remain active." Graceful degradation, spoken out loud.

3. Architecture: two hemispheres, one brain stem

System architecture diagram

The edge never waits for the cloud. The cloud never gets the last word.

4. OpenCV 5 implementation

OpenCV 5 is not decoration here; it is the edge tier and the toolbelt of the cloud agent.

4.1 Monocular metric depth with the new DNN engine

MiDaS v2.1 small (ONNX) is run through cv2.dnn with ENGINE_NEW. The new graph engine accepts arbitrary multiple-of-32 input sizes, so we run at 128 px instead of the fixed 256 px, which is where much of the 47 ms comes from.

4.2 Turning "relative depth" into meters, with no sensor

Depth networks output unitless inverse depth. We fit the floor plane in real time with trimmed least squares, so that inverse depth becomes a linear function of image row:

$$ \frac{1}{Z} \approx a \cdot r + b $$

where \(r\) is the pixel row, and \(a, b\) are fitted on trusted floor pixels. Once the floor is known, every pixel can be back-projected into a 3D body-space corridor split into ground, torso and head zones. Result: under 5 % median error on floor pixels (synthetic scenes), with zero extra hardware.

4.3 Looming without false alarms

Walking forward makes the entire world appear to expand. A naive looming detector would scream constantly. We track pyramidal Lucas-Kanade features, estimate the camera's own motion with a RANSAC similarity transform, subtract it, and what remains is real approach. Time-to-contact comes from the expansion rate:

$$ \text{TTC} \approx \frac{1}{\dot{s}/s} $$

where \(s\) is the object's apparent scale. Slow approaches are deliberately ignored.

Scenario True TTC Detected TTC
Fast approach A 1.13 s 1.18 s
Fast approach B 0.68 s 0.68 s
Slow 4 s approach ignored (by design) ignored
Camera panning / walking no threat 0 false looming alerts

4.4 Names and spatial audio

YOLOX-S (2 Hz) puts names on things: "chair", not "obstacle-shaped feeling". Audio is stereo-panned: left obstacle, left ear. Pitch encodes height, beep rate encodes proximity. Speech is rate-limited and interruptible, because a device that talks constantly gets switched off by day three.

5. Agentic Vision

The rule says a chatbot that merely narrates a fixed vision result does not count: the visual evidence must change what happens next. Ours does:

Tool (OpenCV 5) What it does How the result changes the next step
assess_image_quality Laplacian-variance blur, brightness Blurry? Ask for a better view instead of guessing
find_text_regions Locates text candidates Decides where to zoom
zoom_region Crops and upsamples a region Re-reads a sign the first look couldn't
analyze_region_color Colour check Confirms "the red bottle" is actually red
measure_region_depth Reads edge depth at a box Converts "over there" into meters and steps

Every call is traced, with arguments, outputs and outcome (see eval/agent_traces.json and eval/live_traces.json). Human control: the user can interrupt, repeat, mute or switch to quiet mode at any time; the agent returns needs_better_view rather than inventing an answer.

Agentic tool trace

6. The anti-hallucination verifier

A deterministic SafetyVerifier compares every cloud answer to the device's own depth corridor. Twice: once in Lambda, again on the device right before speech.

The verifier overriding a VLM claim

VLM:        "The path ahead looks clear."
Edge depth: obstacle at 0.9 m, torso zone
Verifier:   VETO
Spoken:     "Stop. Obstacle directly ahead."

Why twice? Because the network between the two checks is also a place where things go wrong. Paranoia is a feature.

7. AWS deployment

AWS cloud architecture

Service Role
API Gateway (REST) API key + usage plan: 5 req/s steady, burst 10, 5,000 req/day
Lambda (Python 3.13, arm64 Graviton) Agent loop, OpenCV tools, verifier
Amazon Bedrock Mantle qwen.qwen3-vl-235b-a22b-instruct (image + tool calling)
DynamoDB Multi-turn session memory, 12 h TTL
S3 Encrypted telemetry, 30-day lifecycle
CloudWatch EMF latency / safety-override metrics, dashboard, alarms
AWS SAM The whole stack deploys with one command (sam build && sam deploy)

Cost: roughly $0.001-$0.002 per question, and low single-digit dollars per month idle (estimates; check current Bedrock pricing).

Honesty box. We use Lambda on arm64 Graviton, but we are not claiming the COOL award: COOL is not the engine of our core workload, and we would rather lose a prize than claim one we cannot evidence. Our paths are Agentic Vision and the overall competition.

8. Evaluation, including where we fail

Metric Result Reproduce
Edge reflex loop (depth + corridor + flow) p50 47 ms, p95 53 ms, OpenCV 5.0.0, 128 px, laptop CPU pytest -m latency -s
Time-to-contact 1.13 s true / 1.18 s measured; 0.68 s / 0.68 s tests/test_optical_flow.py
False looming under walking / panning 0 in synthetic and real walking tests tests/test_optical_flow.py
Floor calibration < 5 % median error (synthetic) tests/test_geometry_hazards.py
Detector (YOLOX-S, 416 px) mAP@[.5:.95] 0.349, [email protected] 0.467 on 60-image COCO navigation subset eval/eval_detector_coco.py
Verifier "Path is clear" overridden when edge sees an obstacle (Lambda + device) tests/test_verifier.py, scripts/smoke_test_api.py
Test suite 42 offline tests pytest
Cloud round-trip ~1.8-3.5 s VLM inference, ~3.5-5.5 s end-to-end CloudWatch dashboard

Failure cases and limitations (we list them so you do not have to find them)

  • Cloud latency misses our 1.5 s target. A 235B model is not quick. This is exactly why safety lives on the edge: a slow cloud is an inconvenience, never a danger.
  • Depth calibration was validated mostly on synthetic scenes and a small number of real walks. It is not a field trial.
  • Detector accuracy is modest (mAP 0.349 on a small subset); it names things, it is not the safety layer. The depth corridor is.
  • VizWiz-style benchmark of the cloud VLM not completed in time; we do not quote an accuracy number we did not measure.
  • Not clinically validated, not a medical device, not a replacement for a cane or guide dog. Blind users and O&M specialists must lead real validation.
  • Blurry, badly framed photos are the main failure for the VLM (around 30 % of blind-user photos are unanswerable in VizWiz), so we return needs_better_view instead of bluffing.

9. Judging criteria map

Criterion (weight) Where we answer it
Technical execution (30 %) Sections 4, 6, 8: OpenCV 5 ENGINE_NEW depth, floor-plane metric calibration, RANSAC ego-motion TTC, deterministic grounding, 42 tests
Innovation (20 %) Split-intelligence with a zero-trust verifier; sensorless metric 3D corridor; agent tools whose output steers the plan
Real-world impact (20 %) Section 1: head-height and fast silent hazards that canes miss, for 2.2 B people with vision impairment
User experience (10 %) Hands-free, ears-open audio; rate-limited, interruptible speech; quiet mode; offline controls
Documentation and presentation (10 %) This document, architecture diagrams, runbooks in docs/, a video under five minutes
Cloud delivery, reproducibility, responsible operation (10 %) One-command SAM, pinned dependencies, throttled API, encrypted S3, CloudWatch alarms, privacy-by-default, honest limitations

10. Technical report

Problem. Mobility aids miss head-height and fast-approaching hazards; VLM assistants are slow and can hallucinate safety claims. Users. Blind and low-vision pedestrians; O&M specialists; rehabilitation agencies. Architecture. Section 3 diagram. Edge (OpenCV 5, offline) + cloud (AWS serverless) + verifier at both ends. OpenCV 5 implementation. cv2.dnn with ENGINE_NEW for MiDaS and YOLOX; Lucas-Kanade pyramidal flow; RANSAC affine/similarity estimation; Laplacian blur metrics; cropping, colour and text-region tools for the agent. AWS deployment. Section 7; template.yaml (SAM), region us-east-1; deployment log in docs/AWS_DEPLOYMENT_LOG.md. Evaluation. Section 8, with commands to reproduce every figure. Limitations. Section 13. Responsible use. Section 13.

11. Reproduce it yourself

git clone https://github.com/yadavmanan/opencv-uncane.git
cd opencv-uncane/uncane
python -m venv .venv && source .venv/bin/activate   # Windows: .venv\Scripts\Activate.ps1
pip install -r requirements-dev.txt && pip install -e .
python scripts/download_models.py                   # MiDaS + YOLOX, ~100 MB
pytest                                              # 42 tests, no AWS or camera needed

Reflexes only (no cloud):

python -m uncane.edge.app --set cloud.enabled=false

Both tiers locally with a mock VLM:

VLM_BACKEND=mock python -m uncane.cloud.local_server
UNCANE_API_URL=http://127.0.0.1:8787 UNCANE_API_KEY=local python -m uncane.edge.app

Deploy to AWS: sam build && sam deploy --guided (details in docs/AWS_SETUP.md). Pinned dependencies are in uncane/requirements-lock.txt.

12. Challenges (a.k.a. things that broke)

  1. Bedrock said "no" to Claude (HTTP 403). Account-level access, not a bug. We built a model factory (MantleOpenAIAgent), tested 60 models on the endpoint with the same photo, and picked Qwen3-VL 235B: the strongest one that could see and call tools.
  2. Windows text-to-speech went silent after one sentence. pyttsx3 loop-lock. We now run native SAPI in an isolated process per phrase. Yes, it is as glamorous as it sounds.
  3. CloudFormation HTTP 409 on usage-plan attachment. We scripted detach and re-attach.
  4. "Why is the whole world approaching me?" Walking causes global flow expansion. RANSAC ego-motion subtraction fixed it.
  5. Depth with no units. Floor-plane regression gave us meters without buying a LiDAR.

13. Limitations and responsible use

  • Safety-critical context. UnCane is an aid, not a replacement for a cane, guide dog, or O&M training. It can miss hazards.
  • Privacy by default. Event summaries stay in memory; camera snapshots are off unless the user opts in. Bystander faces are not stored. Cloud frames are sent only on user request and expire (12 h session TTL, 30-day telemetry lifecycle, encrypted S3).
  • Security. API key + usage plan, throttling, least-privilege IAM, no secrets in the repo.
  • Fail-safe, not fail-silent. Cloud down, camera stalled, or model refusing: the user is told, and edge alerts continue.
  • Human control. Interrupt, repeat, quiet mode, speech rate and environment mode are all user-controlled and work offline.
  • Fairness and validation. Benchmarks are small and partly synthetic. Real validation must include blind and low-vision users, varied lighting, weather and environments, before any deployment claim.

14. What's next

  • Active-perception voice guidance: "Tilt the camera up a little" when the frame is bad.
  • Negative obstacles: curbs, drop-offs and stairs down, from floor-plane discontinuities.
  • Dedicated hardware: Jetson Orin Nano with CUDA-accelerated OpenCV 5.
  • Graviton fleet analytics: replay recorded walks to mine false alarms and tune thresholds (and, later, evaluate COOL against a baseline).
  • Transit integration: live GTFS-realtime arrivals and crowdsourced hazard maps.

The one-line summary for the judges

We gave a wearable the reflexes of a spinal cord, the brain of a 235-billion-parameter cortex, and the healthy distrust of a toddler who has been told the floor is not lava.

Thank you for reading, judging, and for not walking into any chairs while doing so.

Share this project:

Updates

Submission history