Drishti AI

• 

• 

• 

Inspiration

1.3 billion people worldwide live with visual impairment. Current assistive technology is either expensive specialized hardware costing thousands of dollars, or simple AI chatbots that describe every object in every frame. Neither works for real-time navigation.

I asked: what would a skilled human guide actually do? They wouldn't narrate every object in the room. They'd say "Door ahead, 2 steps" and then stay silent until the next thing that matters. The silence IS the safety signal — if your guide is quiet, the path is clear.

That insight — that the hardest problem is knowing when NOT to speak — became the foundation of Drishti.

What it does

Drishti (दृष्टि — Sanskrit for "Vision") transforms any smartphone into a real-time spatial intelligence companion. Mounted on a chest lanyard, it:

Sees through the camera using Gemini 2.5 Flash for scene understanding and Cloud Vision for emergency vehicle detection

Thinks through a cognitive architecture with four behavioral dimensions — alertness, urgency, spatial confidence, and verbosity — that model how a human guide processes spatial information

Speaks naturally through Gemini Live API with concise alerts like "Dog 5 steps ahead, 10 o'clock. Walk towards 3 o'clock for clear path"

Validates temporally — every perception result is checked against phone sensor data (accelerometer, compass, step count) to determine if the user has already passed the obstacle. Stale warnings are silently dropped.

Converses in Hindi, English, or mixed — "Machli kahan milegi?" triggers a navigation goal that persists until the system finds a matching landmark

Stays silent when the path is safe — matching the 8-alerts-per-5-minutes frequency of a skilled human guide, not the 30-per-5-minutes of a naive system

Beyond blind navigation, the plugin architecture supports stationary camera deployments: baby monitoring, elderly care, and security — each with different implicit goals and behavioral tuning.

How we built it

The Conductor Model — three Gemini instances work in concert, orchestrated by a Python World Model:

Gemini Live API is the Voice — bidirectional audio with interruption handling (barge-in), affective dialog that adapts emotional tone, proactive audio filtering for ambient noise, and multilingual support. The user speaks naturally; Gemini responds naturally.

Gemini 2.5 Flash (generateContent with \( \text{thinking_budget} = 0 \)) is the Eyes — event-driven scene analysis averaging 2.4 seconds. Returns structured JSON where each object is tagged navigation_relevant: true/false. This single field solved the attention budgeting problem: in a crowded market, 20 people are "not relevant" but one person pushing a cart is.

Cloud Vision API is the Safety Tripwire — at 200ms latency, it detects vehicles in the walking path before the 2.4-second cognitive analysis returns.

The World Model (Python on Cloud Run) is the conductor. It doesn't see or speak — it decides. It maintains:

• Four behavioral dimensions inspired by cognitive science: alertness \( \alpha \), urgency \( u \), spatial confidence \( c \), and verbosity \( v \) — each computed from environment, sensor state, and temporal validation
• A 9-priority editorial decision engine — from P1 (vehicle emergency) through P9 (memory augmentation), with temporal validation gating every perception-based priority
• Implicit + explicit goals — stairs automatically activate stair_navigation at priority 0.9; user speech "find metro" creates a persistent goal that matches against every future cognitive result
• Temporal validation using phone sensors as a reference frame

Temporal Validation is the key innovation. Every camera frame is stamped with the sensor state at capture time. When a perception result arrives 2.4 seconds later, the validator computes:

$$\text{remaining} = d_{\text{obstacle}} - (\text{steps_moved} \times 0.7\text{m})$$

If \( \text{remaining} < -0.5\text{m} \): user passed the obstacle → STALE, drop silently

If \( \text{remaining} < 1.5\text{m} \): obstacle imminent → URGENT, bypass cooldown

Otherwise: VALID, alert with corrected distance

This works indoors without GPS — step count from the accelerometer and compass heading are sufficient.

Phone Sensors feed the entire system: accelerometer (step detection at 3.5 m/s² threshold, stair detection from gravity-axis deviation), gyroscope (head turn vs body turn discrimination — head turns don't trigger cognitive, body turns do), compass (heading tracking for temporal validation and turn detection), GPS (speed and position outdoors), all fused into a contextual physical state with transition detection.

The Brain's Self-Correcting Loop:

$$c_{\text{spatial}} \downarrow \implies \alpha_{\text{vigilance}} \uparrow \implies \text{trigger cognitive} \implies c_{\text{spatial}} \uparrow$$

When spatial confidence drops (stale perception, user moved far), vigilance automatically rises, which triggers a new cognitive call, which refreshes confidence. The system knows when it doesn't know.

Challenges we ran into

Gemini Live function calling has ~16% success rate from native audio. We ran 6 controlled experiments sending identical tool declarations. The native audio model mechanically drops function calls. This forced a complete architectural pivot: Live handles voice only, a separate generateContent loop handles perception. In hindsight, this separation made the system better.

Free tier rate limits (20 requests/day for gemini-2.5-flash) exhausted during the first real test session — 89 out of 91 cognitive calls returned 429 RESOURCE_EXHAUSTED. Switched to a billing-enabled API key through Google Cloud CLI, going from 20 RPD to 10,000 RPD.

iOS Safari sensor permissions require DeviceMotionEvent.requestPermission() inside a user gesture handler, before any other async calls. Our sensors silently failed until we restructured the entire startup flow. Both DeviceMotionEvent AND DeviceOrientationEvent need separate permission calls.

Cognitive latency: 7 seconds with default Gemini 2.5 Flash. The model was spending up to 8,192 thinking tokens on scene analysis that doesn't need chain-of-thought reasoning. Setting thinking_budget=0 cut latency to 2.4 seconds — a 66% reduction with no quality loss for perception.

The temporal staleness problem was the deepest challenge. At 1.2 m/s walking speed and 2.4s latency, the user moves 2.9 meters between frame capture and warning. "Door 1 meter ahead" arrived after the user walked through the door. "Houseplant ahead" fired 10 seconds after the user passed it. Solved with sensor-anchored frame stamping and the predict-track-verify loop.

CV false positive cascade — Cloud Vision detecting doors, houseplants, and clothing as "obstacles" because bounding boxes have no spatial understanding. Adding more CV filters created more problems (5 false "Stop!" out of 7 alerts). The real fix was temporal validation — letting bad detections die naturally when the user's movement proves they're stale.

Accomplishments that we're proud of

Rev 16 temporal validation session: 6 stale perceptions silently dropped (including "user turned 105°" and "user moved 4.5m past obstacle"), 1 imminent obstacle correctly urgent (couch at 0.8m), zero false positives

"Coffee table 2 steps ahead. Clear to your left." — distance, direction, alternative path in one natural sentence

"Dog 5 steps ahead, 10 o'clock. Walk towards 3 o'clock for clear path." — clock positions for spatial orientation, step distance for scale, navigation guidance for action

2 full minutes of correct silence during stable sidewalk walking — in v3.1, the same walk produced 30 alerts. Getting from 30 to 8 required more engineering than getting from 0 to 30.

Hindi conversation working naturally: user asks "Machli kahan milegi?" → goal created → Cloud Vision OCR finds "FRESH FISH →" sign → World Model injects stored landmark → Gemini responds with directions in Hindi

The brain panel with live dimension bars — judges can watch alertness spike near stairs, confidence drop when perception goes stale, urgency fill red when an obstacle is 1 step away. The system visibly thinks.

17 revisions from first deploy to final architecture, each tested in production with real walking, each producing structured logs that drove the next improvement.

What we learned

Gemini's visual understanding is better than any custom CV pipeline. Our Python tracker with Kalman Filters and Hungarian Algorithm produced 12 false houseplant alerts. Gemini said "Potted plants line both sides of the path" — accurate, contextual, once. We deleted 400 lines of tracker code.

The system needs to know what it doesn't know. The \( c_{\text{spatial}} \) dimension is the self-correcting mechanism. Without it, the system confidently delivers stale information. With it, uncertainty triggers action.

Silence is the most important output. A skilled guide speaks 8 times in 5 minutes. Every unnecessary word erodes trust. Engineering silence required the entire editorial decision engine, behavioral dimensions, cooldown management, and temporal validation.

Time is the hardest dimension. Not space, not objects, not language — time. The gap between seeing and speaking is where every false positive and every late warning lives. Solving it required borrowing dead reckoning from robotics and adapting it to a phone on a chest lanyard.

Phone sensors are underrated for AI applications. The accelerometer, compass, and step counter have been in every smartphone since 2010, yet almost no AI application uses them for spatial reasoning. Step count × stride length gives you distance traveled indoors, where GPS fails completely.

What's next for Drishti

Stationary camera deployments — the plugin architecture already supports baby monitoring (implicit goal: monitor baby safety), elderly care (detect falls, prolonged stillness), and security (unauthorized entry detection), each with different behavioral dimensions and alert personalities

Topological mapping from cognitive scene transitions — building a spatial map without SLAM by using Gemini's environment classification as graph nodes and user movement as edges. The user's route (room → corridor → stairs → lobby) becomes a navigable map over time.

Maps API integration for navigation beyond the camera's view — when the user asks "where is the metro?" the World Model could query Google Maps with GPS coordinates and inject routing information into Gemini's context

Native mobile app for background operation, always-on monitoring, and deeper sensor access including the iOS pedometer API and barometer for floor-level detection

User calibration — learning individual walking speed, stride length, and verbosity preferences over sessions, stored in the World Model's semantic memory

Built With

• css3
• fastapi
• gemini-2.5-flash
• gemini-live-api
• geolocation-api
• google-cloud-run
• google-cloud-vision-api
• google-genai-sdk
• html5
• javascript
• python
• web-devicemotion-api
• web-deviceorientation-api
• websockets