Sarthi — Your destination. Your voice. Your way forward.
Inspiration
For someone who cannot rely on vision, navigating the world is not simply a matter of knowing where to go. The difficult part is understanding what is happening around them right now.
A map can tell you that a pharmacy is 700 meters away. It cannot tell you that there are stairs directly ahead, that an obstacle is blocking the walking path, or that the real-world environment no longer matches what was expected.
That led us to a simple idea:
You choose the destination. Sarthi figures out the way.
We wanted to build something closer to a mobility companion than a camera that simply describes objects.
What it does
Sarthi is a voice-first AI mobility companion designed to help visually impaired users navigate dynamic environments.
The interaction starts naturally: the user simply tells Sarthi where they want to go.
For example:
“Take me to the nearest pharmacy.”
Sarthi can then:
- Understand the user's destination.
- Find relevant nearby places.
- Compare available walking routes.
- Start a navigation journey.
- Provide turn-by-turn guidance through voice.
- Continuously observe the surrounding environment through the camera.
- Identify mobility-relevant hazards such as stairs, curbs, potholes and obstacles.
- Maintain a persistent understanding of the surrounding world.
- Decide which information actually requires the user's attention.
- Communicate important information proactively instead of waiting for the user to ask.
- Handle uncertain, stale or unavailable perception explicitly.
- Reconcile the planned route with the physical environment.
This creates a continuous loop:
Intent → Search → Route → Perceive → Reason → Prioritize → Guide → Recheck → Adapt
The map answers:
“Where should I go?”
Sarthi also tries to answer the more immediate question:
“What do I need to know right now?”
Importantly, Sarthi is designed as an assistive system, not a replacement for a cane, guide dog, or the user's own judgment. When perception is uncertain, the system communicates that uncertainty rather than pretending it knows.
Key Features
🎙️ Voice-First Interaction
Voice is the primary interface of Sarthi.
Users can interact naturally without needing to navigate complex visual controls.
- Voice-based destination requests
- Speech recognition
- Spoken navigation instructions
- Proactive hazard warnings
- Text-to-speech guidance
- Multilingual interaction foundations for English, Hindi and Marathi
A user can simply say:
“Which is the nearest pharmacy to me?”
and Sarthi can interpret the request as a navigation task rather than a generic visual question.
📍 Natural-Language Destination Discovery
Users don't need to know the exact name of a location.
Sarthi understands destination-oriented requests such as:
- “Take me to the nearest pharmacy.”
- “Find me a hospital.”
- “Where is the nearest café?”
- “Can you find me a hotel?”
It extracts the destination intent, searches for relevant nearby places, and prepares them for route selection.
🗺️ Route-Aware Walking Navigation
Sarthi does not simply assume that the geographically closest place is the easiest to reach.
Candidate destinations are evaluated using their actual walking routes.
The navigation system includes:
- Place discovery
- Walking-route planning
- Route distance and duration
- Turn-by-turn guidance
- Navigation state
- Route progress
- Current-position tracking
- Route-aware destination selection
This allows Sarthi to distinguish between:
“closest as the crow flies”
and
“closest to reach by walking.”
👁️ Continuous Environmental Perception
The camera provides Sarthi with a live view of the user's surroundings.
Rather than treating every camera frame as an isolated image, Sarthi maintains a structured World State.
The perception layer can track information such as:
- Object/class
- Confidence
- Bounding box
- Relative position
- Approximate distance
- Perception freshness
- Camera movement context
This gives the rest of the system a continuously updated representation of the environment.
⚠️ Mobility-Focused Hazard Detection
Sarthi is not designed to describe everything that appears in front of the camera.
It focuses on information that can affect the user's movement.
The system can reason about situations such as:
- Stairs and steps
- Curbs and step-downs
- Obstacles in the walking path
- Potholes and holes
- Uneven walking surfaces
- Doorways and entrances
- Large blockers
For example:
“Stop. Stairs ahead.”
The objective is not maximum object recognition.
The objective is useful environmental awareness.
🧠 Intelligent Attention Prioritization
A major challenge in assistive AI is deciding when the system should speak.
A camera can detect many objects at once. Announcing all of them would increase cognitive load and potentially make navigation harder.
Sarthi therefore prioritizes information using:
CRITICAL → CAUTION → GUIDANCE → INFORMATION
Important events can interrupt normal navigation guidance, while irrelevant observations remain silent.
The system also incorporates:
- Relevance filtering
- Warning cooldowns
- Duplicate-warning prevention
- Hazard persistence checks
- Route-corridor filtering
- Perception freshness
This allows Sarthi to focus on what matters now rather than everything it can see.
🧭 Route ↔ Environment Reconciliation
One of Sarthi's central ideas is connecting two different sources of truth:
The map knows the planned journey.
The camera observes the physical world.
Sarthi brings these together through:
Planned Route ↔ GPS Position ↔ Observed World
This creates the foundation for identifying situations where the environment and the planned navigation context do not perfectly align.
Instead of treating navigation as:
GPS → instruction → GPS → instruction
Sarthi is designed around a continuous perception-navigation loop.
🛡️ Uncertainty-Aware Safety
Sarthi explicitly distinguishes between absence of evidence and evidence of absence.
For example:
No detection ≠ no obstacle.
A failed perception request does not mean that the path is clear.
The system tracks states such as:
- Detected
- Not detected
- Unknown
- Unavailable
- Stale
When perception becomes unavailable, Sarthi preserves uncertainty instead of generating a false “all clear” conclusion.
Similarly, safety-critical decisions are not delegated entirely to a generative language model.
Deterministic logic handles important safety and prioritization behavior.
☁️ Cloudinary-Powered Visual Pipeline
Cloudinary forms an important part of Sarthi's visual media pipeline.
Camera frames can be delivered through Cloudinary, while the architecture supports targeted AI Vision analysis for mobility-specific questions.
Instead of asking an AI system to describe every frame in detail, Sarthi can ask focused questions when the navigation system needs additional information, such as:
“Are there stairs or steps visible in this image?”
“Is there a curb or step-down at ground level?”
“Is there an obstacle directly in the walking path?”
“Is there a pothole or hole in the walking surface?”
This makes Cloudinary part of the mobility intelligence pipeline, rather than simply being used as image storage.
📱 Mobile-First PWA
Sarthi is designed as a mobile-first Progressive Web App.
The mobile experience integrates:
- Camera access
- GPS positioning
- Voice input
- Voice output
- Navigation map
- Live environmental perception
- Responsive controls
- PWA installation support
The visual interface provides navigation context, while voice remains central to the experience.
🔄 Resilient Real-Time Perception
Real-time AI systems introduce practical problems such as latency, overlapping requests, network failures and rate limits.
Sarthi is designed around these constraints.
The perception loop uses:
- One perception request in flight
- Controlled perception intervals
- No request backlog
- Rate-limit backoff
- Stale-state handling
- Provider failure handling
- Malformed-output handling
If the perception provider temporarily becomes unavailable, Sarthi does not convert that failure into a false statement about the environment.
Instead, it communicates the limitation and continues operating with the information that remains trustworthy.
How we built it
Sarthi is built as a React + Vite frontend with a Node.js + Express backend, with a modular navigation, perception and AI architecture.
🧭 Navigation Architecture
We separated navigation into dedicated services for:
- Destination intent extraction
- Place search
- Walking-route planning
- Navigation state
- Route progress
- Turn-by-turn guidance
This separation allows navigation decisions to remain structured and testable rather than putting the entire journey inside a single AI prompt.
👁️ Perception Architecture
The camera feeds into a perception pipeline that converts visual observations into structured world information.
Instead of asking:
“What does this image contain?”
the system is designed around the more useful question:
“What information from this image matters for the user's current journey?”
🧠 Reasoning + Deterministic Safety
One of our most important architectural decisions was separating AI reasoning from deterministic safety logic.
Generative AI can help interpret complex visual and natural-language information, while deterministic services handle things such as:
- Safety prioritization
- Freshness
- Relevance
- Cooldowns
- State transitions
- Validation
- Failure handling
This reduces the risk of allowing a language model to make unsupported safety conclusions.
🗣️ Voice Architecture
Voice is treated as a first-class interaction layer rather than an additional feature.
The user can provide intent through speech, receive navigation instructions through speech, and receive proactive warnings when the system determines that something important requires attention.
Challenges we ran into
The hardest problem wasn't getting an AI model to recognize an object.
It was deciding when that information actually matters.
A camera can detect dozens of things in a single frame. Telling the user about all of them would make navigation worse, not better.
We therefore had to solve several deeper problems.
1. When should Sarthi speak?
We introduced relevance, priority, deduplication and cooldown logic so that Sarthi doesn't continuously repeat itself.
The goal is not maximum narration.
The goal is minimum necessary interruption.
2. How do we handle uncertainty?
No detection does not mean no obstacle.
A stale frame, failed model request or unavailable provider cannot be treated as a clear environment.
Sarthi therefore tracks perception freshness and explicitly preserves uncertainty.
3. AI hallucination vs. deterministic safety
We didn't want a language model deciding something like:
“The road is safe to cross.”
Safety-critical behavior requires fresh information and deterministic rules.
AI can assist with perception and reasoning, but it should not manufacture certainty when the underlying evidence is missing.
4. Maps vs. reality
A route can be geographically correct while the immediate environment has changed.
That means navigation cannot simply be:
GPS → instruction → GPS → instruction
Sarthi instead works toward continuously comparing:
planned route ↔ observed world
and responding when those two sources disagree.
5. Real-time constraints
Continuous vision requests introduced latency, overlapping requests and API rate-limit problems.
We built around:
- One perception request in flight
- Controlled perception intervals
- No request backlog
- Rate-limit backoff
- Stale-state handling
This makes the architecture more suitable for real-time mobile use than simply sending every camera frame to an AI model.
Accomplishments we're proud of
We're proud that Sarthi evolved beyond a simple “AI camera for the visually impaired.”
We built the foundations of an actual mobility system:
- 🧭 Natural-language destination understanding
- 📍 Nearby place discovery
- 🗺️ Route-aware walking navigation
- 🎙️ Voice-first interaction
- 👁️ Continuous visual perception
- 🧠 Persistent world-state reasoning
- ⚠️ Deterministic safety prioritization
- 🔄 Route ↔ environment reconciliation
- 🔁 Adaptive navigation architecture
- 🌐 Multilingual interaction foundations
- 📱 Mobile-first PWA architecture
- ☁️ Cloudinary-powered visual media pipeline
- 🛡️ Explicit uncertainty and failure handling
We also built extensive automated validation around the navigation and perception layers, including negative cases where the system must refuse to make an unsafe assumption.
One principle became particularly important to us:
An AI mobility assistant should know when it doesn't know.
What we learned
We initially thought the biggest challenge would be computer vision.
It wasn't.
The deeper challenge was decision-making under uncertainty.
We learned that building an assistive AI system requires much more than connecting a vision model to text-to-speech.
The system has to understand:
What is happening? Does it matter? How urgent is it? Is the information fresh? What should the user do next? And how certain are we?
We also learned that AI should not control every part of the system.
Generative AI is useful for perception and reasoning, but critical safety behavior benefits from deterministic state machines, validation and explicit failure handling.
Most importantly, we learned to design around the user's cognitive load.
The goal isn't to tell the user everything the AI can see.
The goal is to tell them:
the right thing at the right moment.
What's next for Sarthi
Our next step is to make the existing navigation architecture more complete and reliable for continuous real-world journeys.
We're working toward a journey where:
Take me to the nearest pharmacy.
becomes:
Find → Route → Navigate → Observe → Detect → Warn → Reconcile → Reroute → Arrive
Future work includes:
- More reliable real-world walking navigation
- More robust live rerouting
- Stronger route ↔ camera reconciliation
- More robust obstacle and hazard detection
- Interactive camera-guided environmental scanning
- Stronger multilingual voice interaction
- Better handling of GPS and perception uncertainty
- Extensive testing across real mobile devices and real-world routes
Ultimately, we don't want Sarthi to be an app that users constantly have to operate.
We want it to become a quiet companion that understands when to speak, knows when to stay silent, and helps the user move through a changing world with greater awareness.
You choose the destination. Sarthi figures out the way.
Log in or sign up for Devpost to join the conversation.