SightGuide MR – Assistive Mixed Reality Navigation

Result Image 1
Result Image 2

Inspiration

SightGuide MR was inspired by the everyday challenges faced by visually impaired individuals navigating unsafe roads. In many environments, sidewalks are uneven, potholes are common, water collects on walking paths, and vehicles or pedestrians frequently block safe routes. Even small obstacles can become serious risks.

We wanted to explore how Artificial Intelligence and Mixed Reality could work together to provide not just object detection, but true safety understanding — helping users know where it is safe to walk and why.

This led to SightGuide MR: an explainable mixed reality navigation system designed to improve independent mobility.

What it does

SightGuide MR is an AI-powered mixed reality assistive navigation system for visually impaired users.

Using the Meta Quest headset camera, the system analyzes real-world scenes in real time to:

Identify walkable regions
Detect surface hazards such as potholes, debris, and water
Recognize dynamic obstacles like people and vehicles
Generate audio-based navigation guidance such as
“Caution, pothole ahead — move slightly right.”

Logitech MX Ink introduces human-centered interaction. Caregivers or trainers can use the stylus to draw safe walking corridors, mark hazardous areas, annotate obstacles, and assist in training new environments directly in mixed reality.

How we built it

The system follows a modular perception and reasoning pipeline:

Semantic segmentation identifies walkable surfaces and dynamic objects.
Road damage annotations are converted into segmentation masks to detect potholes, debris, and water hazards.
Safety reasoning computes navigable regions using:

[ SAFE = WALKABLE - (HAZARDS \cup OBSTACLES \cup VEHICLES) ]

Mixed reality overlays visualize hazards and safe paths.
MX Ink enables drawing paths, highlighting risks, and annotating environments.
Final navigation instructions are delivered through spoken audio feedback.

To train and validate the system, we unified multiple datasets:

Indian Driving Dataset (IDD) for walkability and road scene understanding
Road Damage Dataset 2022 (RDD2022 – India subset) for potholes, debris, and water
BDD100K for vertical obstacles such as poles and traffic signs
Mapillary Vistas for crosswalks, road markings, and traffic signals

These datasets were mapped into a shared semantic format, enabling spatial fusion of walkability, hazards, and obstacles.

Challenges we ran into

One major challenge was aligning datasets with different annotation styles (polygon segmentation versus bounding boxes) into a unified representation.

Designing safety logic required prioritizing false-negative avoidance while remaining practical for real-time use. Early versions intentionally use conservative hazard masks to favor safety.

Estimating distance from a single camera also required careful handling, leading us to adopt relative proximity zones instead of unreliable metric depth.

Integrating MX Ink required rethinking traditional computer vision pipelines to support explainable mixed reality interaction.

Accomplishments that we're proud of

Successfully unified multiple road and hazard datasets into a single safety-aware representation
Built an explainable navigation pipeline that reasons about safe walking areas instead of just detecting objects
Integrated Logitech MX Ink for human-in-the-loop mixed reality interaction
Designed a modular system that can scale to video, depth estimation, and motion awareness
Created a socially impactful prototype focused on accessibility and independence

What we learned

We learned that assistive AI systems require much more than object detection. They need safety reasoning, explainability for user trust, conservative design for reliability, and human-in-the-loop interaction for adaptability.

We also discovered the power of Mixed Reality as an accessibility platform when combined with AI — enabling intuitive interaction with the physical world in ways traditional interfaces cannot.

What's next for SightGuide MR – Assistive Mixed Reality Navigation

Future work includes expanding from static images to real-time video, adding monocular depth estimation for accurate proximity reasoning, modeling motion and pedestrian intent, and introducing shared-encoder architectures for real-time performance.

Our long-term goal is to evolve SightGuide MR into a production-ready assistive navigation platform that empowers visually impaired users with greater independence, confidence, and safety.