SixthSense

Landing
Dashboard
Whole Setup
Where System
What System
System Architecture

Inspiration

Over 253 million people worldwide live with visual impairments, facing daily challenges in environmental awareness and independence. We were deeply moved by these struggles and saw an opportunity to create something truly transformative. Current assistive technologies often overload hearing—a sense blind individuals already heavily rely on—or provide only basic obstacle detection. We wanted to create something that would open up a new dimension of perception, allowing visually impaired people to "feel" their surroundings in a meaningful, intuitive way.

What it does

SixthSense translates the visual world into a language of touch that becomes more intuitive over time. Our system captures what's around the user through Meta glasses, recognizes objects with YOLO V8, and converts each object into a unique vibration pattern that can be felt—similar to how musical notes combine to form a chord.

What makes our approach special is the dual feedback system: a glove (with a vibration motor) plays unique "tactile signatures" to identify objects, while the gauntlet (with servo motors) gently guide the user's hand toward their location. When you encounter a chair, it doesn't just feel like "something"—it feels specifically like a chair, with a signature similar to other chairs but distinct from tables or doorways. As objects combine in a scene, they create rich, layered tactile experiences that can be understood holistically.

How we built it

We designed SixthSense to mirror how our brains naturally process visual information, with separate pathways for "what" and "where":

Visual Input: Meta glasses capture the environment
Object Detection: YOLO V8 identifies objects and creates bounding boxes
Semantic Processing: Each object name gets converted to a unique embedding vector
Signal Transformation: Using Inverse Fourier Transform, we convert these embeddings into distinctive vibration patterns
Spatial Positioning: X-Y coordinates determine servo motor positions
Signal Normalization: We carefully balance multiple signals for a comfortable experience
Dual Output System: Separate Arduino controllers manage object identity and directional feedback

We built the system through countless iterations, all nighters, and many moments of both frustration and breakthrough. The moment when our first test user correctly identified objects based solely on their vibration patterns was truly extraordinary.

[PRESENTATION attached below contains more details about the exact methods.]

Revolutionary Advantages Over Current Technology

SixthSense represents a fundamental leap beyond existing assistive technologies:

Beyond Simple Detection: While traditional tools like white canes and proximity sensors only detect obstacles, SixthSense communicates what objects are and where they're located—transforming navigation from avoidance to understanding.
Preserving Auditory Channels: Unlike screen readers and audio description tools that compete with environmental sounds, SixthSense leaves hearing untouched, allowing users to maintain awareness of conversations, traffic, and other important audio cues.
Semantic Understanding: Current haptic systems might tell you "something is there," but SixthSense tells you "there's a chair to your left and a table ahead"—providing context that makes all the difference in confident navigation.
Natural Information Processing: Instead of processing one piece of information at a time (like Braille readers), SixthSense allows users to feel multiple objects simultaneously, mimicking how vision naturally works.
Adaptive Learning: The system becomes more intuitive with use, as the brain learns to recognize patterns more quickly—creating a growing vocabulary of tactile sensations that becomes second nature over time.

Groundbreaking Innovations

What makes SixthSense truly novel:

Embedding-to-Vibration Translation: We've pioneered the first system that transforms semantic meanings into consistent tactile patterns through embedding vectors and Fourier transforms—creating a true "touch language" for objects.
Brain-Inspired Dual Pathways: By separating "what" and "where" information, we've created an architecture that works with natural neural processing—making the system feel more intuitive and less like a tool.
Harmonic Tactile Language: We've developed a system where similar objects create similar sensations and complex scenes blend harmonically—allowing users to build an intuitive understanding of their environment.
Additive Sensory Composition: Our unique approach allows multiple object sensations to combine without losing clarity—a breakthrough in tactile information density.
Quantitative Pattern Consistency: Our testing demonstrates mathematically that the system creates learnable patterns, with similar objects showing 71.68% correlation versus just 10.23% for different objects.

Challenges we ran into

Creating SixthSense wasn't without its difficulties:

Pattern Distinctiveness: We struggled to create vibration signatures that were unique enough to distinguish different objects but similar enough for related items—requiring multiple iterations of our embedding approach.
Information Balance: Early prototypes overwhelmed users when too many objects appeared simultaneously—we had to completely rethink our normalization algorithms.
Calibrating Feedback: Finding the right intensity for servo motor guidance took weeks of testing—too strong felt uncomfortable, while too gentle was easy to miss.
Real-time Performance: Reducing latency from seconds to milliseconds required significant optimization of our processing pipeline.
Component Integration: Coordinating all the hardware and software elements (Meta glasses, processing unit, and Arduino outputs) presented numerous synchronization challenges.

Accomplishments that we're proud of

Despite these challenges, we've achieved something remarkable:

Functional Prototype: We've created a working system that allows users to identify and locate objects through touch alone.
Intuitive Pattern Recognition: Our testing shows users can learn to recognize objects through their vibration patterns with increasing accuracy over time.
Multi-Object Perception: We've successfully implemented a system where multiple objects can be sensed simultaneously without information loss.
Seamless Integration: We've combined cutting-edge technologies across computer vision, natural language processing, and haptic feedback into a unified system.

What we learned

This journey has taught us valuable lessons:

Neural Adaptability: The brain's remarkable ability to learn new sensory patterns exceeded our expectations, suggesting exciting possibilities for sensory substitution.
Embedding Semantics: Vector representations that capture meaning relationships translate beautifully to physical sensations.
Signal Processing Applications: Fourier transforms offer elegant solutions for converting abstract data into meaningful physical patterns.
User-Centered Design: Our most important insights came from testing with visually impaired users, completely reshaping our understanding of effective assistive technology.
Interdisciplinary Innovation: The most powerful solutions emerge at the intersection of different fields—in our case, computer vision, linguistics, signal processing, and haptics.

What's next for SixthSense

We're just getting started:

Extended User Testing: We're planning comprehensive testing with visually impaired individuals to refine our approach based on real-world feedback.
Miniaturization: We aim to reduce the system's size and weight for better everyday wearability.
Personalized Recognition: We want to train the system to recognize objects specific to each user's environment.
Power Optimization: Improving battery life for all-day use is a top priority.
Learning Curriculum: We're developing a structured program to help new users become proficient with the system.
Community Partnerships: We're reaching out to organizations serving the visually impaired to ensure our technology addresses real needs and reaches those who need it most.

SixthSense doesn't just bridge worlds—it transforms the visual landscape into a tactile symphony where every object, every distance, and every relationship becomes a note in the beautiful melody of understanding.