mockup1
mockup3
mockup2

HapticMind: The Future of AI is Tangible

"Innovation is only truly meaningful if it includes everyone."

Inspiration

When we heard about the opportunity to build a multimodal live AI experience, our vision was instantaneous. We realized that "multimodal" shouldn’t just be about seeing and hearing—it should be about feeling.

We set out to redefine the interface by introducing a tactile dimension to both AI input and output. HapticMind was born from the intersection of academic research and the urgent need for real-time sensory substitution in an era of evolving AI agents. We moved beyond simple text-to-speech, drawing deep inspiration from:

V-Braille (Jayant et al.): Proving that standard touchscreens can transmit Braille via synchronized vibration without specialized hardware.
Tactile Sensitivity (arXiv 2511.10026): Research validating that visually impaired users exhibit superior tactile sensitivity, with peak performance around $0.270 \text{ cycles/mm}$—remarkably close to real Braille dot spacing.
Haptic Keyboard Layouts (MDPI Electronics): Strategies for integrating Braille input/output on commodity mobile devices.
BraillePlay: Gamified interfaces that lower the barrier for Braille literacy.

What it Does

HapticMind acts as a "World Lens," transforming the smartphone into an intuitive sensory assistant for the hearing and visually impaired:

World Vision: Users ask "What do you see?" and the AI analyzes the camera feed to respond via voice and synchronized haptic Braille.
Contextual Intelligence: Answers general queries (weather, logic, facts) using Gemini's reasoning, intelligently ignoring the camera when irrelevant.
Multimodal Input: * Voice: High-fidelity hold-to-speak.
- Tactile Braille: A 3×2 virtual matrix with an adaptive tap threshold (short tap = blank, long tap = filled).
- Keyboard: For traditional interaction.
Multimodal Output:
- Audio: Natural, low-latency Gemini TTS (Text-to-Speech).
- Haptic Braille: Characters are "played" as 6-dot vibration patterns and reinforced by a row-scan visual animation.
Practice Mode: A dedicated sandbox for users to master haptic patterns and perfect their tapping pace.

How We Built It

We engineered a lean, high-performance stack to handle real-time, bidirectional data. To push the limits of the Gemini Live Agent Challenge, we crafted this proof-of-concept in under 48 hours.

Component	Technology
Frontend	Expo SDK 54, React Native, NativeWind
AI Engine	Gemini Live API (WebSocket)
Model	`gemini-2.5-flash-native-audio-preview-12-2025`
Audio Pipeline	`@speechmatics/expo-two-way-audio` (PCM) & Web Audio API
Haptics	`expo-haptics` + Custom Vibration Sequencing
Camera	Continuous JPEG streaming (~1.5s interval)

Challenges We Ran Into

VAD vs. User Autonomy: Gemini's automatic Voice Activity Detection often cut off speech prematurely. By implementing manual activityStart/activityEnd, we gave the user absolute control over the microphone.
The Adaptive Threshold: "One size fits all" timing for Braille taps failed. We developed an adaptive timing algorithm that learns and adjusts to a user’s specific tapping cadence in real-time.
Platform Audio Parity: Web and Native require vastly different audio stacks. We architected modular media layers to abstract PCM capture and playback across platforms seamlessly.
Haptic Typography: We had to treat vibration like a font. Tuning READ_FIELD_DURATION and READ_ROW_GAP was critical to ensure the feedback felt like distinct characters rather than "noise."

Accomplishments That We're Proud Of

The "Signature" of Words: There is a profound delight in feeling the distinct haptic signature of each letter as the AI communicates.
Seamless Multimodality: Achieving a stable, real-time loop of voice/Braille input and voice/haptic output on a single device.
Research-to-Reality: Successfully translating complex tactile metrics from research papers into a working, consumer-grade application.
Speed of Innovation: Building a cross-platform (iOS, Android, Web) solution from scratch in a single weekend.

What We Learned

Expo & Real-Time Data: While Expo is elite for prototyping, real-time audio/video interactions require deep native bridge mastery to avoid performance traps.
The Power of WebSockets: Bidirectional WebSockets are the "nervous system" of assistive tech, maintaining a natural, human-like flow.
Inclusive Prompting: System instructions are vital; teaching the AI when to "look" and when to "listen" prevents cognitive overload for the user.
Tactile UI: Timing isn't just a setting—it is the user interface itself.

What's Next for HapticMind

1. Field Validation We are moving into extensive user testing with the blind and low-vision community to ensure our "haptic typography" feels as natural as a heartbeat.

2. Deep OCR Integration While Gemini can describe text, we are building a dedicated OCR layer to instantly translate menus, signs, and documents into tactile Braille pulses with zero latency.

3. On-Device Autonomy To ensure safety in all environments, we are developing an Offline Mode using efficient, on-device models for basic object detection and navigation.

4. Hardware Ecosystem We are architecting support for external refreshable Braille displays, allowing HapticMind to serve as an intelligent bridge to existing tactile hardware.

5. Global Inclusion A world without borders. We are expanding our localized Braille tables and voice agents to support diverse regional standards and dialects.