Inspiration

Having worked directly in accessibility, I've seen where standard assistive technology falls short. While speech-to-text tools handle spoken dialogue, the vast majority of our sensory environment is non-verbal: a fire alarm across the hall, a boiling kettle, a car horn, or sudden applause. We wanted to build something practical and empowering that bridges this ambient information gap, giving Deaf and Hard of Hearing individuals richer, customizable access to their immediate surroundings.

What it does

Visuan transforms ambient environmental sounds into real-time visual banners and physical haptic alerts. Live Environmental Listening: Streams background audio continuously to identify critical acoustic cues. Intelligent Classification & Urgency: Categorizes sounds by context and priority (e.g., immediate safety warnings like sirens vs. routine household cues like door knocks or timers). Multi-Sensory Feedback: Delivers immediate, glanceable UI notifications accompanied by distinct vibration patterns, providing situational awareness without relying on spoken transcription.

How we built it

Prototyping & System Design: Designed the layout, UX flows, and core features using Google AI Studio to test how ambient soundscapes could be converted into structured data. Gemini Audio Processing: Leveraged Gemini's real-time multimodal audio capabilities to process incoming sound chunks and trigger structured function calls (e.g., emit_audio_alert(sound_type, urgency, description)). Front-End & Sensory Output: Built a lightweight, accessible user interface paired with the Web Vibration API to deliver responsive visual banners and tactical haptic feedback.

Challenges we ran into

Audio Calibration & Latency: Real-world soundscapes are messy. Fine-tuning the audio input to distinguish background noise from actual critical alerts took significant trial and error. Balancing Urgency & Alert Fatigue: Making sure routine sounds (like keyboard clatter or light footsteps) didn't trigger high-priority alerts, while ensuring safety-critical sounds (like alarms) were captured reliably and instantly. Hackathon Time Crunch: Coordinating the UI design, API integration, and sound-event testing within a tight deadline required ruthless prioritization of core features

Accomplishments that we're proud of

End-to-End Multimodal Pipeline: Built a functioning prototype that bridges real-time audio detection directly into structured visual and haptic outputs. Resilience Under Pressure: Debugging ambient sound thresholds repeatedly during a strict time crunch was tough, but we pushed through and delivered a cohesive, working demo. First Hackathon Milestone: For our very first hackathon, moving from an abstract accessibility concept to an operational, functioning prototype was a huge achievement for the team.

What we learned

Translating Ideas into Functional Code: Gained hands-on experience taking an app from conceptual UI wireframes to an actual working build under hackathon constraints. Multimodal AI Integration: Learned how to effectively prompt, structure, and utilize Gemini’s audio processing capabilities to handle streaming non-verbal sounds. Product Scoping: Learned how to devise an actionable technical plan, prioritize essential features, and coordinate rapid debugging during a live build sprint.

What's next for Visuan - Visual Soundboard and Environmental Audio Detection

Native Mobile Optimization: Visuan is fundamentally designed for on-the-go awareness. We plan to port the web build into native iOS and Android apps to fully leverage background service audio capture and high-fidelity device haptics. Directional Sound Indicators: Integrate multi-microphone arrays to show users not just what happened, but the direction the sound originated from. Expanded Classification Library: Broaden the detection taxonomy and refine model prompts to reliably distinguish subtle, nuanced sounds across varying home, transit, and workplace environments. Wearable Integration: Extend visual and haptic pings to smartwatches for discreet, wrist-based situational updates.