FurrySound

Inspiration

Music is a universal language, yet it remains a closed door for the millions of people living with hearing impairments. Our team was inspired by the simple question: "How can we let those who cannot hear music still feel its soul?" We set out to bridge this gap by transforming auditory signals into a rich, interactive visual reality, ensuring that the rhythm and emotional depth of music are no longer restricted by physical hearing.

What it does

FurrySound is an AI-driven music visualization platform and an interactive rhythm game. It performs real-time audio analysis to extract tempo, beats, and frequency intensities, translating them into a "Visual Rhythm Language." Whether through high-contrast accessibility modes or a 4-lane rhythm game, it allows users to "play" the music through sight and touch.

How we built it

Data Pipeline:The React frontend captures audio files via FormData and streams them to a - - - Node.js/Express** backend. We utilize Multer for secure file handling and temporary storage.
AI-Driven Analysis: To extract the "DNA" of the music, the backend spawns a Python child process. Using the librosa library, we perform Short-Time Fourier Transforms (STFT) and onset strength envelopes to identify tempo, beat frames, and frequency intensities: $$STFT{x[n]}(m, \omega) = \sum_{n=-\infty}^{\infty} x[n] w[n-m] e^{-j\omega n}$$
The Visualization Engine: Once the backend returns the JSON analysis, the frontend initializes the Web Audio API to sync the audio buffer with a custom Canvas 2D rendering loop.
Interactive Logic: We built a dedicated game engine that calculates the timing offset between the user's keystrokes (D/F/J/K) and the pre-analyzed beat timestamps, providing instant feedback through a dynamic score and combo system.

Challenges we ran into

Our biggest hurdle was the Visual Mapping Paradox: how do you represent a complex song without overwhelming the user? We went through dozens of iterations to find the right balance between "too simple" and "visual noise." In the rhythm game section, we struggled with Hit Precision and "Game Feel." Initially, the notes didn't feel "snappy." We had to recalibrate our judgment windows based on pixel-distance thresholds and implement a dynamic particle system. To make it "cooler," we added beat-driven screen shakes and lane-flashing effects that sync perfectly with the audio onset strength, ensuring a satisfying tactile response even without sound.

Accomplishments that we're proud of

Accessibility First: Creating a functional visual language where a Red Pulse = Kick and a Blue Flash = Snare.
Seamless Integration: Successfully piping complex Python audio data into a real-time web interface.

What we learned

Building this project was a humbling experience. We learned that translating sound to sight is incredibly difficult; it made us truly realize the daily challenges faced by the hearing-impaired. Technically, we mastered the art of Audio Feature Extraction—learning how to distinguish a beat from background noise. We also deepened our knowledge of Frontend Performance Optimization, specifically how to render thousands of particles at 60 FPS while processing high-bitrate audio. Also, we learned how to do deployment for a project, and also use our own domain for the web app.

What's next for FurrySound

We are just getting started. Our roadmap includes:

Latency Calibration: Every device has different audio-visual lag. We plan to add a user calibration tool to adjust the $t_{offset}$ for perfect synchronization.
Expanded Visual Vocabulary: Currently, we use 3-4 elements for rhythm. We want to introduce more complex shapes and textures to represent different instruments (e.g., strings vs. synths).
Aesthetic Evolution: We aim to refine the UI/UX with more sophisticated themes, better color gradients, and a more polished "Neo-Tokyo" or "Minimalist" aesthetic for the gaming interface

See our project demo video at https://youtu.be/ZQ0iUdEK9LQ