Deaftones

DeafTones Frontend

Inspiration'

We started with our three guiding words from the beginning of the hackathon: "grain", "sync", and "voice." We wanted to represent a person's "voice" through music, such that we can analyze the emotional "grain" of a song to create relevant photo covers. We also wanted to "sync" the song with a real-time moving color panel display along with the audio and its chunks, to create an aesthetic experience.

What it does

Deaftones is a real-time music emotion visualization tool that listens to audio in short chunks, analyzes musical features (energy, tempo, and key), and uses an AI agent to classify the emotional character of the sound. The system then transforms these emotions into dynamic visuals, allowing users to see the feeling of music — bridging sound and emotion through color, motion, and light. How we built it We used Python + Librosa to process incoming audio, slicing it into 6-second chunks and extracting key features like RMS energy (loudness), tempo (rhythm), and key (major vs. minor tonality). These features are passed to an Agentic LLM classifier that predicts the emotional distribution (e.g., happy, sad, energetic). The results are streamed via FastAPI and HTTP streaming responses to a Next.js frontend, where a real-time canvas visualizer translates the emotional data into evolving animations and color gradients. Challenges we ran into

Some challenges we ran into include:

-Aesthetic Music Photo covers (It was difficult to find the write image generation software that provided photo covers that accurately matched the emotional distribution)

-color panel moving display(It was difficult to get the audio and the color panel syncronized correctlty)

-Real-time Analysis( It was diffucult figuring out the how we should do real-time display, and we decided to use HTTP streaming to implement it)

Accomplishments that we're proud of

-We are proud of the synchronization of all of our parts(the chunks, audio timing, and the moving color panel) -We are proud of the HTTP streaming, as that is something new and interesting that we implemented -We are proud of the audio analysis of emotions as we used new techniques that we learned today.

What we learned

  -  HTTP Streaming
  -    Luna Photon Flash for photos
   -    Audio Processing techniques

What's next for deaftones

-Since this tool has been deployed, hopefully we can have our tool be popularized and used by many people as a way to enjoy music aesthetically and understand how their music is being interpreted emotionally.