Prism

See the music: Prism turns any song into a living, generative painting so everyone, including deaf and hard-of-hearing people, can experience its shape.

Comment

Prism

Inspiration

Music is one of humanity's most universal art forms, yet millions of people who are deaf or hard of hearing experience it in a fundamentally different way. We wanted to explore a simple question:

What if music could be seen instead of heard?

Rather than creating another traditional audio visualizer, our goal was to build a new artistic medium that translates sound into living, evolving paintings. Every song should create its own unique visual world that reflects its emotion, rhythm, texture, and energy in real time.

We also wanted the experience to feel beautiful for everyone—not just as an accessibility experiment, but as a completely new way to experience music.

What it does

Prism is a browser-based music player wrapped around a real-time generative art engine.

Users can:

  • Play built-in music, upload their own songs, or use microphone input.
  • Watch the audio transform into dynamic artwork that evolves continuously.
  • Switch between five unique visualization modes:
    • Warp — a hyperspace tunnel driven by energy.
    • Bloom — expanding waves of glowing light.
    • Ribbons — twisting silk-like strands flowing with the music.
    • Aurora — animated curtains of colorful light.
    • Bubbles — iridescent, metamorphic glass blobs.
  • Save any frame as a high-resolution PNG artwork.

Instead of assigning fixed colors, Prism extracts a palette directly from each track's album artwork, automatically recoloring both the visuals and interface so every song feels like its own visual universe.

Every animation responds to the actual music using the live frequency spectrum, waveform, loudness, and beat detection, making the artwork feel synchronized rather than random.

How we built it

Frontend

Built using:

  • Vite
  • React
  • TypeScript

Audio Processing

We built the audio pipeline using the Web Audio API.

An AnalyserNode provides:

  • FFT frequency spectrum
  • Time-domain waveform

From these, we derive:

  • Loudness
  • Bass, mid, and treble energy
  • Dominant pitch
  • Beat detection through a lightweight onset detector

To make movement feel musical rather than noisy, we smooth the incoming signal using temporal damping with fast attack and slow release.

Rendering Engine

Everything is rendered in real time using an HTML5 Canvas 2D engine.

Different rendering techniques power each visualization:

  • Additive blending for glowing effects
  • Conic gradients for iridescent glass
  • Quadratic Bézier curves for smooth ribbons and auroras
  • Particle systems and flow fields
  • Motion trails that accumulate into lasting paintings

Despite using only Canvas 2D, careful layering of gradients, highlights, rim lighting, and blend modes creates convincing depth and a pseudo-3D appearance.

Dynamic Theming

To make every song visually unique, Prism samples the album artwork using Canvas getImageData().

From the image, we generate:

  • Primary accent color
  • Secondary accent color
  • Background tint

These colors drive both the visualization and the application's interface, allowing every track to have its own identity automatically.

Challenges we faced

One of the biggest challenges was making the visuals genuinely respond to music instead of simply moving randomly.

Raw FFT data changes rapidly and can produce jittery motion. We solved this by introducing temporal damping and a shared beat envelope that allows animations to react immediately to strong beats while decaying smoothly afterward.

Another challenge was creating convincing iridescent glass using only the Canvas 2D API. Since Canvas lacks advanced shader support, we recreated the effect manually using clipped conic gradients, radial sphere shading, rim lighting, and carefully layered specular highlights.

Performance was another major obstacle. Multiple glow passes, particle systems, gradients, and motion trails can quickly become expensive, so we optimized the rendering pipeline to maintain smooth 60 FPS animation while preserving visual quality.

What we learned

Building Prism taught us how expressive raw audio data can become when transformed thoughtfully.

We learned how to convert FFT data, waveforms, and beat detection into natural, musical motion instead of mechanical animation.

We also discovered that HTML5 Canvas 2D is far more capable than we initially expected. With careful use of gradients, blending modes, and layered rendering techniques, it's possible to create visuals that resemble modern shader-based graphics without requiring WebGL.

Most importantly, we learned that accessibility can inspire entirely new creative experiences rather than simply adapting existing ones.

Accomplishments that we're proud of

  • Built five visually distinct real-time rendering modes.
  • Created artwork that genuinely synchronizes with live music.
  • Developed automatic cover-art color extraction that gives every song its own visual identity.
  • Maintained smooth real-time performance while rendering complex layered effects.
  • Reimagined listening as a visual experience that can be appreciated by everyone.

What's next for Prism

We're excited to continue expanding Prism by adding:

  • More visualization modes.
  • User-adjustable parameters and customization.
  • Video recording that captures both the artwork and audio.
  • Sharing and gallery features so users can collect and showcase visualizations generated from their favorite songs.
  • Additional accessibility features and creative tools for artists and musicians.

Built With

Share this project:

Updates