Songaio Music

💡 Inspiration

We wanted to break the boundary between audio and visuals. Instead of a traditional, static music player, we set out to build an immersive, interactive 3D "sound sandbox" where users can physically see and interact with the shape of sound in real-time.

🛠️ How we built it

The project is driven by React, bridging native browser APIs with high-performance 3D rendering:

• Audio Engine: We utilized the Web Audio API to handle real-time audio routing, oscillators, and Fast Fourier Transform (FFT) frequency analysis.
• 3D Visual Engine: We used Three.js via React Three Fiber (R3F). This allowed us to map live frequency data directly to the scale, color, and lighting of dynamic 3D meshes declaratively.
• AI Assistance: OpenAI Codex 5.6 was heavily utilized to accelerate the development of complex audio-routing logic and generate custom WebGL shader boilerplate.

Core Algorithm Note: Translating raw soundwaves into visualized spectrum data relies on the Discrete Fourier Transform (DFT). The foundational math driving our audio analyzer is: $$X_k = \sum_{n=0}^{N-1} x_n \cdot e^{-i 2 \pi k n / N}$$

⚠️ Challenges we ran into

Our biggest bottleneck was rendering performance and synchronization. Processing high-resolution audio data at 60FPS while rendering complex 3D geometries in React is incredibly demanding. Initially, using standard React state (useState) to update the visuals caused severe frame drops and lagging.

We solved this by leveraging R3F's useFrame hook alongside useRef. By directly mutating the Three.js mesh properties (like .scale and .material.color) within the render loop, we completely bypassed React's standard component lifecycle, achieving buttery-smooth, zero-latency reactivity.

🧠 What we learned

1. Gained a deep understanding of the Web Audio API's Audio Node routing graph.
2. Learned the best practices for bridging React's declarative UI with imperative, high-performance 3D rendering loops (Three.js/WebGL).
3. Experienced firsthand how next-generation LLMs can drastically reduce friction when working with niche, low-level browser APIs.

Built With

• three.js
• webgl