WaveLens

Captured Item
Captured Item
Customizable Instruments
Modification Page
App Icon
Saved Rendered Audio Clip
Collections Page
Extend Loop with ElevenLabs

Inspiration

WaveLens started with Qiyuan's idea of using body movement to modify music in real time, such as moving up and down to control volume. After several iterations, we expanded the idea: what if the objects around us could become part of the music?

That became WaveLens. Users can capture an everyday object, give it a unique musical identity, and gradually build an ensemble as more objects are added.

What It Does

WaveLens turns the physical world into a musical interface. The user takes a snapshot of an object, WaveLens recognizes it, assigns it an instrument, and generates a unique musical motif.

Capturing more objects adds more instruments to the ensemble. Users can then modify each instrument's properties, such as volume and musical complexity.

How We Built It

One of our biggest challenges was figuring out how to modify music dynamically. Aaron (Jiamu), who built the core app, music engine, and computer vision pipeline, designed the system to represent music as MIDI-style notes and instruments instead of manipulating finished audio.

This lets us render music in real time and change patterns, density, and complexity while keeping all instruments musically compatible.

Hank (Chenghan) built the hardware side, including the FPGA components, connecting the physical environment directly to the music system through computer vision, object tracking, and real-time audio.

Challenges

We faced challenges with real-time music modification, object detection and tracking, dependency management, project planning, and hardware adaptation.

Many of these problems required repeated brainstorming and redesign. ChatGPT from OpenAI also helped us debug, research unfamiliar technologies, and iterate on our architecture throughout the hackathon.

What We Learned

WaveLens taught us how computer vision, music production, real-time audio, and hardware can work together in one creative system.

We also learned that generating music is not just about producing notes. The music needs structure and constraints so that independently generated instruments still sound like one ensemble.

We are grateful for the support from Hack the North and its sponsors, which gave us the opportunity to explore technologies including OpenAI, Gemini, and ElevenLabs.

What's Next

In the future, we want WaveLens to understand more than just what an object is. Properties such as color, shape, material, position, and movement could influence melody, rhythm, timbre, volume, and complexity.

Our goal is simple: turn the world around you into something you can play.