Concerto

Inspiration

At concerts, the audience usually participates by watching, singing, or turning on their phone flashlights. We wanted to take that further and turn every audience member’s phone into part of the sound system.

Concerto transforms a crowd of phones into a distributed orchestra. Each phone is assigned a musical part based on where its owner is sitting, allowing different sections of the audience to play melody, vocals, or percussion together.

What it does

Audience members scan a QR code and open Concerto in their browser. The application automatically connects their phone, synchronizes its clock with the server, and downloads the required audio.

To determine where each phone is located, Concerto displays a timed red-and-blue pattern on every participating screen. Cameras record the audience, and our computer vision pipeline decodes each phone’s unique optical ID and estimates its position in the venue.

The operator can then:

• Review a map of the detected phones
• Select groups using rectangles, a lasso, or adjustable regions
• Assign Melody, Vocals, or Percussion to each group
• Control playback from a shared timeline
• Adjust gain, mute or solo channels, pause, seek, and stop the performance
• Immediately mute every phone using an emergency control

All phones follow the same server-controlled playhead. If optical localization fails, audience members can manually select the left, center, or right section.

How we built it

Concerto is a browser-based distributed system with four interfaces:

• The audience experience
• The operator console
• A projector view with a QR code and live readiness counts
• A camera upload interface

The frontends use Next.js, React, and TypeScript. A Bun and Hono backend manages device identities, synchronization, assignments, playback state, media uploads, and operator commands.

For synchronization, each phone performs NTP-style clock exchanges with the server. We estimate the server-to-device clock offset using the lowest-latency samples. Playback commands are scheduled for a shared future server timestamp instead of being executed when each message arrives.

Audio is downloaded and decoded before the show using the Web Audio API. Each phone only plays its assigned channel, but every channel follows the same global timeline. The client also handles browser autoplay restrictions, reconnection, stale commands, audio interruptions, and late joins.

Our optical localization worker is written in Python using OpenCV, PyAV, and NumPy. Each phone transmits an 11-bit device ID encoded as an extended Hamming code. The 55-symbol packet includes timing guards, color pilots, a preamble, a run identifier, and two complementary copies of the codeword. The worker tracks screens through the recording, samples their colors, rejects ambiguous observations, and produces a reviewable audience map.

Shared Zod schemas validate messages between the TypeScript services and the Python worker. We also use SHA-256 hashes to verify audio and uploaded recordings.

Challenges we ran into

The first major challenge was synchronization. Sending a “play now” message is not enough because network latency varies between phones. We had to separate message delivery from execution and schedule every action against a synchronized server clock.

Browser audio introduced another layer of complexity. A connected phone may still have a suspended AudioContext, missing assets, or an output path that has not warmed up. We track connection, clock, visibility, audio, and asset readiness separately so the operator does not see false readiness.

Optical localization was also difficult. Real recordings include motion blur, occlusion, exposure changes, reflections, compression, and screens that occupy very few pixels. Our decoder therefore treats uncertainty as an erasure, requires bounded error correction, and rejects conflicting or duplicated identities instead of guessing.

Finally, the system needed to recover safely. Reconnecting phones must resynchronize before playing, stale commands must not affect a newer session, and disconnected phones must eventually become silent even if they cannot receive an emergency mute message.

Accomplishments that we're proud of

• Built an end-to-end workflow from QR-code join to synchronized multichannel playback
• Created a computer vision pipeline that decodes unique phone IDs from recorded screen patterns
• Designed a conservative optical packet using extended Hamming error correction
• Built an interactive audience map with click, rectangle, lasso, and region selection
• Implemented scheduled assignment, playback, seeking, mixing, and emergency mute controls
• Added resumable, integrity-checked camera uploads
• Supported automatic reconnect and playback recovery at the current shared playhead
• Tested the control system with 1,500 simulated WebSocket clients
• Kept optical, audio, clock, and operator state clearly separated throughout the architecture

What we learned

We learned that clock synchronization and audible synchronization are related but different problems. Accurate timestamps help every phone schedule playback consistently, but speaker hardware, browser audio pipelines, and the physical speed of sound still affect what the audience hears.

We also learned that computer vision systems should be designed to reject uncertain results. Returning “unknown” is much safer than assigning the wrong musical part to a phone.

Most importantly, we learned that readiness is not one boolean. A phone can be connected without being synchronized, synchronized without having audio enabled, or audio-enabled without having the required music downloaded. Treating those states independently made the system much more reliable.

What's next for Concerto

Our next step is testing Concerto with a larger group of physical iOS and Android devices in a real venue. We want to measure acoustic timing, camera coverage, network capacity, and optical decoding at different seating distances.

We also plan to improve the camera pipeline for smaller and partially blocked screens, simplify the operator workflow, and explore musical arrangements designed specifically for spatial playback across an audience.

Built With

• codex