Title Slide
UI Mockups

Inspiration

We miss the energy of sitting around a campfire and telling thrilling stories with your friends. Syncopation is here to revive dynamic storytelling by interpreting speeches and producing sound effects based on story context, keywords, and tonality.

With the rise of technology, we crave genuine human interactions and demand communities. Syncopation is here to revive dynamic storytelling by interpreting speeches and producing sound effects based on story context, keywords, and tonality.

What it does

Based on your context prompt, the program grab sounds effects and background music from fetch apis and a pre-set library and loads it into the active story session. Once the context is completely loaded, the user is then prompted to tell the story and listen in as Syncopation generates contextual sound bites alongside the story!

How we built it

Syncopation runs as two loops that work at different speeds, because an LLM is too slow to react to a spoken word but smart enough to see where a story is going.

• Turning a theme into a sound library. The storyteller holds a button and says their theme. Claude (Haiku) turns it into concrete sound events people actually name when telling that kind of story: for “a family dinner”, that’s chopping, a sizzling pan, a doorbell, a smoke alarm, plus music moods and background sound. We search Freesound, Claude picks the file that really is the sound, and ElevenLabs generates anything we can’t find. The first sounds are fetched in the background while the user is still on the setup screens; the rest stream in during the talk. • The slow loop (Claude, every sentence, about 3 s). Claude tracks the story’s arc, tension, and mood, and predicts the next 5 moments. It also lists the trigger words to listen for, and bets on sounds the library doesn’t have yet, so they’re fetched before they’re said. • The fast loop (about 20 ms per word, no network). Deepgram Nova-3 streams words as they’re spoken. A lightweight online logistic-regression scorer combines Claude’s predictions with word matches, tension, and a local embedding model (all-MiniLM) that checks whether a sentence actually fits the story. After each sentence, Claude judges in hindsight which sound would have fit, and that becomes the training label, so the scorer keeps learning across talks. • A monitor for surprises. When a speaker names something nobody predicted (“a wolf howled”), a small Claude call describes the sound and we fetch it in 2–3 s. Soft sounds fade in late; sharp ones are kept, so the next mention fires instantly. • A custom mixer. We built our own 44.1 kHz mixer in Node: music changes crossfade in time with the beat and key music dips gently under each effect each sound gets its own fades, softening and room echo, from metadata Claude assigns (softness, attack, tail, importance) • Claude also sets the scene’s acoustic space (room, hall, cave, outdoors) and how much effects blend into the music. • Story rooms. The storyteller creates a story and gets a random code. Others join with the code from their phones and follow the same talk page live, shared through a Cloudflare tunnel.

Stack: TypeScript/Node, React + Vite, Claude Haiku, Deepgram Nova-3, Freesound, ElevenLabs, transformers.js, sox/ffmpeg, Cloudflare Tunnel.

Challenges we ran into

• Latency. Sounds must land on the word, but Claude takes about 3 s per call, so we split the system into a slow prediction loop and a fast (about 20 ms) reaction loop. • Missing sounds. “The doorbell rang” fails if there’s no doorbell in the library. Claude now predicts 5 steps ahead and fetches likely sounds early, and a monitor fetches anything unexpected in 2–3 s. • False triggers. Speech recognition guesses words before correcting them (“breathing” fired a radio, “bells rang” fired a phone). We added stability checks, one-word-one-sound, a 15 s repeat limit, and tense matching (“rang” → “ring”). • Natural mixing. Effects were 15–20 dB louder than the music, so they sounded pasted on. A scene-level mix decided by Claude, plus gentler ducking, fixed it. • API limits. We handled Freesound’s rate limits and ElevenLabs permission errors with caching, request pacing, a key checker, and Claude reusing fitting sounds from past sessions. • No hardcoding. We kept every sound decision with Claude and improved its instructions instead of mapping themes to fixed sounds. • Deployment. The backend needs a mic, speakers, and a long-running process, and event Wi-Fi blocked phones from reaching the laptop. We solved it with a one-port share mode and a Cloudflare tunnel.