SlopRock

Inspiration

Camera-only games like 67 Speed already have 17M+ players, and hand tracking now runs in real time in a browser with no special hardware.
Guitar Hero and Rock Band proved people will pay for the fantasy of being on stage, but they locked you into a fixed setlist and $60–$180 plastic controllers that ended up in closets.
SlopRock keeps the fantasy and deletes the plastic: your webcam is the instrument, and any song you upload becomes a playable band arrangement in about a minute.

Upload anything: drop in an MP3, WAV or MP4 (music videos included). We separate it into stems, auto-chart playable note highways for drums, guitar and bass, and extract the sung melody and lyrics, all in about a minute.
Play together: create a room, share a 4-letter code, and up to four players each claim an instrument or vocals. Scores, combos and a shared Band Energy meter sync live for everyone.
Hands-free control: webcam hand tracking (MediaPipe) reads finger curls as note hits. Each player calibrates their own instrument. The keyboard always works as a fallback.
Real recording: your instrument stays quiet until you hit its notes, then the actual recorded track swells in. Miss and it fades out while the band plays on.
Sing for real: a Vocal Booth with live autotune tuned to the song's own key and melody, plus reverb and polish knobs, mic and speaker pickers, and hum/noise filtering.
Karaoke mode: a scrolling pitch lane against the song's real vocal line, word-by-word lyric highlighting, and a live on-pitch score.
Video backgrounds: upload a music video and its footage plays full screen behind the game, synced to the audio, with a draggable seek bar.
Live on the web: Vercel frontend plus a media service on Azure Services

ElevenLabs (the AI engine)

Stem separation: splits each upload into drums, bass, guitar, piano, vocals and other. This is what makes "any song becomes a playable band" possible.
Speech-to-text (Scribe): run on the isolated vocal stem for accurate lyrics with word timestamps.
Pitch and melody: the same vocal stem goes through pitch detection to produce the sung melody, which drives the karaoke chart and the autotune.

SpacetimeDB (multiplayer backbone)

A Rust module holds authoritative state for rooms, seats, hits, scores and the song catalog.
Clients subscribe over WebSockets, so every screen updates live.
A shared start timestamp keeps countdowns and song start aligned across laptops.
A song uploaded by one player appears for everyone in the catalog, and the server fills empty seats with bots.

Frontend

Next.js and React on Vercel.
A canvas note-highway renderer and a Web Audio engine that schedules stems sample-accurately.
Autotune runs in an AudioWorklet (pitch detection plus a pitch-synchronous shifter), so it never blocks the main thread.
Hand tracking runs in a web worker.

Media pipeline

Python/FastAPI on Azure Container Apps, deployed through GitHub Actions and Azure Container Registry.
ffmpeg converts audio and video, and librosa and Basic Pitch turn stems into charts and melodies.
Storage is on Azure Files, so uploads persist.

Real-time pitch correction in the browser: we needed added latency low enough to sing along to, so we built a custom AudioWorklet that adds 0 ms while you're in tune and only a few ms while it corrects, and tunes to the song's actual key instead of a fixed scale.
Keeping everything on one clock: six stems, a background video, a live microphone and multiple laptops all had to stay in sync. Naive timers drifted, so we moved to Web Audio scheduling and fixed a bug where audio started late after decoding.
Speaker vs. headphone audio: browser voice-processing on macOS silenced the song on laptop speakers, so we replaced it with our own hum filter, noise gate and feedback-safe monitor level.
Auto-charting that feels fun: turning raw onset and pitch data into charts that are playable by hand and by webcam took a lot of tuning of note density and lane assignment.
Making a public link safe: we retrofitted a password and rate limits onto endpoints that were built for a trusted network.
Camera input reliability: hand tracking needed per-player calibration to avoid lag and false triggers.

Live, browser-based autotune with latency low enough to actually sing along to, with no plugin or app.
A pipeline that takes any song from "just an MP3/MP4" to a playable band arrangement with lyrics and a vocal chart, with zero manual charting.
Multiplayer rooms that stay in sync across devices with hand tracking, audio and video.
A publicly deployed full-stack app (Vercel and Azure), not just a local demo.

Web Audio scheduling, not timers, is the only reliable way to keep multiple audio sources and video in sync.
AudioWorklets are the right tool for anything latency-sensitive.
Auto-generated content (charts, lyrics, pitch targets) needs a "feel" pass, not just a "technically correct" pass.
Deploying for real (CORS, secrets, rate limits, redeploys mid-demo) is a different problem from getting a demo working on one laptop.