BeatForge for Beatsaber

Interface for people/agents
Generated beatmap for Tame Impala's Is It True

Inspiration

I'm a top ranked player in Beatsaber and I got tired of playing the same tracks over and over again.

Beat mapping sits between music, interface design, and safety engineering. A mapper has to hear a song, decide how its energy should move through space, and check hundreds of technical rules.

I built the BeatForge Beat Saber level-generation pipeline alone. In my tests, MCP access reduced generation setup and iteration time to 1/10th of my previous manual process.

What it does

BeatForge is designed for any song. The track has to be uploaded locally as an MP3, WAV, OGG, FLAC, M4A, or MP4.

The public WebMCP connection has a separate rights-safe preview path. It can find metadata, album artwork (which it then takes the color palette of the artwork and applies it to the level environment)

The agent can read studio context, find song metadata, apply a mapping plan, start generation, review timing and safety results, and record human playtest evidence. It cannot claim that it played the map. Human headset judgment remains essential.

How I built it

FastAPI handles full audio uploads, background jobs, progress events, validation, packaging, and the local Beat Saber workflow. The analyzer converts the full song to a 44.1 kHz integer-sample timeline. Demucs is used for audio separation. BeatForge uses the htdemucs_6s model to separate the song into drums, bass, guitar, piano, vocals, and other stems. This helps identify which musical layers drive each section. Full-mix timing consensus still has to pass strict timing gates, so stem analysis never hides uncertain timing.

The generator independently creates Easy, Normal, Hard, Expert, and Expert+ maps. It checks hand reach, saber paths, recovery, collisions, walls, bombs, lighting, and structural validity. A generated chart remains a playtest_candidate until a human completes VR and fresh sight-read checks. The WebMCP layer registers eight typed JSON-Schema tools through document.modelContext.registerTool. The interface displays tool activity, mapping state, album artwork, generation status, and safety results. The public GitHub Pages demo uses a synthetic groove so judges can test the collaboration loop without copyrighted audio or local setup.

I spent a full week running BeatForge in local reinforcement-learning environments. Each environment stepped through a beat grid while exposing Demucs-separated stem energy, audio features, red and blue hand kinematics, difficulty targets, and lookahead context. A PPO policy selected idle or note poses for each hand. Rewards encouraged onset alignment, musical energy, readable density, natural flow, and safe two-hand coordination. Penalties covered repeated cuts, insufficient recovery, handclaps, saber-path collisions, vision blocks, and other unsafe transitions. These training runs improved candidate search and helped the generator produce more musical and playable patterns before validation. The RL policy could never override the hard safety contract. Deterministic timing checks, map validation, and human VR playtesting remained mandatory before release. The RL implementation is included in skills/beat-saber-mapping/scripts/rl/.

Challenges I ran into

I could only test 6 or so songs in a session because I’d get tired from swinging my arms around so much

Accomplishments that I’m proud of

Agents can now change a real mapping plan through typed operations instead of fragile UI guesses. People see those changes immediately and can accept, reject, or revise the direction. The activity panel makes the collaboration inspectable. We are also proud that the WebMCP work preserved BeatForge’s existing safety behavior. Timing uncertainty, mapping failures, and missing human evidence remain visible.

The repository now includes the WebMCP implementation, browser tests, setup instructions, an MIT license, and a deployment definition.

What I learned

WebMCP tool descriptions are part of the product experience. A useful schema tells an agent what an operation means and tells the person what will change. Demucs showed me that source separation can provide better evidence about which musical layer should drive a phrase, while full-mix timing verification remains necessary.