Not Quite My Tempo

Practicing
Movie night (research ;))
In-game screenshot
Showcase

Another showcase: https://youtu.be/8UGVPzbgZ8s

The tyranny of Terence Fletcher from Whiplash, now available at home with the power of VR!

Inspiration

The movie Whiplash! We came into Cal Hacks wanting to build something unique and decided to make something inspired by the film given that we've repeated the memes for so long :P

Watching "Whiplash" as part of, uh, mandated research

What it does

The drummer is situated in a VR environment replicating the famous room B16 from the movie Whiplash. Fletcher, as the band's director, will be demanding to the player, and they must follow the sheet music in front of them, or else risk being humiliated by Fletcher himself.

When the drummer plays, Fletcher determines: a) if a note was missed, and b) if a note was played "rushing" (early) or "dragging" (late). Fletcher's animations and dialogue adapt in real time depending on how poorly the drummer. If the mistakes continue, the practice ends and the drummer is presented with a window of statistics of the session to help with their learning.

How we built it

We rely on an agentic workflow, but subdivided it per team-member and per task.

Firstly, the drum trainer detects mistakes by turning the microphone feed from the headset into a live stream of notes through the use of an audio recognition ML algorithm. It then compares those notes to the sheet music's notes and calculates time differences. Fletcher toggles between "rushing" or "dragging" mode if the player drifts beyond a certain threshold away from perfect (by default, 10ms).

This data is fed live into the VR program to determine Fletcher's dialogue, which is generated through Claude and a voice generator. The VR program was made entirely in Godot. Asset design (texture maps, etc.) was done by agents, but arrangement and fine-tuning was done by the hackers.

Challenges we ran into

The live audio processing was a big challenge, but we managed to get it done. We had to source an actual drum kit, and also collect sound data to train our drum recognition model. The combination of collecting data and getting it to work while maintaining a small latency genuinely took us over half a day to nail down.

Recording drum samples

Accomplishments that we're proud of

We are incredibly proud that we were able to bring together so many parts, including the physical intricacies of a drumkit, all into one package. The VR program was also painstakingly done - as a team we absolutely loved hearing Fletcher's explosive reactions!

What we learned

In the words of one of our team members, it is often too easy to trivialize parts of the job. We thought that the drum audio recognition module would be easy, but we ended up spending half a day just fine-tuning audio and hitting drums.

It was also some of our first times working with VR in this capacity, and we learned very much in a short timespan about using Godot. We also learned how to integrate LLMs effectively into 3D development, which was very dependent on careful prompting and preparation.

Also, even our programmed rendition of Fletcher is absolutely terrifying!