Type The Rhythm

augment unfamiliar pinky proprioception with hand tracking
borrow your laptop's keyboard
words have letter positions that match the music

Inspiration

Type The Rhythm is a rhythm typing game, inspired by Beat Saber and Keybr (the adaptive typing trainer).

What it does

It plays a song, and it cues you to type words in sync with the music. It picks words where the physical arrangement of letters on the keyboard matches the melodies in the song, so you get proprioceptive reinforcement not only from the typing but also from the music.

Keybr was a breakthrough in typing trainers by starting you off with a limited set of common letters. Once you demonstrate proficiency in those letters, it adds some more, until you are proficient at the entire alphabet. With Type The Rhythm, you have to use all 8 fingers from the very beginning, but each finger is only responsible for a single letter. You start to reinforce the proprioception of full touch typing immediately. As you progress, the vocabulary expands.

We also have a format where you can ask an LLM "here is the Type The Rhythm spec, give me a level for practicing Excel formulas and shortcuts". Then you can use Keybr's sync engine to drill Excel (or any other punctuation and shortcut-heavy typing application).

How we built it

iWSDK, WebRTC for the "keyboard borrowing" feature

Challenges we ran into

hand tracking missized https://github.com/facebook/immersive-web-sdk/issues/14
severe WebXR stutter on keypress https://github.com/facebook/immersive-web-sdk/issues/16

Accomplishments that we're proud of

We realized very late that WebXR has a complicated relationship with keyboards. So we built a quick WebRTC coordination layer so that users can "borrow" any computer's keyboard for use when playing our game on the headset.

This is great because it lets our users drill on whatever real keyboard they will be using day-to-day, and removes the requirement to get a USB or bluetooth keyboard for users who don't have one. It also works around the WebXR limitations mentioned above.

We're also really happy with how fun "typing in rhythm" turned out to be.

What we learned

iWSDK

What's next for Type The Rhythm

We're putting together a launch pack of original songs whose lyrics promote good typing ergonomics, as well as well-known holiday songs (Nutcracker suite, etc) to offer for sale as a holiday gift for the typist in your family.

We're also very busy adding more scoring mechanics (we weren't able to get all of the letter-by-letter expansion mechanics finished in time for the deadline).

We plan to launch on flatscreen first, then add VR support. The VR is way more fun though hahaha. Hopefully we can help convince our flatscreen users that they need headsets too! We think there's a big overhang in how capable these headsets are vs how much people are using them, and we think keyboard fluency within VR is a big part of the problem. If we can improve that situation, then all sorts of opportunities open up!

https://ttr-vr.pages.dev/ (we're freezing this Dec 9 @ noon) http://typetherhythm.com/ (we hope to ship the real thing here ASAP)