ROCK HAND

A Rhythmic Game Using American Sign Language to Make & Play Music

Comment 1

⚒ What it does ⚒

ROCK HAND is an educational rhythmic and song generation game that lets players learn and practice American Sign Language (ASL). Players can select JAM to create a song using their instrument of choice. With finger tracking, JAM maps letter gestures to notes that, when combined, create a song. In BAND, players practice 4 different signs at a time (ex. A B C D) that map to the notes in a song. Their score is based on getting the sign correct at the correct time as well as how many they get correctly in a row.

✨ Inspiration ✨

When brainstorming impactful problems to solve, one of our teammates brought up the idea of fidget cubes and making a game to help people who fidget but without using any hardware.

Fidgeting can be an outlet for people with ADHD and Autism. Fidgeting tools are a self-regulation mechanism for regulating lower attention levels.

That brought us to think about how someone can fidget with just their hands.

We went off of that idea as well as another one we had about learning languages and came up with Rock Hand – a game where users can create music using ASL as well as play rhythmic games from their created songs. Worldwide, 70 million people use ASL to communicate. So, we wondered if we could combine both of these ideas to create a game that is accessible for people who know ASL or who want to learn it as well as people who fidget.

Through our Rock Band/Guitar Hero themed game, we hope to:

  1. Help people who tend to fidget do so in a mentally stimulating way
  2. Lower the barrier of entry for people who want to learn ASL
  3. Allow people who communicate through ASL to create music

👷‍♂️ How we built it 👷‍♀️

MACHINE LEARNING We started with training a model on the Sign Language MNIST dataset using PyTorch and connecting it to a camera using OpenCV so that we can detect what letters the player is signing

BACKEND Python w/ Flask and WebRTC (aiohttp & aiortc) to be able to load a camera stream in from the browser, convert it for OpenCV and then apply the PyTorch classifier to determine the sign. The WebRTC server streams back the original video with the ML overlay + a data channel containing game data (current letter, score, etc.)

FRONTEND we used React (javascript) as well as Three.js to create the BAND component of our game

RockHand Architecture

💪 Challenges we ran into 💪

For this hackathon, we took on a project with a complicated user flow as well as learned many technologies we had never used before. We spent a lot of time trying to improve our ML model’s accuracy by changing the way we preprocess our data as well as trying out two different data sets. It is still difficult to track certain letters because the data set we used requires a clean background as well as the player’s hand being centred in the boxed frame. We also struggled to connect the frontend and backend components because of limited WebRTC and React knowledge. Also once we got the camera onto the front end of our app, it was really laggy so that is something that we need to work on improving. Overall, it's never easy to develop a whole app in 24 hours, but we learned a lot about rapid development and teamwork.

🥇 Accomplishments that we're proud of 🥇

We are super proud that we were able to bring this project to life! We spent 3 hours on Friday going over dozens of different ideas and we finally landed on one that we were all excited to make because of how unique and challenging it was for all of us.

Also we were able to make a full stack app with real ML which is pretty cool!

🚸 What we learned 🚸

Each member of our team learned at least one new technology. This project really taught us a lot about project management and how to split up tasks effectively 🔥

😲 What's next for ROCK HAND 😲

We want to be able to add the ability for users to save their songs from JAM and use them in BAND. In JAM we also want users to be able to switch between different instruments and also loop over specific sounds. In BAND, we want to be able to store the user’s score to keep track of high scores between friends.

Share this project:

Updates