Inspiration

Music is not all about melody, rhythm is a core part of music: “it’s the rhythm that provides the driving force to keep our ears tuned in and satisfy our deep instinctive appreciation of music more than anything else [1].”

Our hack, Lucid Drums, was motivated by our dedication to promoting music education and early childhood development. Notable learning scientists and psychologists, such as Lev Vygotsky, believe that children learn by experiencing their environments and making observations from the world around them. Carl Orff, one of the major figures in childhood music education, promotes the idea that music education should combine with movement, playing and improvisation [2]. We believe that with VR technology, we can create a fantastic virtual reality environment that enables children to explore the world of music rhythmically and freely; through fantasy, we create a unique learning environment.

While our VR drum game was initially motivated by our passion for childhood education, we discovered drumming can also be a positive activity to help medical patients. Rhythm has been shown to help adults who are in recovery from motor impairments and drug addiction. The American Music Therapy Association (AMTA) conducted research that has shown drums to be particularly effective in helping stroke patients communicate; patterns help patients regain movement and speech ability [3]. Drumming also enhances recovery for patients overcoming drug addiction. Findings published The American Journal of Public Health highlight the benefits of percussion which include producing pleasurable experiences and enhancing awareness which decreases sensations of isolation [4].

What it does

Lucid Drums is an Oculus Quest Game in which we generate a fantastic virtual reality environment that enables children to explore music freely and rhythmically. It helps children learn different kinds of rhythms by playing drums to music in a VR world. We use machine learning to generate rhythms to songs, leverage hand-tracking to capture real-time movement, and apply game UX and learning principles to engage users.

Apart from music education, the Lucid Drums can help patients with recovery. Patients - ranging from those immobilized in hospital settings, stroke victims at home, or drug recovery patients - will have an engaging experience in the Lucid Drums which can alleviate pain and enhance recovery.

How we built it

  1. Quest + Unity for VR gaming experience
  2. Deep learning (RNN from madmom[5]) for rhythm auto-generation
  3. User/learner friendly design according to UX/learning design principles

Challenges we ran into

1. VR game modeling

The most challenging part in VR game modeling is how to track the user's hand gesture and determine whether the user is hitting on the correct beat or not without using a controller. In order to detect the user's motion quickly and precisely, we spawned a sphere in each finger, then detected each single sphere if they hit the drum. We implemented an invisible sphere over the drum to realize the beat detection. Every time the sphere on a user's finger collides with the sphere over the drum, it is be considered as a correct hit. We adjust accuracy of a hit depending on difficulty selected; in learning mode there is a larger accuracy for hits whereas in competitive mode the window of a correct hit is much smaller.

The second challenge was how to make the beat-balls - which signified accuracy hitting a drum- move according to the beats generated by RNN. We made the realtime connection between Unity and the music: 1 unit in Unity equaled 1 second in music. We then moved the balls with the music base in rhythmic time.

The other challenge was that we originally designed the game with 1 virtual drum, but we were later inspired to implement 2 drums. After rounds of user testing and research, we realized multiple drums would be better to keep both users hands occupied; with both hands playing, they are more immerse in the game. Therefore, we implemented two drums. We have a drum master that reads the music beat data and distributes it the two drums according to left and right signals. Meanwhile, both of the ball lines (for left and right hand movement) move forward consistently with music.

2. Rhythm auto-detection As humans, we can catch certain rhythms easily, but an artificial system is challenged to do so. Beat tracking may sound like a straightforward concept, but it’s actually an unsolved problem until now. We learned audio can be complex and noisy that this can befuddle an algorithm and lead it to produce false positives while detecting beats [6].

We used the Recurrent Neural Network to identify beats, but it was not giving us the desired output and predicted a lot of false positives as well. Therefore, we used neighboring methods to pick up the prediction that matched with the true beats in a song. Improved detection gave us the best/appropriate beats that nicely incorporate with each song.

3. UX Design As part of UX game design, it was challenging to create assets that were both appealing, visually consistent, and intuitive for players. Since our game targets both beginners and advanced players we had to balance having user-friendly designs for a virtual reality environment with also ensuring functionality. To guide users, we developed UI cards that allow players to select different songs and the playing difficulty and, when in-game, to easily pause, restart, and go back to the main menu. The aesthetic instructional cards integrate in the VR interface to give intuitive guidance for the game.

Another challenge for UX design was that Lucid Drum targets two different user groups: children learners and adult patients. These two groups have distinguished expectations for the game. For children, the goal is to optimize their rhythm learning experience thus yielding a better learning outcome. For adult patients, the goal is to optimize their gaming experience to entertain and help develop skills towards recovery. To make the game adapt to both children learners and adult patients, we designed two different modes: learning mode with easier rhythms to motivate the learning and competitive mode with harder beats that enable better gaming experience for patients.

Accomplishments we’re proud of

We were able to build a VR application with aesthetic features using Unity, Oculus Quest, Recurrent Neural Network and Audacity! We created a virtual environment for children as well as patients that is accessible, affordable and entertaining. As specified previously, music is an art form that satisfies everyone in one way or the other, so building this application was a creative endeavor.

Through our hack we successfully:

  1. Built the first VR beat game for a social good purpose, implementing a fantasy virtual reality environment with billboarding.
  2. Achieved hand-tracking that doesn’t require a controller. It is more authentic and yields an optimal user/learning experience!
  3. The auto-beat-generation algorithm enables the game to generate the playlist automatically. Literally you can play with any song you want! Specially mentioned: we bring the excellent drum beats from the Stanford Cardinal Calypso and the traditional Scottish Pipes and Drums from Carnegie Mellon into our games (team spirit!).
  4. We targeted different user groups through learning and competitive modes
  5. Essentially, we created a user-friendly game and we achieved that!

Ethical Considerations

While our game has implications for social good through targetting education and recovery, lots of young kids and also adult patients haven’t experienced VR games before, and there is an ethical risk that users would overindulge in the VR world which would negatively impact their connection with the real world. This can be disastrous for children’s socialization development and patients’ mental health. To prevent this from happening, we considered an anti-indulgent system that automatically reminds users to take a rest every 30 minutes. We believe this anti-indulgent system is a crucial component in VR game design and should be included in every VR game as ethical consideration for user mental health.

References:

  1. https://www.musical-u.com/learn/topic/ear-training/rhythm/
  2. https://en.wikipedia.org/wiki/Carl_Orff#Musical_works
  3. https://www.everydayhealth.com/stroke/music-therapy-for-stroke-recovery.aspx
  4. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1447805/
  5. https://github.com/CPJKU/madmom
  6. https://www.analyticsvidhya.com/blog/2018/02/audio-beat-tracking-for-music-information-retrieval/

Built With

  • deep-learning
  • figma
  • lots-of-hard-coding!
  • madmom-music-processing-package
  • oculus-quest-vr
  • recurrent-neural-network
  • unity
Share this project:

Updates