Music should be a multi-sensory experience, but it is currently limited by our everyday two-dimensional sheet of music paper. Driven by a desire to bring music to life in our 3 dimensional world, we embarked on a mission to develop an augmented reality music development and playing app. At the same time, we understood the power of an interactive and fun multi-sensory game in healthcare, where providers want to track patient’s exercise habits for rehabilitation, and also developed a movement tracker for providers to ensure that patients are getting enough physical activity using the game.
What it does
Intarstellar is an immersive augmented reality music development game that allows players to drag notes around them, with fun musical controls such as dragging notes to make them longer. It also connects to a webapp that determines the player’s body landmarks when they play the game, and connects to a database that health providers can access to track the player’s activity.
How we built it
We utilized Photoshop, Maya, Figma, Procreate, and Unity 3D to design and draw the art, meshes, animations, etc. Unity 3D was used to develop the Augmented Reality app, and React.js was used to develop the webapp. The React app connected to the database via Firebase, and Mediapipe for js was used to determine the pose landmarks, and a custom post-processing script was used to determine activity levels of the limbs.
Challenges we ran into
Magic Leap Development Magic Leap’s SDK has a lot of unexplainable errors and confusing/inconsistent behavior. In addition to this, documentation was more sparse and less detailed than documentation for competing AR/VR headsets such as the Hololens and Oculus Quest. Rendering Pipeline Integration Shaders and effects were built in Universal Render Pipeline (URP) however when trying to merge the aesthetics with the technical implementation of the project, rendering for the project failed and we could not find a solution. Ideating a 3D musical experience Determining how things like time, volume, pitch, etc are changed by working within a spatial instrument was an interesting challenge given the abstractness of the idea.
Accomplishments that we’re proud of
The UI is absolutely beautiful. The game is really fun to tweak around with and interact with the world. The webapp collects data very easily and displays it in easy to understand graphs. We are also very proud of its applicability to healthcare, as one of the leading problems in current stroke care is the inability for providers to track patient activity after the leave the hospital, and many times patients are not motivated to do their exercises due to lack of interest. This app provides an unique and immersive experience for users that captures interest and also forces them to use their vision, hearing, and motor skills to make music. On top of that, the webapp tracks their movement during the game and can confirm with physicians that users are getting the required activity in each of their affected limbs.
What we learned
How to develop for the Magic Leap While we struggled with this greatly for the majority of the hackathon, by the end we had a pretty good understanding of how development for the Magic Leap worked and we could efficiently and effectively implement features. Designing for AR spatial applications While our team has experience designing for VR, AR is entirely different because one must consider the world around them. The opportunities and design considerations that this opens up are entirely different from VR and a space we would like to explore further.
What's next for intarstellar
We would like to expand more on the aesthetics of the project. While we developed visual assets, many of them unfortunately could not be implemented in time. In addition to this, exploring the theme of cosmic oceans more is something our team would really like to do.
The gameplay could be further made more captivating by adding additional musical functionality and instrumental sounds, such as drums. We plan to showcase this to nursing homes and hospitals for use in their therapy units.