Music is a multi-sensory experience, utilizing senses of sight, hearing, and touch (proximity) to render the world around us.
However, abstracting music notes and rhythms are currently confined to the limits of 2Dimensional paper music and volume graphs depicting the pitch changes within a song.
Driven by a desire to bring multi-sensory experiences back to sound in a digital form, we created a spatial augmented reality instrument that allows a user to create visual audio experiences.
Our collective instrument playing experiences (clarinet, alto saxophone, guitar, piano), love for music games (Beat Saber, My Singing Monsters, Smule, Piano Tiles) naturally alchemized us to this idea. At the same time, we saw the potential in a interactive multi-sensory game as a powerful habit-tracking app that has direct applications in:
1. Healthcare: tracking patient’s exercise habits for rehabilitation, and movement for providers would be a powerful tool to ensure that that patients are getting enough physical activity using the game.
2. Creativity: users are able to easily exercise their creativity in the project through mixing their own music, and Intarstellar shatters the intimidation to get started with creating.
3. Entertainment: it's fun to learn to play a new instrument, and Intarstellar is exactly that, but allows you to do it in an extra dimension, opening up infinitely new music creation possibilities.
What it does
Intarstellar is an immersive augmented reality music development game that allows players to create music spatially by dragging notes around them. The experience makes use of the note position (relative height and distance from you) to determine the pitch and volume of the generated note. For example, a farther note would be softer, while a closer note would be louder, and a note higher up would be higher pitched than a note closer to the ground. The musical controls unlock a new world of music creation possibilities within the 3D space.
In addition, the game is paired with a web-app that determines the player’s body landmarks, and connects to a database that health providers can access to track the player’s activity. The dual functionality allows Intarstellar to be extended beyond games and in tracking progress towards a set habit or goal directly applicable in a healthcare setting.
How we built it
We utilized Photoshop, Maya, Figma, Procreate, and Unity 3D to design and draw the art, meshes, animations, etc. Unity 3D was used to develop the Augmented Reality app, and React.js was used to develop the webapp. The React app connected to the database via Firebase, and Mediapipe for js was used to determine the pose landmarks, and a custom post-processing script was used to determine activity levels of the limbs.
Challenges we ran into
Magic Leap Development
Magic Leap’s SDK has a lot of unexplainable errors and confusing/inconsistent behavior. This led to difficulties in setting up and in debugging. In addition to this, documentation for the Magic Leap is more sparse and supported for as compared to competing AR/VR headsets such as the Hololens and Oculus Quest.
Rendering Pipeline Integration
Shaders and effects were built in Universal Render Pipeline (URP). However, when trying to merge the aesthetics with the technical implementation of the project, rendering for the project failed and we could not find a solution.
Ideating a 3D musical experience
Determining how things like time, volume, pitch, etc are changed by working within a spatial instrument was an interesting challenge given the abstractness of the idea. We used various methods of prototyping in the 3D environment (Tilt Brush for sketching in 3D, Figma for laying out UI elements) and envisioning how space could affect sound.
Accomplishments that we’re proud of
1. The UI is absolutely beautiful. It represents sheet music on a music stand, in which "notes" are abstracted as dots that can be expanded when zooming into a specific part of a composed song.
2. The game is really fun to tweak around with and interact with the world. Previous music creating platforms are an abstract of real instruments in the world, and it was an interesting challenge to try our hand at creating our very own instrument in AR.
3. The webapp for collecting and displaying playing data is really accessible. The web functionality allows any user with a webcam to track data without a need for extra downloads. It easily tracks their movement during the game to allow physicians to use the information to prescribe or monitor exercises in the game that target usage of affected limbs.
4. Direct applicability of the game to healthcare: Current stroke care is currently one of the leading problems because providers find it difficult to track patient activity after they leave the hospital. It is difficult to motivate patients to do their exercises due to lack of interest. This app provides a unique and immersive experience for users that captures interest and also forces them to use their vision, hearing, and motor skills to make music.
What we learned
1. How to develop for the Magic Leap While we struggled with this greatly for the majority of the hackathon, by the end we had a pretty good understanding of how development for the Magic Leap worked and we could efficiently and effectively implement features.
2. Designing for AR spatial applications While our team has experience designing for VR, AR is entirely different because one must consider the world around them. The opportunities and design considerations that this opens up are entirely different from VR and a space we would like to explore further.
3. In-person hackathons are an amazing experience Thank you to the Pinnacle organizers who spend so many countless hours making this event possible. Our team members have really limited experience for in-person hackathons (only 1/4 of us have been to one), and it was a really great experience getting to check out the working progress of other projects.
What's next for intarstellar
1. We would like to expand more on the aesthetics of the project. While we developed visual assets, many of them unfortunately could not be implemented in time. In addition to this, continuing to expand and explore upon the theme of cosmic oceans, and draw, model, and animate more space-themed sea creatures that would enhance the experience of creating notes in space.
2. The gameplay could be extended upon. It could be expanded further by adding additional musical functionality and instrumental sounds such as drums.
3. Testing for usage in nursing homes and hospital units. We would like to do some testing of the experience alongside tracking the data to determine benefits and drawbacks to the current platform and iterate on changes (types of graphs displayed, ease of use, etc)