Ripppple (Audio Spectrum Visualizer) - Team 13

Fusing together listening to music and data visualization, our project uses an audio spectrum analyzer to display various forms of visual effects within the user’s point of view.HoloLens2+MRTK+Unity

Comment

Inspiration

Enhance the user's listening experience by providing a welcoming environment with spatial sound and visual effects. Historical music visualization techniques include Hi-Fi stereo systems (circa 1976) and mp3 wave spectrum analysis such as WinAMP (circa 1997) or MilkDrop (circa 2001).

What it does

Rippple provides a visually tangible and engaging experience by using the HoloLens 2 for 3d music visualization, fused between the audio wave spectrum and the environment.

How we built it

Each team member was interested in working on music visualization for HoloLens 2, using Unity and MRTK. All of the skills and experience of the team members enabled fusion development, overcoming complex challenges with hardware, software and tools. Mentor and community support enabled a wavy path to our final solution, harnessing the power of these 3 technologies in just 2.5 days!!

Tools Visual Studio, Windows SDK (UWP/MR), Unity, MRTK, Shader Graph, URP, VS Code, Figma, GitHub Desktop, Poly.pizza

Microsoft Mixed Reality Toolkit (MRTK v2.7.3) • Dynamic Scene Understanding • Pressable Button, Slider, Hand Menu, Interactable 3d Objects

Challenges we ran into

Our plan included using the above technologies and tools to quickly build an immersive experience, but several unexpected problems caused us to have to learn, re-think and react to the issues quickly. Our team demonstrated perseverance and tactical cooperation, despite all of the challenges (things we learned) when adopting these new technologies.

Challenges

  • One of our team members (SME/Industry expert for music visualization technology) could not participate due to impending obligations at work. The team was relentless in attaining the necessary knowledge and experience to follow through with the original vision
  • Setting up the environment: we changed from 2 Macbooks to 2 Windows (Razer) laptops in order to support the necessary tools (Unity/HoloLens 2)
  • Getting everyone on the same version of Unity, and then changing versions at the end of the 1st day to resolve compatibility issues with MRTK
  • Integrating UX designs into the project from MRTK (Examples) and Figma was very time consuming, with at least 1 person working constantly in Unity to design, build, test and iterate
  • The Spatial Understanding capabilities for HoloLens 2 and MRTK enabled us to build the environmental effects we were imagining; however, customizing the examples to fit our needs required more time and experience than we originally expected. Learning about this technology helped all of us to learn more about the architecture and best practices of ALL technologies involved - Microsoft provides great documentation and the open-source Mixed Reality Toolkit for Unity is a great reference to learn from!
  • Keeping up with all of the documentation, links, versions and setup steps was a moving target - as we learned new things throughout the entire hackathon, our team made incremental progress by keeping track of Best Practices, links and technology-specific knowledge and online inspiration
  • Staying focused on the design/solution while being presented with various ways to improve or refactor our approach. Our team purposely did not add capabilities like networking, streaming or personalization in order to reduce the risk of introducing complexity. Surprisingly, many other teams did adopt these features in their design, and we are inspired by the accelerated nature of this industry!

Accomplishments that we're proud of

We had several breakthrough moments when our team got to experience their ideas coming to life. Whether it was connecting multiple ideas into a working solution or providing the help needed for our team members to succeed, we learned more as a team and with the amazing mentors than we thought was possible!

  1. Audio spectrum analysis in Unity was adapted as a texture using a custom pixel shader
  2. All team members learned how to create mixed reality applications using Unity, MRTK and HoloLens 2
  3. Designing the user experience around a concept which our team all could visualize - in the end we each experienced the manifestation of our vision in some way

What we learned

How to ideate, design, develop and test mixed reality applications. Thank you MIT, Microsoft, Unity and all the hackers and mentors who helped us to become #MRDevs4Life!

  1. Having the right hardware early in the process was more efficient
  2. Choosing the right architecture and technology can change the features and capabilities which your team can work on - planning, prototyping and adjusting for change are a critical aspect to building MR experiences
  3. Team alignment and leadership needs to happen early on - encouraging everyone to have an open mind about change and keeping a spirit of learning and empowerment
  4. Using Unity prefabs and packages can allow for parallel work between developers and designers - our team was able to build and test the UI experience (hand menu, welcome scene, interactivity) in a separate Unity scene, then we documented the steps for applying the package/prefab to a scene and used that for the final integration between each component

What's next for Rippple (Audio Spectrum Visualizer) - Team 13

Share this project:

Updates