Unity Object Manipulation
Visual Studio 2017 Unity Scripting
Unity Build-in Emulator (PS3 Controller)
App Testing Using VR Headset
Inspired from mobile and web music sequencing applications such as "Drumbit" and "GarageBand". All of us had our first experiences in producing music through these kinds of apps that allow even children, like us back then, to easily maneuver and creatively create music. We all held sentiment in music sequence apps throughout most of our lifetimes. With the introduction of new technology that can further push the boundaries of what we can experience digitally, we wanted to build a product that we were fond of and passionate in and make the application to the mixed reality, where we can experience a completely different immersion and interaction that is easy-to-use and ultimately enjoyable.
What it does
With the use of the Microsoft HoloLens, the user will be able to interact with the program in order to make music loops using different pre-set and custom sounds. The selection of any button is done through the Hololens "gazing" mechanism where a cursor follows the direction of the user's head. When the cursor is on top of a button, the button will highlight itself to confirm that that button is mutable. With the HoloLens default click functionality, which is a simple click hand gesture, the user can click on the buttons to manipulate the sounds, note, beats, play/pause, clearer, etc.. After completing a music piece, we are hoping to create a save/load file system for storing and retrieving the soundboard button's specifications.
How we built it
We built this application using Unity as the game engine and Visual Studio 2017 to write the scripts in C#. Because none of us had notable experiences in either Unity or C# it was a deep learning experience for everyone on the team. We would eventually split into different roles to utilize our unique expertise and work efficiently to finish a viable, working program.
Challenges we ran into
Initially we had an idea that was hard to implement since the HoloLens have trouble tracking fast moving objects. We also had challenges with setting up Unity and utilizing the virtual environment to create our desired objects. Since none of us had a great experience with Unity, it was helpful to have experts from Microsoft guide us to our goals. Coding using the C# and the Mixed Reality Toolkit was also another challenge. Although the Toolkit was a very helpful guide, like all programming projects, debugging took quite some time. The built-in emulator in the Unity Editor only worked with a "game controller". Because of this fact, we spent a long time figuring out how where to get one, how to connect it, and how to use it. The entire Unity experience was filled with obstacles, but they were all solved through team-work and persistence in the end, and that was one of the best parts from the Hackathon. Additionally, time was another constraint. Since we had only 36 hours to complete our project, we needed to balance sleep, work, and troubleshooting.
Accomplishments that we're proud of
We are proud of making a working product. Although we were not successful in implementing all our desired ideas, we made great progress towards a final application. Considering that we started with nothing and had little knowledge of what we had to do in terms of reaching our goal. We are also proud of creating this minimum viable product with our given time restraint and challenges.
What we learned
We learned a variety of skills related to creating and programming in Unity, C#, and SolidWorks. We learned about the design process involved in creating a product. We started with an idea, then design, then build, then troubleshooting, then finally a mostly complete product. Researching and looking for solutions to our problems was also a major skill we acquired this weekend.
What's next for Music Sequence MR (MSMR)
The Music Sequencer Mixed Reality (MSMR) is not fully complete. We would like to see a working sampler be a part of this project in the future. We would also like to implement musical bars that are synchronized to the rhythm of the music. Trackers to identify the beats that the application play in real time. In the future we would like to add more instruments and notes as well.