We have assembled our team together while noticing that the ideas we have posted on the wall are both related to mental health improvement. Our entire team is interested in exploring the subject of how real time intervention through AR toolkit could help people dealing with different kinds of anxiety. We all come from the different backgrounds like: Human-centered design innovation and technology, Integrated design and media, Applied psychology, Interactive telecommunications, Psychology, Computational Neuroscience, Business Economics and Data Science. Through long discussions about our ideas (up until 6pm on the first day!!) and combination of our own experiences and the shared experiences of our friends and colleagues, we have decided to select the use case scenario of a young adult entering the job market. All members of our team are in different stages in their professional career, some had more exposure to working environement, some had none. We all have felt that debilitating sensation, that had overwhelmed us and discouraged us from sharing our ideas or opinions in the social group setting. One of the most common (and one of the most stressful) situation that had appeared in our conversation was a work meeting. It does not matter what triggers your anxiety and stress during the meeting. We will provide you the tool that will help you keep your anxiety "at bay".
What it does
BraiNgage provides a real-time, on hand access to the grounding techniques for stress management, that can be used in the exact moment when the trigger appears (meeting at work). You are accesing the experience through AR lenses (Spectacles or Magic Leap), and people around you are not aware that you are using the set. You can access the solution in your workplace before the meeting or during the meeting. The solution will also be integrated with the EDA sensor device called Bitalino, that measures your stress level through the sweat. If your stress threshold will be reached, application will start by itself. If the stress threshold will be decreased - application will shut down automatically You can choose your experience from the starting menu as well. The option to choose are various grounding, energising or distracting techniques. For the purpose of the hackathon we are going to present the user going through grounding technique called 4,7,8 in the environment with the increased stress level.
How we built it
We built our experience in Unity with the help of Magic Leap's documentation and their Unity SDKs and plugins. Some of us focused more on the storyboarding and ux sketches, and others focused on Unity implementation, including setting up Magic Leap hand-tracking, hand pose gestures, eye-tracking, world reconstruction, fluid UI menus, spatial audio, and mixed reality anxiety-reducing experiences such as visually guided breathwork, a location-aware grounding exercise, and a confident energy/affirmation voiceover.
Challenges we ran into
We have had a major challange callibrating the Arc Top device, and we had to abandon our idea to assess the person's excitement and focus metrics through the experience, as we were not able to set up EEG device. We have had zero advanced developers on our team, and the developer work has been done through the designers (who were marvelous). This challenge has turned into the opportunity to gain new skills for the designers. We had to change our idea to use the Spectacles as the go to tool (given they would be unrecognisable in the work environment set up), and we have used the Magic Leap instead of Hololenses, which was our 2nd to go tool. None of us has ever worked with Magic Leap before.
Accomplishments that we're proud of
Magic Leap integration Lucas: proud of quickly learning how to work with Magic Leap MR experience design / development in Unity for the first time, through the many hours of experiment and debugging; also proud of working fluidly with a team with diverse backgrounds and constant creative open-mindedness! Chenwei: Proud of handling AR interaction and Particle system. Collaborating with people from different background. Gloria: Proud of learning more about real-time data streaming, 3D data visualization in unity, and real-time data visualization; Ayanna: proud of attempting to learn something I have no experience in: data streaming. Joanna: went through very fast-track course to Unity, MRTK, Hololenses, and EEG - and really enjoyed it (zero experience with this before)
What we learned
Lucas: learned how to design/develop for many Magic Leap features like head-locked UI, hand-tracking, fluid hand pose UI gestures, eye-tracking, crafting mixed reality anxiety-reducing experiences, and collaboration tools for complex media pipelines Chenwei: Project management skills. AR settings in Unity. Handling C# scripts for the first time to control objects in Unity. Gloria: (hopefully) implementation of real-time EDA data streaming into Unity Ayanna: how neural networks work; Joanna: discovering the 3d data visualizations libraries available in unity and python integration to unity, EEG tracker data (although I did not manage to callibrate it, I know that this is something I want to explore after hackathon), hololenses integration with Power Bi (also for me to explore more after the hackathon)
What's next for BraiNgage
We will integrate the tool with EEG tracker to understand person reactions through out the experience. We will build up on the personalization aspect of preferable techinques, and we will introduce more of the stress management techniques into the app. We will introduce the "journal with friend" feature, that will collect the person experiences at the end of the day, through voice to text, and save it as their "journal post". This feature will be growing with the user engagement, as we will introduce the sentiment analysis over the journal entires and redesign the words content in the affirmations to the ones that are most meaningful for the person using the app.