Auditor.io: Unleash Your Public Speaking Powers in VR

Inspiration

Public speaking is considered to be the single most common phobia, according to numerous studies. To tackle this fear, we believe that continuous practice and real-time feedback to the speaker are efficient ways to improve their skills. We were inspired by the limitless applications of virtual reality and wanted to design a project intended to deliver a social good to users regardless of their experience or background. Applying the principles mentioned in the best-selling book "Talk like TED" (e.g., be authentic, less filler words, dramatic pauses, and more gestures) to our algorithms, we ultimately hope that our application would empower many with the ability to communicate the ideas that matter most to them, the skill to win over hearts, and the confidence to deliver the talk of their lives.

What it does

Our virtual reality application immerses the users in a 360 environment of real public speaking scenarios (e.g., auditorium, lecture hall, interview) that we captured with the GoPro 360 Fusion. As the user speaks, our app provides real-time feedback on speech patterns based on the selected audience and or environment settings. Leveraging the Google Cloud Platform API, we can analyze tone and speech patterns to evaluate performance. Suggestions pop up in the VR interface to provide helpful advice to users in real time.

How we built it

We built the front-end using Unity (a cross-platform game engine) along with the HP Mixed Reality and Oculus Rift VR headsets. We built the backend using Google Cloud API to convert speech to text data, a WebSocket communication protocol and the Python programming language for our algorithm. We captured the training videos using a GoPro fusion 360.

Challenges we ran into

None of us had experience developing virtual reality application prior to the hackathon so this project was a steep learning curve. We spent most of the time on Saturday going over tutorials and asking mentors for help...It was great to found a team that does not focus much on winning but share a common goal of learning. From pair programming, helping each other out, to late night snack run, we strengthen our bond and friendship. We're proud of what we made.

Accomplishments that we're proud of

We are proud that throughout the span of two days, we developed an app that integrated many moving parts, such as the Unity engine, the HP Mixed reality headsets, and speech conversion and analysis APIs. We developed a unique app that is relevant to users of all backgrounds and levels of experience; designed a simple interface that helps users continuously learn how to communicate effectively while using the app. We were also proud of learning how to use many cutting-edge technologies involving virtual reality and speech analysis, and we were proud of the fact that our finished app would have a large impact on users and deliver an effective way to improve their public speaking skills.

What we learned

We learned how to integrate many moving parts with each other, such as the Unity engine, the Google Cloud APIs, and the HP Mixed Reality VR headset. We learned how to process audio and speech content using python libraries and APIs, and we learned how to fix bugs relating to VR integration so that we will build even cooler projects in the future.

What's next for Unleash Your Public Speaking Superpowers in VR

Some future features for the Public Speaking VR app may include storing and analyzing a history of data for each user so that the user can see trends in their app sessions and see if they are improving or not. This data can also help us determine if our in-app suggestions are useful or not and help us provide more relevant and effective feedback. We also want to design some variations in the app interface and possibly include different environments and scenarios to present different levels of challenges to users. In the future, we also plan on implementing functionality to not only analyze the user's speech patterns but also their body gestures (using the Leap Motion or Microsoft Kinect) in order to improve how they present themselves in front of a crowd. We had planned to refine our personalization system using Partially observable Markov decision process (POMDP) but ran out of time!

Built With

• google-cloud
• hp-mixed-reality
• python
• unity
• virtual-reality