Imagine a VR Environment where people from all over the world can interact in Real Time. Applications built on underlying technologies such as augmented reality, virtual reality, and telepresence will benefit from massive data pipes and ultra-low latency. As web and technology improve, especially with 5G, we’re pushing closer and closer to an emotional fidelity of this original experience. Just like Stadia bring developers, gamers and streamers together, our platform intends to bring Actors, Audience and Content creators together. On a global scale, the entertainment and media market was worth 1.72 trillion U.S. dollars in 2015 and is set to rise to 2.14 trillion by 2020. 5G technology is arriving and is going to revolutionize the internet with latency rates under 1ms.
The abilities of the storyteller occur only in his voice and his body - this is the highest emotional fidelity you can get and is achieved by sitting in the room having this experience. Technology has separated us from that original experience and our emotional connection is put at a distance when the storyteller is placed behind glass or projected on a screen.
What it does
Even after the advent of advanced Human-Computer Interaction, our theaters remain such that there is little interaction between audiences and actors from different parts of the world. So we built a web platform which enables content creators and actors from all around the world to post real-time action content using avatars on a custom virtual background. The audience can enjoy this immersive experience remotely with interactive devices like Magic Leap. The platform also tracks the response of the audience in real time and enables them to give live feedback to the creators.
How we built it
The Performance of the actors is captured in real time through the perception neuron suit and our platform which simulated the whole experience in VR in real time. It enables us to put an Avatar on our actor and also enables us to share this experience with other similar content creators. The platform also recognizes the response of the audiences and gives real-time feedback to the content creators maintaining the emotional connection between audience and creators.
Challenges we ran into
The main technical difficulties we faced were due to the lack of proper connectivity between the technologies we were using, namely between the Perception Neuron Software and the Magic Leap OS, and Perception neuron software and Unity. The connection between Magic leap OS and Unity was very difficult to establish because it had to be from a LAN server and not from the Unity build. It was also incredibly hard to get the Magic Leap to connect to the required LAN network on which the Perception neuron was operating, and all this required more than ten hours of hard work by us and the Magic Leap team. We discovered that there was a bug in the MIT licensing of Magic Leap. At the end, we had to move on from Unity to Web, on Helio. Then again the amount of rendering this project demands is tremendous and the browser can handle only so much render without latency at this point, and this is mostly what we’re trying to solve using 5G. The other glaring issue we faced is the emotion analysis data of the audience could not be augmented on the browser as it was too much for it to be able to render, and for now, we have to resort to showcasing them on two separate systems.
Accomplishments that we're proud of
We were able to establish connections between our platform and magic leap which was incredibly hard because it was difficult to communicate with the Lumin OS in a short period of time. This is our first project on Magic Leap and possibly the first ever project to combine the perception neuron and Magic Leap 1. Also, our emotion detection system is working very well with accurate results.
What's next for From Motion To Emotion
With the advent of 5G, our platform will become a real possibility for content creators to collaborate real-time from anywhere in the world. We want to make this prototype into a polished and scalable product. We are also interested in starting a venture by providing a limited number of features in the beginning and scale it to a large extent once 5G technology becomes stable. With low latency rates and scalable features, we can revolutionize the way that we create and watch content.