We were inspired by the Google Cloud Platform APIs that were presented to us. This led us into researching how Google has pursued VR development and found the opportunity to create a VR player using Google VR's View that would help people on opposite ends of the globe communicate in an intimate, immersive manner in crucial points in life. This ranges from emergency services handling a situation occurring remotely to a family member welcoming a new child to the world.
What it does
Leonardo gives smartphone users the ability to experience a situation without having to physically go there. Whether it be an ill child in a hospital bed live VR streaming their friend's weekend getaway to a remote police station accumulating data about an ongoing scene to inform approaching first respondents, Leonardo puts the physical world virtually into your fingertips. This is especially helpful in life-threatening situations, such as forest fires or active shooter situations.
How we built it
After we finally obtained the proper hardware to take our application in the right direction, we began to research the development required to create a live VR viewing platform. We began taking a web development approach, including WebVR, WebRTC and WebAudio and soon realized the time necessary to create a web application would be beyond what we had for SheHacks. We took a step back and researched Googles VR View which creates VR experiences using 360 video. We created an adaptive video player that had ability to playback a local static video. After implementing this, we were able to successfully view the video using the googles and wanted to move forward creating the live platform. To do this, we researched exporting the live content from the Theta S and uploading it in real-time in order to allow the live component to take effect. The capabilities of the website were not well suited for what we desired, so we researched Real Time Messaging Protocol in order to find a video player suitable to broadcasting a VR Video. We obtained the software that works with the Theta S and found that we could use the built in Wi-Fi to directly export the video from the Open Broadcast Software (OBS) to YouTube Live. We downloaded the driver for the camera to ensure the audio and video components worked correctly and adjusted YouTube 360 and Live broadcasting settings. We inserted the stream address into the OBS so that the program could connect to the YouTube stream and adjusted the bitrate to accommodate for the VR video output.
Challenges we ran into
Leonardo was a very long learning experience with MANY obstacles. From having a Google employee bring in a pair of Daydream VR goggles to traveling out of Boston for the right 360 degree camera to produce the adequate video output to coding it all and put it together, we ran into challenges at every building block of Leonardo as we learned about VR for the first time while creating our project.
Accomplishments that we're proud of
We are extremely proud and honored to have had the help of the Google employees and mentors that allowed us to get the proper hardware to move forward with creating our software. We are ecstatic to have our VR headset and 360 Ricoh Theta S camera connected to YouTube Live and streaming its surroundings.
What we learned
We learned nearly everything we know about developing VR through this experience. From the web development phase using Google's VR View, WebRTC and Web VR to having to use RTMP to broadcast the VR video live, the experience in its entirety was a learning one. We were unaware about creating and playing VR 360 content on YouTube Live and quickly adapted to using the player in order to use the Theta S and connect the camera to the computer in order to produce a live output.
What's next for Leonardo
Leonardo has the potential to change the way people experience life from milestone moments to emergency situations. Leonardo can be further developed to be applied to personal communication, military applications, and even educational and medical training sessions.