F8 Opening Night Happy Hour Photo Booth Functionality.
Our 360 World Facebook Page
Desktop - Content Library
Mobile - VR Content List
Mobile - Onboarding Preference Selection
Mobile - Login
Thinking about the 4 Features SDGs
SDG 9 - Creating an Platform to support multiple SDGs.
A little image saturation!
Standup meeting time
Our team visionary David, has spent the last two years of his life granting children in hospital wishes. For the kids who aren’t able to step outside of the hospital, he brings them the outside world with the gift of VR. This brings many people great happiness as it empowers the powerless with the joy of the most simple of life’s experiences.
As a wish granter and a content creator he often finds it hard to scale his wishes, because of the lack of a platform to source, share and curate immersive and rich content. He’d also like to share this content without the bounds of language translation and to create an easier UI for VR users who don’t have the physical ability to hold the VR wand or to see a menu.
The hope and vision of the team was to build a platform for David to help with the health and wellness of not only the children he is helping in Toronto, Canada but for children everywhere. He wants to be able to create a curriculum to “Train the Trainers” so that people anywhere can request, upload and share VR content that can be localized for a memorable immersive experience.
In building out the specs for the platform, it became apparent after receiving feedback from the many subject manner experts and mentors is that what we were building is beyond an education tool, and really a platform to enable deployment of an immersive, accessible curriculum for multiple Sustainability Development Goals outside of education. This is why we think we’re directly addressing SDG 9 Industry Innovation & Infrastructure. One of the key opportunities listed under this SDG is to “Impact and expand upon the achievement of other SDGs (e.g. Job Creation, Better Health, Better Education)”, and we feel that our platform does exactly that. This is how we came to the name of our project, Our 360 World.
Our hope is that not only can Our 360 World affect kids and patients in hospitals, but also refugees, natural disaster crisis sufferers, and enterprising kids who want to learn the skills that are most relevant to them, and taught in an immersive yet localized format.
What it does
Our 360 World leverages many existing Facebook technologies to provide a robust and simple platform for users to consume both curated and discovered VR content. Upon onboarding, a user signs up on the web app or a mobile app using Facebook OAuth to create an account with a simple onboarding. Upon connecting, the app looks through the groups (and pages coming soon) they belong to which feature VR content. The content is indexed and tagged using image recognition to better understand user preferences. After selecting the content groups and categories which are most interesting to them, they can start their VR journey with a non-stop feed of live and video-on-demand content. The app detects the user’s language by their voice and translates the video content into their native language while also providing subtitles for the hearing impaired.
We bucket our users into 3 main categories:
- Consumers - These are the end beneficiaries of who will be consuming the VR content. It could be children in hospitals, persons with disabilities, victims of crises, kids in schools, adults learning new skills.
- Creators - these are the people who are creating 360 VR content to be consumed on the platform. Examples include teachers, caretakers, architects, enterprising young artists, trainers, and caretakers.
- Curators - these are people who may create an layer of organization or curation over VR content to provide more meaning, context or certification. These would likely be partner organizations who want to create curriculums to complete a series of VR experience and gauge the retention and success. Examples of these stakeholders are schools, cities, disaster relief orgs, hospitals, therapists, and private companies.
How we built it
We have a team of 9 who quickly fell into natural roles. Many of which fell into familiar roles, and some fell into roles which challenged them with solving new problems. The product team worked together to provide a rough overview of functionality with some light wireframes. The dev team did an architectural break down of the platforms we’d have to use and the resources we’d need to build it out. They identified what we could agree was our MVP, and we identified a number of stretch goals. We had a good idea of the technologies we wanted to use, but not all of them worked out. We had a great split of work committed to the back-end, front-end, and special projects (like AI & language processing). We got on a roughly 3-5 hour sprint schedule and had regular check-ins with the entire team to talk about successes, address blockers, prioritize features and to re-allocate resources in order to move on. We used almost the entire ~46 hours of the hackathon to continue to build.
The technology it's built with:
- Graph API
- Facebook Account Kit
- AWS Recognition
- AWS Amplify
- MS Azure
Challenges we ran into
We had a number of technical challenges in which we had to seek out mentorship from the supporting Facebook developer team. There were a lot of technologies we explored and wanted to utilize but felt that we had to pivot. The technologies we looked into but didn’t use were PyTorch, React360, AWS Sagemaker, Messenger, Wit.ai. Aside from the usual technical challenges, we also faced challenges with app permissions. Understanding that we can't get certain permissions for public page content and group content approved in a short timeline, we were limited to just pull content from groups we created and are the administrators of.
Accomplishments that we're proud of
- A working MVP with some stretch goals completed
- Robust API built on bleeding edge new GraphQL
- Playing cross-platform, accessible VR content from Facebook in Oculus
- Language detection, transcription, translation, text-to-speech
- Voice controlled UI
- Content detection within VR videos
- No meltdowns, no steamrolling, no egos
- Operating at 239% utilization
- Amazing teamwork, positive attitudes, commitment, great listening and bonds to last a lifetime
- Sense of pride for the entire team's contributions.
- This was a very special hackathon. The talent was great. People were amazing. Thanks for bringing all of these people from around the globe for us to work on such a powerful mission.
What we learned
We learned an amazing amount of things on both the technical side from our Facebook & technical mentors as well as the overall feasibility and market need from our Sustainability Design Group mentors. The amount of knowledge gained by our tech team from our Facebook mentors is too vast to collect here. But I can assure you that they are all leaving the hackathon with great new skills and an amazing level of confidence. We can’t thank the Facebook team enough for sitting down, rolling up their sleeves and digging into the code no matter what the time.
From understanding the state of the world, and to how to better understand the capabilities and limitations of our product we learning a number of things in talking with the mentors, and we certainly found some common threads.
- Lack of infrastructure in the developing world and post-crisis. Don’t expect VR headsets can get deployed everywhere, or that there is any guarantee of connectivity or even power.
- Train the trainer and secondarily, incentivize the trainer. This helps to localize messages and content for those who are culturally far different than what we know as the world we live in. People from other cultures will respond better to content product or curated by people who are more like them. If you can find a way to incentivize the trainers, you can get a network of people who are not only creating content, but also producing or localizing it and teaching others how to consume it.
- Lack of a single source of immersive VR content for anyone anywhere. While YouTube may enjoy an advantage as a ‘discovery-first’ platform for video content, and Twitch for live esports and gaming content, Facebook with Oculus is the clear frontrunner to be the go-to content aggregator for VR content. While the discovery of VR content on Facebook isn’t as natural as those we have talked to would like it, we are hoping to encourage the contribution of VR content to Facebook, while being able to curate consumption and discovery of it to users everywhere.
- Focus in one area before we try to boil the ocean. Focusing on supporting an existing partner like Wishplay in the category of health & wellness with education as a natural next step.
SDG use cases
As an infrastructure project, we feel that we can support the goals of a number of the SDGs including the 4 focused on in this hackathon as well as a number of others.
9. Industry Innovation & Infrastructure
- OPPORTUNITY: “Impact and expand upon the achievement of other SDGs (e.g. Job Creation, Better Health, Better Education)”
4. Quality Education
- OPPORTUNITY: “Transforming teaching and leadership and Applications to promote and enhance positive home learning environments
8. Decent Work & Economic Growth
- OPPORTUNITY: “Substantially reduce the proportion of youth not in employment, education or training and devise and implement policies to promote sustainable tourism that creates jobs and promotes local culture and products”
11. Sustainable Cities & Communities
- OPPORTUNITY: “Platforms that support increased participation of citizens leveraging new emerging digital technologies”
What's next for Our 360 World
After an amazing hackathon working together, the team wishes to keep the project moving along, understanding this truly could be a v0.1 for a truly functional product. Feedback from mentors was encouraging to say that there is a future and a market for this product.
We will continue on to build out our website and try to gauge interest from third parties across many different industries who are interested to partner up to try the platform. If interested and traction are encouraging, we will incorporate and make the fundraising rounds. We have come to understand that it will likely be important for us to focus on a lane rather than trying to accommodate many industries and use cases at once. So it’s most likely to first address Health & Wellness or Education.
While we feel that we accomplished an amazing amount of functionality, we have a long list of stretch goals we’d like to accomplish. We feel we already have a road map to sustain another year of development and improvement.
- Facebook app approval for additional Facebook Graph API scopes
- Support for 360 images, 3D images
- Support for VR content from Facebook Pages
- Support for traditional 2D videos
- Ability to create playlists
- Filters and advanced search
- Expand supported languages
- Content recommendation engine improvements
- Simple and Free Cardboard Viewer Kits
- Live surrogate mode - ability to control a proxy human with live streaming 360 camera for real-time live experiences
- Live surrogate mode with co-viewing
- White label UI and playlists for curated content by NGOs, Govt’s, partners, etc.
- Education Modules
- Quizzing functionality
- Gamification for completing
- Monetization Modules
- Allow content creators to license their content. This is intended to help underprivileged kids from developing nations and inner cities to create content on the platform.
- Disaster Relief Training
- Create a curriculum for common natural disasters with localized content addressed on how best to survive.
- Would like to sideload this content onto cheap smartphones with cardboard views to drop with disaster relief orgs.
- Cities & Communities
- Live Town Hall virtual engagement streaming VR live for humans who can’t make it or fit into a town hall
- Live Town Hall polling or feedback mechanism to gauge support