UnfoldVR

WebRTC - Native App diagram
One of the 3D lobbies
3D NFTs with web app Paired
Creation Tools
VR streaming demo
Creation Glove NFT Teaser

Inspiration

The original idea for UnfoldVR was originally significantly different. As a boutique VR development studio we were tired of B2B work and wanted to create our own product. We knew what people enjoyed about VR and what they didn't so we set out to create something to take full advantage of the VR technology. Fast forward a couple of years and many conceptual iterations, users and mentors feedback and we realized that the idea of the Metaverse has to be linked to decentralization and IC was the obvious target. So from CoachVR, an education VR app we landed on UnfoldVR - The Creative Metaverse. We were also quite frustrated with the general public perception of NFTs so we wanted from the get to make a product that uses NFTs to their true potential as ownership management systems and have everything in the VR app NFT driven.

What it does

We envision UnfoldVR as a key building block of the wider IC metaverse ecosystem. But we also believe that you need native apps for high computation environments, such as VR, to complement web-based solutions. The app has the following features implemented:

Link Web App with Native PC VR app
Select user Lobby Space
Configure Lobby Space with 2D and 3D NFTs
Create real-time-ready 3D worlds/items
Load user creations

Proposed future features include:

Discover other users' creations
Multi-user VR streaming
Minting 3D creations as NFTs

Our first challenge was having a native PC VR app communicate with the IC. We solved this using WebRTC technology. Users can login on our web app where they get a pairing code. After starting a local signaling server (included in our app) they can start the VR app and type in the pair code. This will link their Principal ID to their local VR app instance and in the backend we read/write and call functions between the VR app and the IC app.

We decided to demonstrate this by bringing NFTs assets from the web app to the VR app. Users manually add NFT links in the web app and those NFTs become available in an NFT Library feature inside the VR app. At this point, given the fragmentation of the NFTs ecosystem and the lack of 3D NFTs we're only loading 2D NFTs dynamically. With the approval of their creators, Poked, VRSTL and Distrikt, we've brought in 3D assets from their NFT collections inside our VR app. The objective was to research and investigate compatible formats, optimization requirements and to demonstrate, at a POC level, how interoperability should work between metaverse projects. We also brought their models in Web AR, fully on the IC.

The VR app then communicates back to the IC app which NFTs were placed in the user's lobby.

The PC VR app is built with Unreal Engine 5 and we're publicly sharing the IC WebRTC protocol with both the IC community and the Unreal community. The creation tool, lobby spaces and save-game features are all built using Unreal Engine. Currently we're storing saved game data locally but in the future that will be uploaded on the IC user account.

How we built it

We're using Motoko for everything IC related with the frontend handled in React Typescript. Besides the web frontend, the backend has a database for each user where we store relevant information about their interaction in VR. In UnrealEngine we're using their visual scripting language, Blueprints to handle both internal VR app logic and the Web App communication. For the WebRTC component, we're making use of a public plugin which is a much simplified version of the built-in Pixel Streaming plugin which comes with Unreal Engine. We've modified the JS files associated with this plugin to manage the specific functions and requests we needed for our specific use case. Currently, we're using a local signaling server, which is installed on user's PCs, based in JavaScript, also publicly available. The end goal is to move this on the IC.

Challenges we ran into

Our first challenge was learning IC. We've been watching on the sidelines how the IC ecosystem grows but we never coded anything on the IC prior to receiving our dfinity grant, shortly before the Supernova kickoff. Fortunately VR development in Unreal Engine proved to be a very similar coding environment to the IC since it's also a medium that evolved very rapidly.

One of the most difficult challenge we faced was finding a workable configuration for the WebRTC pairing. We tried many approaches and we succeeded in a few but reaching the current solution took many iterations. While it is an elegant solution, it can be improved still and it is not compatible with native Mobile VR which is our end goal, so there's a room to research further there.

An unexpected challenge was rendering 2D NFTs. We knew the ecosystem is not yet standardized but we didn't expect it to have so much variation in implementation. This issue was compounded by the fact that rendering web frames in native VR isn't as flexible as rendering them on a web app. Here too there is much room for improvements.

Accomplishments that we're proud of

By far we're most proud to have demonstrated how a native PC VR app built with Unreal Engine can communicate securely with an IC app. We're also very proud to have pieced together UnfoldVR vision, including features yet to be implemented, in such a short amount of time.

We're also extremely grateful for the openness of so many in the community, projects and individuals alike, especially with regards to figuring out a common standard for 3D content on the IC. We're committed to validating their trust.

What we learned

We've learned that IC is quite a flexible blockchain to work with. We originally expected to code in a more constrained environment with longer deploy times. We also learned a lot about WebRTC as a protocol, and we're excited about the associated opportunities of it in relation to IC. We also have a greater appreciation of the complexities of NFTs on the IC in general.

What's next for UnfoldVR

UnfoldVR is a much bigger project than the scope of this hackathon. We want to become the building block of the IC metaverse ecosystem. We aim to achieve this by leveraging our WebRTC integration system to empower many other IC projects. Also we want to give users the tools to create 3D web-friendly content to bring in other projects.
Finally we aim to empower creators and leaders to communicate their vision in an immersive, more rich way through VR Multiplayer and Video Streaming of their sessions to other VR users and to the web, using an expanded architecture of WebRTC.

All of this while creating a community around creators and creative content, where users can bring in 3D content from other projects, create 3D content to be used in other projects and discover and interact with the IC community.

At this stage we're preparing to finalize our last dfinity grant milestone all while looking for funding for launching a significantly more robust VR app.