The True View

• 

  "The True View" logo

Inspiration

We are always presenting something. For example, when we are selling a product at work or recommending a favorite painting or hobby to others. However, even if we do our best to talk about the value and story, or use presentation slides to explain it, we often feel that we are not fully explaining the inherent value and story of the product. As we usually develop applications for Quest, we thought that if we combined Meta's Presence Platform with AI, we could create an innovative and creative tool that fully utilizes the MR space, "The True View "We started this project with the idea that we could create "The True View.

What it does

When multiple people put on Quest and enter the space created by this tool, they can see the real world as it is, so they can naturally start a conversation about what they want to talk about or about things each other. As the conversation progresses, the AI analyzes the content of the conversation and generates images and objects in the MR space that may draw out mutual interest and deepen the conversation. As they continue to talk about the objects that appear in the MR space, their mutual interest deepens, and the MR space is gradually filled with such objects. This space filled with conversation and interest can be preserved to become a continuous environment for further conversation and interest.

How we built it

This tool was built primarily using the Mixed Reality and Interactions features of Meta's Presence Platform. Conversations are recognized by the Voice SDK and translated by the AI. The content of the conversation is also analyzed by the AI, which scores the importance of words according to the context of the conversation and determines how they will appear in the MR space, for example, on a wall, falling in front of you as an image, or a 3D model appearing as an object. If the object to appear is a 3D model, it will be generated by AI or loaded from a 3D model library and appear as an object that can be held in MR space. These MR spaces are shared by the Muiti Player SDK and Shared Spatial Anchors, allowing multiple people to share the same space to further the interest of the conversation.

Challenges we ran into

Our development experience so far has been focused on VR or AR. Our challenge this time was to design an unprecedented experience created by mixed reality, where reality and information space are fully integrated by linking Meta's Presence Platform and AI. Although this was a challenging 3-day development period, we were able to share MR space using Paththrough and Shared Spatial Anchors, translate Japanese to English in real time using the Voice SDK, and analyze conversational content and generate objects using generative AI. We are now ready to take on the challenge of analyzing conversational content and generating objects using generative AI.

Accomplishments that we're proud of

In the development of this project, we felt that we were able to expand the unique communication experience that is unparalleled in the world, where information is derived one after another and fills the space from a conversation that starts in the real space. The surprise and enjoyment of seeing the object you were conversing with appear in front of your eyes through the generated AI is still a challenge due to the low accuracy of the generated AI, but with future quality improvements (such as Meta's recently announced "Meta 3D Gen"!), we are confident that we will be able to create even better experiences. ), we believe it is a great achievement to have realized a part of the potential to create further experiences.

What we learned

It was a very valuable learning experience for us to deepen our understanding of the Presence Platform SDK for Mixed Reality, which we have not had much exposure to in our daily work. We also learned that the efforts we have made in VR to connect information space and humans are very useful and continuous know-how for MR as well. However, the most important thing I learned was that designing and developing MR space with Quest is a lot of fun, and that there is still a lot of potential to greatly expand the world's communication.

What's next for The True View

Since the implementation of some of the functions we wanted to realize during these three days ended up being incomplete, we would like to improve the completeness of them first. After that, we hope to improve its functionality as an MR space and to deepen its potential to support communication generated by AI from the context of a conversation. For example, object recognition and contextual completion of the surrounding environment through image recognition of the real space captured by Quest. For example, the ability to complement the context of each speaker's utterance within a space in relation to each other. Through these, we hope to achieve our goal of creating an innovative and creative tool that allows people to share the real value and stories of things in the real world as they see them.

Built With

• c#
• mr