[T28] - Habibi

Bridge the gap with our mixed reality app. Share notes, images, and objects on a virtual wall with loved ones—powered by AI and stylus. For lovers, friends, and family in long-distance relationships.

Comment

Inspiration

With multiple team members living away from home and family or in long-distance relationships, the concept of struggling to communicate from afar is well known. We wanted to build a tool that would help bridge the distance and create a sense of close interaction.

What it does

In a mixed-reality environment, users can place objects, images or text onto a virtual wall. These elements are shared in real time with the intended recipient, allowing both parties to interact with the same virtual space. To create these elements the tool leverages Wit.ai and Chat-GPT. Wit.ai converts spoken words into text, while GPT and DALL-E generate corresponding images or text based on the input, fostering a more dynamic and personalized interaction. Users also can use a Logitech Pen, to place objects or text onto a virtual wall. The Logitech Pen allows for drawing features, enabling users to create and attach images or objects directly onto the wall.

How we built it

  • We made use of the Meta Presence platform building blocks to use passthrough, interactable object, controller tracking, button mapping and movement.
  • Meta MR Motif package was used to transition between passthrough and Virtual Reality.
  • Meta voice SDK and wit.ai allow the use of speech-to-text prompts to generate both images (via Dall-E API) and text responses (via ChatGPT API).
  • We leveraged the Mixed Reality Utility Kit (MRUK) to reference the Key wall (or next largest wall) in the scanned environment to use as the Shared canvas anchor.
  • We implemented the MX ink Logitech pen alongside the corresponding meta-building block to facilitate both 2D and 3D drawing on the shared canvas.
  • Throughout the process, we also made use of GitHub and SourceTree for project source control.

Challenges we ran into

One of the main challenges we faced was transferring recognized speech to GPT efficiently. If the environment was even slightly noisy, the speech recognition became unreliable, causing disruptions. To mitigate this, we implemented a "waking word" system, ensuring that the tool only activates when a specific word is heard. Another challenge was working with the Logitech Pen. It was difficult to balance its functionality as both a drawing tool and a controller. Additionally, achieving smooth interactions, like precise touches using the pen, proved to be a significant hurdle.

Accomplishments that we're proud of

We’re proud to have built a system that allows seamless communication inside a VR environment using AI and voice recognition. The ability to generate text and images through AI and convert them into prefabs that can be easily grabbed and placed anywhere using a controller is a standout feature. Additionally, we implemented room-scale measurement to identify the largest wall surface and automatically generate a board on it. We also made it possible to freely change the colour and images on the wallboard, providing a highly customizable experience.

What we learned

Throughout the development of this project, we learned the importance of optimizing voice recognition in noisy environments, which led us to implement a waking word system for more reliable interactions. We also realized the challenges of integrating multiple input devices, like the Logitech Pen, and the complexity involved in making it function seamlessly as both a drawing tool and a controller.

Furthermore, working with room-scale measurement and creating dynamic, customizable wallboards taught us valuable lessons in spatial awareness and user experience design within a VR environment. These experiences have deepened our understanding of how to make virtual interactions feel natural and intuitive.

What's next for Habibi

Looking ahead, we will implement either Photo Fusion or Unity Net Code to facilitate the networking of the shared canvas and objects.

Also, we plan to enhance the interaction capabilities within the mixed reality environment, making communication feel even more immersive and intuitive. We aim to refine the voice recognition system further, possibly adding support for multiple languages and accents to make the tool more accessible.

Additionally, we plan to improve the precision and responsiveness of the Logitech Pen, allowing for smoother drawing and control. Expanding the customization options for wall boards, such as integrating 3D objects and animations, is also on our roadmap. Ultimately, we hope to make Habibi a go-to platform for staying connected with loved ones, regardless of the distance.

Built With

Share this project:

Updates