Project Poster
Project Vision
Final Project Idea
Tech Stack

inspiration

The dawn of the internet in 1990 revolutionized the way we connect. It introduced an era of hyper connectivity through emails, social media, and voice calls, shrinking the world into a global village. Gradually, people began to leverage this connectivity for remote work, a trend that only a few companies initially embraced. However, the advent of COVID-19 in 2020 was a turning point. It underscored the significance of real time communication over mere emails and texts. Consequently, platforms like Zoom, emphasizing voice and video calls, gained mainstream acceptance.

Today, the corporate world is inundated with apps for internal communication, yet their effectiveness remains questionable. The current landscape forces a choice between cumbersome Zoom calls or continuous back-and-forth messaging. This dilemma poses a critical question: How can we enhance workplace culture by facilitating office-like communication for remote workers, while also preserving the convenience of home?

As emerging technologists, we identified this gap and recognized the potential of AR/VR in transforming workplace interactions. While AR/VR is still in its infancy with limited early adopters, its capabilities are undeniably transformative. Drawing lessons from history, we understand that for technology to endure, it must be accessible and appealing to all, without compromising the experiences of early adopters. With this vision, we donned our designer and developer hats to bridge this gap and redefine workplace communication.

What it does

Revolutionize Collaboration between XR and non-XR users with Talk to Me: Boost productivity and deepen workplace connections. Embrace the future of communication today!

How we built it

We used Presence Platform building blocks including Voice SDK interacting with the Wit.ai, Interaction SDK, and passthrough. The building blocks allowed us to build the immersive AR experience rapidly. The passthrough feature allows users to use their current productivity tools while enjoying the features in the Quest3 headset.

We represented the user's colleague using an avatar and animated the avatar. When a text message is received on Slack, the message is uploaded to a Photon Server and sent to Quest 3. The avatar then runs into the user's field of view and reads the text to the user, using the Voice SDK powered by Wit.ai.

If the user wants to respond to the message, the user can simply use the controller powered by Interactive SDK to start the recording of their voice to the avatar. Then, the Voice SDK will transcribe this back to a text message and send it to the user's colleague. This process is again facilitated by the Photon Server.

Challenges we ran into

Boy oh Boy were there challenges We encountered several main challenges and countless small challenges. The biggest two are related to the integration of Photon Chat and the Wit.AI speech-to-text functionalities into our product flow. Regarding technical problems, we had to read the super obscure and obsolete javascript documentation of Photon Chat API and integrate it organically into a Slack Bot backend. We also had to figure out how to set up the speech-to-text and text-to-speech SDKs within and beneath the Meta Presence Platforms integration layer. On the other hand, in terms of product design decisions, we had to decide when and how should we determine the start of conversations, which finally resulted in this version of a flexible and fluent flow of communication.

Accomplishments that we're proud of

One of our most significant accomplishments as a team was our dedication to extensive brainstorming and research, which led us to identify a unique and impactful problem space within the market. We could have easily settled on creating another XR platform for workplace efficiency as our hackathon theme. However, we delved deeper, exploring the nuances of human interactions in the real world and how they could be authentically augmented in XR environments. Leveraging the latest technology, we successfully incorporated several advanced features found in the Meta Presence platform. But our proudest achievement lies in developing a compelling and insightful demo. This demo not only showcases the technological capabilities we've harnessed but also eloquently communicates the problem space we are addressing. Our dedication to going beyond the superficial and striving for a deeper understanding of human connectivity in virtual environments sets our project apart and defines our journey in this hackathon.

What we learned

Throughout this hackathon, our journey into the realm of XR unveiled its tremendous potential, particularly in reshaping workplace cultures and human interactions. We were fascinated by the relative ease of developing XR products using Meta's Presence Platform, which opened new vistas for innovation and creativity. A pivotal lesson for us was the critical importance of human elements within XR environments. We delved into the challenge of augmenting these virtual experiences to reduce the sense of disconnection from the real world. This exploration underscored the necessity of designing XR platforms that are not only technologically advanced but also deeply empathetic to human needs and interactions. Moreover, the hackathon was a testament to the power of collaboration and effective communication within our team. Engaging with a diverse array of mentors, each bringing a wealth of domain expertise, significantly enriched our understanding and approach. This experience has not only broadened our technical horizons but also deepened our appreciation for the multifaceted aspects of XR development.

What's next for Talk to Me

Several ideas and feature thinking occured to us as we were developing "Talk to me" Some of them include : 1) Building an SDK to connect all mainstream work communication apps on the Quest series of devices. 2) Support group chat communication 3)Trigger activation of chat using word cues 4) Use intent analysis to enrich feature