M.E.R.A. (Meta Emulated Reality AI)

Omniverse processing
VR output

Inspiration

We wanted to build a project in VR, so we carried a MetaQuest 3 for YHacks. During our 18+ hours long drive from Atlanta to New Haven, we brainstormed several ideas and built on top of each other's ideas and ended up with this project idea.

What it does

M.E.R.A provides immersive conversational experience with 3d models that replicate the speaking style and mannerisms of historical figures. Currently, we've built a single model that replicates Steve Jobs.

How we built it

We've utilized Azure's Speech Recognition tool to capture user's audio and convert it into text, which is then passed onto our customized GPT 3.5 Turbo Model that replicates the speaking style of Steve Jobs. The textual response is then passed onto ElevenLabs to generate audio bytes through the custom-trained voice that replicates the accent of Steve Jobs. We've also designed a custom Steve Jobs model utilizing MetaHuman extension within the Unreal Engine and then established a pipeline connecting the Audio2Face with Unreal Engine, allowing us to lipsync the voice of 3d model. Finally, we've utilized MetaQuest3 to display the rendered final product.

Challenges we ran into

We faced significant challenges in rendering the MetaHuman character onto the MetaQuest3 due to the significant processing power involved in the process. The technical difficulties we faced:

Building a pipeline to link Audio2Face with Unreal Engine Optimizing the runtime of the application Casting the MetaHuman Character onto the MetaQuest with maximum quality

Accomplishments that we're proud of

We're proud of being able to finish the product on time, given the technical complexity of the project.

What we learned

We were able to explore 3d modeling and rendering as well as learned the importance of learning through the way.

What's next for M.E.R.A. (Meta Emulated Reality AI)

We believe that this project has a significant upside, allowing us to add multiple figures and improve our historical catalog. With improved processing and enough time, we’ll be able to create significantly improved models, offering the highest render quality and optimal speed. Aside from educational institutes, this project can be used in museums to provide immersive experience to its visitors.