-
General Conversation UI
-
Ability to Place Items In XR
Project Replicant VR: Bringing Humanity to AI
🌐 Inspiration
I (the lead of the group) was inspired by SAO: Ordinal Scale from a young age. I loved the idea of how AI and AR/VR could improve our lives across various sectors. However, as I grew older, I saw the technology stagnate, so I began developing it myself. When my old friends from school told me about Shellhacks, I figured it would be a great challenge outside of my everyday routine. Project Replicant VR is part of a greater initiative to bring AR/VR and AI to the forefront of public attention.
💻 What It Does
Project Replicant sets the stage for interacting with AI in ways that bring real value to users. In a 3D space, you can view different items such as engineering projects, media, or simply place objects in your environment for comfort. But the star of the show is ArhuaSys (Artificial Reality Humanoid Understanding and Assistance System), featuring Jane, an avatar designed to help with your needs.
Initially, Jane was meant to follow you and respond to physical instructions, but time constraints limited this functionality. Still, with a press of a button, you can speak to her and, with another click, give her access to your point of view.
🔧 How We Built It
We utilized several cutting-edge technologies to bring this project to life:
- Ollama backend: Supports high-speed chats with GPT-4 level strength on your local device.
- Whisper: Converts user speech into text.
- Piper TTS: Generates realistic voices quickly for a natural feel.
- Nous Hermes 2 Vision: Enables sending images and text to a high-speed LLM running locally.
On the XR side, we used UltimateXR (specifically uMuVR) for its extensive features over other Unity-based frameworks. The entire project was built in C#.
Links to key tools used:
- Piper TTS for Unity
- Ollama Nous Hermes 2 Vision
- Ollama Unity Integration
- UltimateXR
- Whisper Unity Integration
🚧 Challenges We Faced
Our biggest challenge? Time. We aimed for a much more advanced system, but the time constraints were our biggest limitation. The tech stack also consumed a lot of time. Gio, Dilan, Juan, and I spent the last 24 hours awake, working mainly on the backend and selecting the best libraries.
For example, we initially planned to use Emotivoice for generating voices with emotion. While it worked, the speed wasn’t fast enough to maintain immersion. Similarly, I restructured the Ollama backend code three different times before finding the ideal C# wrapper for our needs.
🏆 Accomplishments We're Proud Of
This was undoubtedly one of our most ambitious projects, and seeing it come together feels amazing! While there's room for improvement, we created a solid backend for AI in just 36 hours. We believe this system has the potential to add social value to XR, connecting people with information they might not otherwise access through traditional 2D monitors.
📖 Lessons Learned
We learned a lot about planning and restructuring our workflows. We had great ideas but struggled with execution due to inadequate planning. In the future, we'll focus on improving how we distribute workloads, allowing for more independent-driven outcomes.
🚀 What's Next for Project Replicant: Bringing Humanity to AI
I’ve been working on a pair of XR glasses for some time now. You may have seen my work on Hackaday.io:
I created an alternative to Pancake lenses and open-sourced the headset design. Since then, I’ve greatly improved my CAD skills, and with this team, we aim to develop something more polished. My next steps are to expand this project into AR, VR, and my Shell Operating System!
While ShellHacks may be over, the development of Project Replicant has only just begun.
👾 P.S.
You can check out the old Project Replicant (which this system is based on) here:
While none of the systems are compatible, this project will be open source! Feel free to join the Walker Industries Discord for updates on all my open-source work:
Log in or sign up for Devpost to join the conversation.