Eduverse

ducky
Duck the Duck welcomes you
Our architecture!
Stay locked in as you read your CS notes
Use the mic to ask any questions that come up
Duck the Duck will always give you quality answers

Inspiration

Traditional learning lacks interactivity, making engagement difficult. Eduverse leverages AI to create an inclusive, accessible learning experience through speech transcription, natural language processing, and voice synthesis, pushing the boundaries of AI in education.

What it does

Eduverse enhances learning with AI-powered assistance for note-taking and interactive studying:

AI Speech Transcription: Converts speech to text using Groq Whisper for seamless note capture.
AI Tutor: Uses a visual LLM Agent to analyze and explain content on the screen.
Speech Synthesis: Eleven Labs transforms text responses into natural speech. You can clone any voice! We cloned Grant Sanderson's voice: 3Blue1Brown
Interactive Canvas: Displays notes, images, and educational materials with AI-driven explanations.
Gesture-Based Controls: Scroll through content with hand gestures and clap to interact with the AI assistant.

How we built it

Developed for Meta Quest 2 integrated with ThreeJS WebVR.
Groq Whisper for real-time speech-to-text conversion.
OpenAI Visual LM for intelligent content interpretation.
Eleven Labs for AI-generated speech responses.
Interactive Canvas for real-time educational engagement.

Challenges we ran into

Our team had no prior experience developing in VR and had to start from scratch. We ended up pivoting from Unity to using ThreeJS after realizing that Unity had a lot of developmental overhead.
Real-time AI responses were hard to achieve due to long inference times with LLMs. We minimized the latency by using the Groq API and by parallelizing steps in the AI agent.
Hand gesture detection using the WebXR Device API, which was quite challenging due to its sparse and complex documentation. We overcame this by referencing community examples and by fine-tuning gesture recognition to ensure smooth and responsive interactions.

Accomplishments that we're proud of

Successfully integrated AI for speech recognition, image comprehension, and voice synthesis.
Created an interactive learning assistant that enhances engagement.
Developed gesture-based controls for a more immersive learning experience.
Designed an engaging VR-based educational assistant despite limited experience.

What we learned

AI voice technologies unlock new possibilities for interactive education.
AI-driven note-taking and comprehension improve student engagement.
Real-time speech and text processing enhance learning accessibility.
Gesture-based interactions make VR learning more intuitive and effective.

What's next for Eduverse

Advanced AI Capabilities: Enhancing AI’s ability to handle complex educational queries.
Expanded Learning Tools: Support more document formats such as spreadsheets and handwritten notes for real-time AI assistance.
Personalized AI Tutors: We'd like to adapt AI model to cater towards different learning preferences. We could support different learning styles, whether visual, auditory, or kinesthetic.
Study Tool Integration: Connecting with existing note-taking applications would improve the experience and allow for automatic tagging and organization of study materials for easier retrieval.
Improved Gestures: Expanding the set of supported gestures, allowing users to navigate, select, and interact with AI more intuitively. Examples include having a laser pointer gesture as well as an ability to highlight notes directly within the VR world.