✏️ Stu: A Friendly AR Study Companion

💡 Inspiration

Working alone is notoriously difficult for first-generation students, children of late-working parents, and individuals with ADHD who often struggle with focus and motivation. Getting stuck on homework can feel isolating. That inspired Stu, an AR study companion that brings natural conversation, emotive expressions, and dynamic visuals directly to a student's desk to make independent learning engaging and supportive.

🎧 What it does

Work: Tackle a problem at your physical desk.
Visualize: Stu spawns dynamic 3D diagrams matching your worksheet right in front of you.
Detect: Using facial detection, the app notices if you're struggling (e.g., furrowed brows, staring off).
Guide: Stu unintrusively offers help via an on-screen prompt. If accepted, Stu guides you with synchronized animations, hints, and encouraging conversation.

Key Features

🗣️ Voice-first companion: Conversational AI that provides validation and emotive gestures.
🪐 Interactive 3D AR: Dynamic, in-context diagrams that animate to aid understanding.
👀 Unintrusive support: Reads facial cues to offer gentle, opt-in guidance without interrupting.
🎬 Synchronized actions: Stu interacts with the scene and physically cheers when you get an answer right.

🔧 How we built it

Mobile AR & Vision

📱 Unity: The base engine handling the AR environment, placing Stu and educational models on the user's desk.
📷 Google MediaPipe: Processes facial detection data to identify signs of confusion or distraction, triggering the unintrusive help prompt.

Voice & AI (ElevenLabs)

🧠 Pipeline: ElevenLabs SDK (TTS, STT, LLM) → Unity Client Tools → Dynamic AR Actions.
🛠️ Client Tools: We gave the AI agent direct control over the AR scene using client tools. Stu can retrieve diagram/problem data to understand the current context, execute diagram animations, perform emotive gestures like cheering and pointing, and pass the user's spoken input into Unity scripts to drive the experience.

🚧 Challenges we ran into

Synchronizing voice timing, character actions, and 3D diagram animations into a seamless interaction.
Optimizing mobile AR performance, scaling, and precise 3D model placement for small workspaces.
Calibrating Google MediaPipe to accurately detect subtle facial cues without triggering false positives.

🏆 Accomplishments that we're proud of

Seamlessly connecting conversational AI with real-time AR interactions using ElevenLabs client tools.
Designing a respectful, unintrusive help system that reads non-verbal cues and asks permission before engaging.
Building a highly supportive educational flow centered around quality handcrafted 3D diagrams and positive reinforcement that encourages critical thinking.

🚀 What's Next

📚 Expand the question library and create more interactive 3D diagrams.
🔊 Add customizable accessibility settings and multilingual voice support.
📈 Integrate long-term learning progress tracking.
🏫 Explore privacy-conscious ways to help educators identify which topics students find most difficult.