AURA (Adaptive Unbiased Reality Agents)

AURA is an unbiased XR speaking and presentation app for communication disorders. We move beyond static drills, using adaptive AI 'role-players' to help our users practice real, dynamic conversations.

Comment

Inspiration

We were inspired by a fundamental problem that affects millions of people: How do you practice for a high-stakes, judgemental conversation without a high-stakes, judgemental audience?

For the 1 in 12 individuals with a communication disorder (like social anxiety, stuttering, or aphasia), the fear of judgement is the single greatest barrier to improvement. Practicing in front of a mirror is safe but useless because it doesn't simulate the stress of a real interaction. Practicing at a therapist's office is also safe, but it's a safe space that can't scalably or repeatably replicate the dynamic, biased, and stressful scenarios of the real world.

We saw that existing VR tools were just static "exposure" rooms or mechanical "speech drills" with funny looking animated characters. We were inspired to build the missing piece: a truly adaptive, non-judgemental, and private AI "role-player" that can safely bridge the gap from a private, identity-neutral practice space to a confident, real-world performance. The best part is that the user gets to slide a "privacy" slider which gives them control over how much of exposure they want. i.e. a person who is a beginner would practice fully in AR with AI coach assistants vs. someone who would practice in an mixed reality envrioment with live feedback and metrics.

What it does

AURA (Adaptive Unbiased Reality Agents) is a bias-aware XR therapy and coaching platform. It's a "speech therapist's super-tool" that, for the first time, simulates real, interactive conversations for patients with communication disorders.

Instead of just standing in a static room, a user can:

Engage with Adaptive AI "Role-Players": A user doesn't just "practice a speech"; they must order a coffee from an AI-powered barista, face an interview with a biased recruiter, or navigate a difficult conversation. The therapist can set the AI's personality (e.g., "impatient," "distracted") to create a dynamic, effective therapeutic loop.

Receive "Abstract" Body Language Coaching: Our custom AI provides high-level, meaningful feedback in real-time. It doesn't just show a skeleton; it warns the user, "Your arms are crossed," or, "You are slouching," by interpreting the relationships between body-tracking points.

Use the "Privacy-to-Presence" Slider: AURA provides a seamless workflow that maps to the user's psychological journey. They can "slide" from a 100% private, AI-only coach (VR), to an avatar-mediated practice, and finally to a "see-through" live presentation mode (MR) with an augmented teleprompter.

Activate the "Bias & Judgement" Toggle: This is our core ethical feature. Users can train in a completely "identity-neutral" space, focusing purely on the mechanics of their speech and posture without fear of judgement based on their identity, accent, or disability.

How we built it

AURA is built on a "Trust-by-Design" architecture, ensuring that our powerful AI remains private, secure, and under the user's control.

Platform: Meta Quest 3 (used for prototyping and PoC), leveraging its mixed-reality and passthrough capabilities.

Engine: Unity for all 3D/XR interactions and UI.

Body-Tracking & AI: We use Google's MediaPipe for robust, 100% on-device body tracking. Our core technical innovation is a custom logic layer built on top of MediaPipe's 33 landmarks. This layer uses vector math to derive abstract behavioral insights (e.g., detecting crossed arms by checking wrist-to-elbow proximity, or hands in pockets by checking wrist-to-hip proximity and visibility).

Speech & Content Analysis: We use a self-hosted instance of Whisper X for real-time transcription and GPT-5 (hosted on a private cloud) for deep content analysis (structure, coherence) and to power our generative AI "role-players."

Privacy & Ethics: All data collection is opt-in, not opt-out. Body tracking is processed locally and never leaves the device. All cloud communication is end-to-end encrypted, and our self-hosted models are not trained on user data.

Challenges we ran into

From Data to Meaning: Our biggest challenge was building our "abstract" body language AI. MediaPipe gives you 33 points in space, but what does that mean? Writing the custom algorithms to reliably and quickly interpret abstract human behaviors (like "slouching" or "crossed arms") from just raw coordinate data was incredibly complex and required constant tuning.

Real-Time AI Immersion: Latency is the enemy of immersion. Architecting our system to run transcription (Whisper X) and a generative conversational AI (GPT-5) in near-real-time—without causing frame drops in Unity was a big technical hurdle.

The "Bias-Toggle" Design: We faced deep ethical questions. How do you simulate bias for training without being offensive or harmful? How do you define an "identity-neutral" avatar? This forced us to build our "Trust-by-Design" and "opt-in" data policies from the very first line of code.

Getting Unity to work properly on limited hardware: Oh boy! We will never use unity again.. We spent hours fighting the build pipeline just to get a runnable APK. We battled constant, obscure OpenXR errors, It felt like we spent more time debugging unity than building our features.

Accomplishments that we're proud of

The "Abstract" Body-Language AI: The first time our custom algorithm correctly flashed an "ARMS CROSSED" warning in the user's view, we knew we had built something beyond a simple tracker. We built a true automated coach.

Building an Actually Private Platform: We are immensely proud of our "Trust-by-Design" architecture. We proved that it's possible to build a powerful, personalized AI coach without resorting to invasive, "opt-out" data harvesting. This is critical for earning the trust of therapists and patients.

The "Privacy-to-Presence" Slider: Seeing the full workflow in action—practicing with an AI in VR, then "sliding" to the passthrough mode to see an augmented teleprompter float in the real room—felt like magic. It's a complete, intuitive, and novel workflow that no one else offers.

Our badass demo: Honestly, we're proud that it all works together and paints a picture of our idea. Seeing the on-device body tracking correctly identify a "closed" posture, which then triggers the real-time AI coach's feedback in the MR overlay, while the user is in a live conversation that's the whole vision on screen.

What we learned

Focus is a Superpower: We started with a broad "corporate-training" idea and were lost in a crowded market. By pivoting to the high-impact, underserved therapeutic market, our entire project gained a crystal-clear, passionate mission. We learned that solving a deep "painkiller" problem is infinitely more powerful than building another "vitamin."

On-Device AI is the Future of XR: The privacy and latency gains from running MediaPipe 100% on-device are non-negotiable for a truly immersive experience. It proved that the future of XR isn't just about bigger cloud models; it's about smarter, more efficient, on-device algorithms.

In XR, the UI is the User's Reality: Designing the "Privacy-to-Presence" slider taught us that the most powerful UI isn't a button—it's a seamless, world-bending transition that maps directly to a user's psychological state.

What's next for AURA (Adaptive Unbiased Reality Agents)

This prototype is the foundation. Our next steps are focused on moving from prototype to pilot and achieving market validation from real users.

B2B2C Pilot Programs: We will partner with German university speech-language pathology (SLP) programs, and speech health services to get AURA into the hands of licensed therapists and their patients. We will try to stay out of the highly regulated DiGA category of applications.

Customer Validation: We will work with individuals to co-develop features and run studies, with the goal of publishing a whitepaper that validates AURA as an effective tool and speech coach.

Scale to New Markets: Once AURA is the validated, "trust-certified" gold standard in the coaching space, we will leverage our powerful and ethically-sound AI engine to expand into the corporate and educational markets, who also desperately need a training tool they can trust.

Built With

Share this project:

Updates