StabilityCoach

Inspiration

Falls are the leading cause of injury for older adults, but that data didn’t truly hit us until last month, when my own grandfather suffered a fall that led to two emergency brain surgeries. One slip, one moment of fear, and his entire world changed. And for our family, so did ours.

When we started this hackathon, we kept coming back to one painful truth: older adults don’t fall because they’re weak, they fall because they’re scared. Fear changes how you move. You hesitate. You freeze. You lose confidence in your own body. Today’s fall-prevention tools only react after someone is already hurt.

So we asked a different question: what if mobility support could understand a person’s emotions, not just their steps? What if a system could sense rising stress, slow down guidance, or pause entirely, the same way a real caregiver would?

That’s what inspired us to build an XR mobility coach that listens to both the body and the mind. Using passthrough AR to see the environment, Afference’s neural haptics to guide movement through touch, and OpenBCI’s emotional sensing to adapt in real time, we’re creating support that feels human, not technical. No interfaces, no learning curve. Just intuitive cues that meet seniors where they are.

For us, this isn’t about making a cool demo. It’s about dignity. It’s about independence. It’s about making sure another family doesn’t get that same terrifying phone call.

What it does

Our project is an emotionally adaptive XR mobility coach designed to prevent falls before they happen. Using passthrough AR, neural haptics, and real-time BCI signals, it guides older adults through everyday movements, like stepping forward, approaching a chair, and walking around obstacles while continuously sensing their stress level and adjusting support on the fly.

Here’s how it works:

It sees the room with you: Using Meta’s passthrough and scene understanding, the headset recognizes obstacles like chairs, furniture, or rugs and maps safe paths around them with simple, calming visual cues.
It senses how you feel: OpenBCI measures changes in emotional arousal. If stress rises, the system instantly slows down guidance, shows fewer steps ahead, or pauses entirely to let the user breathe.
It guides movement through touch, not menus: Afference’s neural haptic ring gives subtle directional pulses, a gentle nudge to step forward, a soft vibration when too close to a chair, or a slow rhythmic pulse during grounding.
It adapts in real time to the user’s nervous system. Calm: Shows more tiles, normal coaching. Mild stress: Slows down pacing, reduces visual load. High stress: Stops movement, dims UI, activates grounding mode.
It requires zero tech literacy: There are no controllers, no buttons, and no gestures to learn. The user simply stands, breathes, and takes small guided steps, and the system does the rest.

In short: Our system combines spatial awareness, emotional sensing, and intuitive haptic coaching to help older adults move safely and confidently in their own homes, reducing fall risk and restoring independence through technology that feels human, not overwhelming.

How we built it

We built the entire experience in Unity, which served as the backbone for mixed reality, spatial awareness, and real-time physiological feedback. Unity allowed us to bring together multiple devices, SDKs, and intelligent subsystems into a single seamless application tailored for safe movement and emotional support.

To situate the user in their real surroundings, we integrated the Meta XR SDKs, enabling passthrough, scene understanding, and dynamic scene mesh generation. With this, the app could recognize floors, obstacles, and furniture, allowing us to spawn guidance markers directly inside the user’s room and generate safe walkable paths that adapted to the actual physical layout.

For haptic interaction, we incorporated the Afference SDK, which connected the Afference ring to the Unity experience. This allowed us to synchronize hand movement with gentle haptic cues and use them as part of the calming and breathing sequences. In parallel, the OpenBCI Cyton allowed us to bring real EEG data into Unity. A small host application forwarded the alpha and beta wave readings to Unity via WebSockets, giving the app a live sense of the user’s stress state.

The brainwave data was then sent to Ollama, a locally running Meta’s Llama 3.2 model. Ollama interpreted the user’s real-time stress patterns and returned guidance that blended reassurance, breathing support, and adaptive next steps. These AI-generated messages replaced the need for predefined scripts and instead offered context-aware emotional support tailored to how the user was actually feeling in the moment.

To bring these messages to life, we used Meta’s Text-to-Speech SDK, generating comforting, conversational voice feedback that felt empathetic and responsive. Finally, we authored custom C# systems in Unity to handle the breathing halo animations, path spawning on the scanned room mesh, trigger detection when users reached destinations, and audio sequencing with callbacks.

Challenges we ran into

Our biggest UX challenge was making haptics feel supportive, not startling. Older adults can be sensitive to unexpected sensations, so every pulse had to be gentle, predictable, and emotionally calming.

Software Integration - One of our biggest blockers was the OpenBCI pipeline. On macOS, the OpenBCI GUI often failed to establish a stable connection with the Cyton board, preventing us from accessing EEG data early in development.

EEG Signal Instability - Even when the Cyton finally connected, several channels showed noticeable interference. This made the alpha–beta readings unreliable and harder to use for generating meaningful feedback.

Hardware Reliability - Beyond software and signal issues, some hardware components were unstable. The Cyton occasionally dropped its connection, and the Afference ring had intermittent firmware and wiring problems that required constant troubleshooting.

Designing for Elderly Users - Since the experience was built for older adults, we had to prioritize clarity, comfort, and simplicity. Ensuring calm audio cues, high-visibility passthrough visuals, and a non-overwhelming flow added design constraints that ultimately improved the user experience.

Accomplishments that we're proud of

We built an end-to-end adaptive loop that reads EEG signals from the Cyton in real time, processes them with an on-device LLM (Ollama), and delivers personalized visual and audio feedback inside Unity based on the user’s stress level.
We integrated passthrough, scene understanding, and spatial interaction features from the Meta Quest 3 to create a calm, intuitive mixed-reality environment.
We designed the entire flow around older adults, focusing on simplicity, safety, and accessibility to ensure the experience remains comfortable and easy to follow.

What we learned

Designing for older adults taught us to rethink clarity, pacing, and comfort in every interaction.
Working with real-time EEG signals showed us how unpredictable biosensor data is and why fallback strategies matter.
Coordinating Quest ↔ Laptop ↔ Cyton ↔ Afference Ring required robust error-handling to keep everything in sync.

What's next for StabilityCoachXR

This weekend was just the beginning. We’re building a coach that grows with every step a user takes learning their movement patterns, sensing their stress, and predicting fall risk before it happens. The next version adapts entire homes through spatial anchors, smart lighting, and personalized AR cues, and connects directly with families and therapists so no one has to face fear alone. Our goal is simple and massive: use AI, XR, haptics, and emotional sensing to give older adults their independence back, not someday, but now.