FenceXR

FenceXR is an immersive XR training system where users learn sword techniques by practicing movements and receiving real-time feedback in a spatial, interactive environment with guided interaction.

Comment

Inspiration

We were inspired by the gap between watching and actually learning physical skills. Most training today relies on videos or static instruction, making it difficult to understand and correct form in real time.

Learning martial arts is a challenging and highly technical process, and we wanted to create a way for people to train with more authenticity and guidance.

By leveraging XR technology, we can track user movement through controllers and headsets, enabling real-time feedback and a full 360° understanding of each technique.

Unlike many VR experiences that focus on unrealistic “hack-and-slash” gameplay, our goal was to build something grounded in real technique. FenceXR focuses on teaching practical sword movements inspired by both historical and modern training methods.

EXPLORE PRODUCT DESIGN & TECHNICAL ARCHITECTURE

What it does

FenceXR is an immersive XR training system where users learn sword techniques by practicing movements and receiving real-time feedback in a spatial environment.

A virtual sword master demonstrates a move, the user attempts it, and the system responds with feedback to guide improvement.

The experience is built around a simple loop:

  • Demonstration
  • Practice
  • Feedback

This allows users to build muscle memory through repetition.

How we built it

The system follows a structured flow: Demonstration → User Attempt → Feedback → Repeat

We built a simplified XR training loop focused on core interaction:

  • Unity for the XR environment and experience flow
  • OpenXR (Pico XR) for headset integration
  • Mixamo and external assets for character animations
  • Blender for asset preparation
  • Google Gemini for generating structured training scripts
  • Lovable for rapid prototyping and visualization
  • GitHub for version control
  • ElevenLabs for voice generation

EXPERIENCE AUDIO DEMO

Challenges we ran into

Our biggest challenge was scope.

The original concept included:

  • Real-time motion analysis
  • Adaptive AI feedback
  • Full combat systems

These were too complex to implement within the hackathon timeframe.

We also faced:

  • Hardware limitations with motion tracking
  • Simplified detection using controller-based inputs
  • Losing a team member with XR experience
  • We ultimately ended up pivoting to using Unity due to the difficulties around building an android project and generating an .APK file in Unreal

This forced us to quickly adapt and refocus on building a functional core experience.

Accomplishments that we're proud of

We successfully built a working XR interaction loop that demonstrates training through movement and feedback.

We are especially proud of:

  • Creating a clear and interactive XR experience under tight time constraints
  • Integrating AI-generated scripting and voice into the system
  • Translating a complex idea into a functional and presentable prototype

What we learned

We learned the importance of scoping down to a focused, achievable core.

Building a strong interaction loop proved more valuable than attempting to implement every feature.

We also gained experience:

  • Integrating AI tools into XR workflows
  • Designing experiences centered around physical interaction instead of traditional UI

What's next for FenceXR

Future development could include:

  • Real-time motion analysis
  • Adaptive training based on user performance
  • Expanded use cases beyond martial arts (sports training, physical therapy)

FenceXR has the potential to evolve into a fully interactive training platform that personalizes learning through spatial computing.

Built With

  • elevenlabs
  • github
  • google-gemini
  • loveable
  • mixamo
  • open-xr
  • pico-xr
  • unity
Share this project:

Updates