SideKik — Your Personal Taekwondo Coach in VR

Inspiration

Martial arts are practiced by hundreds of millions of people worldwide, and in many disciplines, patterns (also known as forms, tuls in ITF taekwondo, or katas in karate) are an essential part of training. They require precision, coordination, balance, and repetition. While learning patterns under the guidance of an instructor is essential, improving them often comes down to one simple thing: practice.

The inspiration for SideKik came from my own family. I'm a father of two girls who practice ITF taekwondo, and I've seen firsthand how beneficial additional practice at home can be. The challenge is that when practicing alone, it's difficult to remember every movement correctly, and learning from online videos can be cumbersome.

With standalone VR headsets becoming more capable and portable, I saw an opportunity to build something that could make independent martial arts pattern training more accessible and engaging.

That's how the SideKik prototype was created.

What It Does

SideKik is a virtual reality training application for Meta Quest that helps users learn and improve official ITF taekwondo patterns.

Instead of simply watching instructional videos, users step into a virtual training environment where a full-body animated instructor demonstrates each movement of a pattern.

Using the Quest's built-in hand and body tracking capabilities, SideKik analyzes the user's movements and compares them against reference poses. It evaluates aspects of technique such as stance, hand positioning, fist orientation, and palm direction.

When something isn't quite right, SideKik provides immediate spoken feedback, such as "Turn your left palm" or "Move your right foot forward," allowing users to make adjustments while practicing.

The application also includes:

  • Virtual instructor: Demonstrates ITF patterns movement by movement.
  • Real-time pose evaluation: Compares tracked body and hand positions with reference poses and provides scores.
  • Spoken corrections: Offers feedback in English or Slovenian.
  • Visual guidance: Footprints and floating hand targets help users understand where to step and position their hands.
  • Multiple practice modes: Users can follow automatic demonstrations, practice at adjustable speeds, or use grading mode to evaluate their performance.
  • Standalone operation: Runs directly on Meta Quest without external cameras, controllers, or additional tracking sensors.

The goal is not just to show users how a pattern looks, but to help them understand what they need to improve.

How I Built It

SideKik was developed using Unity for standalone Meta Quest headsets, combining immersive VR interactions, body tracking, hand tracking, and motion-based pose evaluation.

One of the core components is the virtual instructor. To create realistic martial arts movements, I used markerless motion capture from video footage and processed the captured motion in Blender to create animations suitable for a 3D character.

Building the training system required connecting these animations to a sequence of reference poses representing important positions throughout each ITF pattern.

The application then compares the user's tracked movements against those reference poses to evaluate their performance.

Rather than relying exclusively on a numerical score, I wanted the system to provide meaningful feedback. This led to implementing a correction system that identifies specific differences between the reference pose and the user's estimated pose and communicates them through AI-generated spoken instructions.

I also developed visual guidance elements, including floor footprints and hand-position targets, to help users understand movements spatially rather than relying only on verbal instructions.

The result is an interactive training environment that combines demonstration, movement analysis, and feedback into a single experience.

Challenges I Faced

The biggest challenges were related to motion capture, tracking accuracy, and translating real-world martial arts movements into VR.

1. Tracking Legs Without External Sensors

Meta Quest offers impressive tracking capabilities, but accurately tracking the entire body without dedicated sensors remains challenging.

While hand positions can be tracked relatively well, lower-body tracking relies heavily on estimated positions rather than direct measurements.

This is particularly problematic for taekwondo, where foot positioning, weight distribution, and stance geometry are essential parts of correct technique.

I had to account for these limitations when designing the pose evaluation system, recognizing that some lower-body measurements are approximations rather than perfectly reliable observations.

2. Creating Martial Arts Animations Using Markerless Motion Capture

Another significant challenge was creating realistic and technically accurate animations for the instructor.

Traditional motion capture requires specialized equipment, which wasn't practical for this project. Instead, I worked with markerless motion capture technology to extract movement from regular videos.

This introduced its own problems. Martial arts movements can be fast and complex, involving rotations or temporary occlusion of body parts. Markerless tracking sometimes produced inaccurate joint positions, unnatural transitions, or incorrect limb orientations.

Cleaning up and adapting these animations for use in Unity required additional effort, especially because precise movement is central to the purpose of the application.

3. Physical Space Requirements

Unlike many VR applications that can be used while sitting or standing in one place, taekwondo patterns involve stepping, turning, and moving through space.

This means SideKik requires a sufficiently large, obstacle-free physical training area.

I had to consider how to communicate movement instructions and position visual guides in a way that makes sense in both the virtual and physical environments.

4. Making the Experience Approachable

ITF patterns contain many technical details, and even with a virtual instructor, the amount of information can be overwhelming for someone who has never practiced martial arts.

Balancing accurate technical feedback with an approachable experience was an important challenge.

Adjustable practice speeds, step-by-step demonstrations, and visual movement guides help make the experience easier to follow, but SideKik is currently better suited to people who already have some familiarity with taekwondo or experience in similar martial arts.

What I Learned

Developing SideKik taught me a great deal about the opportunities and limitations of standalone VR for physical skill training.

One of the most important lessons was that tracking movement is not the same as understanding movement.

Being able to estimate where someone's hands or feet are is only the first step. Translating that information into useful, actionable feedback requires understanding which aspects of a movement matter and how reliably they can be measured.

I also learned how challenging it can be to turn real-world motion into believable and technically meaningful character animations, particularly when working with markerless motion capture.

Most importantly, this project reinforced my belief that VR can be more than entertainment. It can become a practical tool for developing real-world skills.

What's Next for SideKik

My long-term vision is to make SideKik a useful companion for martial artists who want to practice more consistently and receive helpful feedback outside scheduled training sessions.

Future improvements could include:

  • Expanding the library of supported ITF patterns.
  • Improving stance evaluation as headset and body tracking technologies evolve.
  • Adding support for third-party SlimeVR trackers to improve full-body tracking accuracy.
  • Developing more beginner-friendly tutorials and explanations.
  • Introducing progress tracking to help users see how their technique improves over time.
  • Supporting more languages and personalized training experiences.
  • Creating higher-quality 3D assets and animations.
  • Developing a more accurate and reliable motion capture workflow.
  • Exploring support for other martial arts disciplines that use structured patterns or forms.
  • Creating spin-off applications that apply similar VR training techniques to learning other physical skills.

I'm particularly excited about the future of smaller, lighter, and more portable XR headsets. As these devices evolve and become easier to use, experiences like SideKik could become increasingly practical for everyday training.

An important distinction: SideKik is not intended to replace qualified martial arts instructors or traditional training. Martial arts involve much more than reproducing individual poses, including balance, power, timing, breathing, and principles that cannot be fully evaluated using current standalone headset tracking.

Instead, SideKik is designed as a supplementary practice tool that encourages repetition, reinforces learned techniques, and helps students stay engaged between training sessions.

For me, the project started with wanting to help my two daughters get more out of their taekwondo practice.

I hope it can eventually do the same for many others.

Built With

Share this project:

Updates

Submission history